aminechr54/pan26-style-change-deberta

DeBERTa cross-encoder for sentence-level author-change detection (PAN 2026)

Given two consecutive sentences of a document, this model predicts the probability that the author changed between them. It was fine-tuned for the PAN@CLEF 2026 Multi-Author Writing Style Analysis shared task, on the hard difficulty tier, where every sentence of a document is on the same topic — so a model cannot fall back on topic drift as a proxy for authorship.

The model is the submitted system of a Praktikum project that also measures how much each method relies on topic rather than style, by re-scoring every model on a POSNoise-masked copy of the corpus (content words replaced by POS placeholders).

• 📄 Report: report.pdf in this repo
• 🧑‍💻 Code: https://github.com/aminechr54/multi-author-writing-style-analysis-praktikum
• 🤗 Demo: https://huggingface.co/spaces/aminechr54/pan26-style-change-demo (runs in your browser, no backend)
• 📚 Corpus: PAN 2026 MAWSA, Zenodo record 19068843

Architecture

[CLS] s_i [SEP] s_i+1 [SEP]  ->  microsoft/deberta-base  ->  h_[CLS] (768)
                             ->  Linear(768, 256) -> ReLU -> Dropout(0.1) -> Linear(256, 1)
                             ->  sigmoid -> P(author change)

A cross-encoder, not a bi-encoder: the attention layers see both sentences at once and can compare tokens across the pair, instead of only comparing two independently computed sentence vectors. DeBERTa has no pooler output, so the first token's last hidden state is read directly. Trained with BCEWithLogitsLoss on all adjacent sentence pairs of the training documents.

Results

Validation split of the hard tier: 127,380 adjacent sentence pairs, 27,065 of them a change (21.2% positive). The task metric is macro F1 over {no change, change}.

	τ = 0.5	τ = 0.37 (dev-tuned)
Macro F1 (raw text)	0.7485	0.7556
Macro F1, POSNoise-masked text	0.6798	0.6980 (τ = 0.32)
all-zeros baseline (predict "no change" everywhere)	0.4406	0.4406

Further metrics on raw text at τ = 0.5:

Accuracy	Balanced acc.	MCC	ROC-AUC	PR-AUC	Brier
0.8534	0.7201	0.5166	0.8633	0.6949 (baseline 0.2125)	0.1094

Per class at τ = 0.5:

class	precision	recall	F1	support
no change	0.8733	0.9520	0.9110	100,315
change	0.7328	0.4883	0.5861	27,065

Confusion matrix at τ = 0.5: TN 95,495 · FP 4,820 · FN 13,849 · TP 13,216.

Topic reliance. Scoring the same model on the POSNoise-masked validation split costs 0.0688 macro F1 (0.7485 → 0.6798), i.e. 22.3% of the headroom above the all-zeros floor. Most of what this model has learned survives the removal of content words, which is the point of the hard tier — but the remainder is topic, not style.

The thresholds above are tuned on a held-out dev slice of the training split, never on the validation split that is scored. τ = 0.37 is the default of the shipped inference code.

Usage

The checkpoint carries the custom classification head, so it does not load with AutoModel/AutoModelForSequenceClassification — you need src/cross_encoder.py from this repo (a dependency-free copy of the training-time module; the parameter names must match the state_dict).

from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer
import sys

repo = "aminechr54/pan26-style-change-deberta"
sys.path.insert(0, hf_hub_download(repo, "src/cross_encoder.py").rsplit("/", 1)[0])
from cross_encoder import load_checkpoint, pair_probs

model = load_checkpoint(hf_hub_download(repo, "model.pt"), device="cpu")
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-base")

pairs = [
    ("The algorithm converges after roughly forty iterations.",
     "well i mean it kinda just stops getting better after a while lol"),
    ("The algorithm converges after roughly forty iterations.",
     "Convergence is reached once the gradient norm falls below the tolerance."),
]
probs = pair_probs(model, tokenizer, pairs, device="cpu")
print([round(float(p), 3) for p in probs])        # P(author change) per pair
print([int(p >= 0.37) for p in probs])            # decision at the dev-tuned threshold

Whole documents, in the PAN input format (problem-N.txt, one sentence per line, output solution-problem-N.json with {"changes": [...]}):

python src/predict.py -i ./problems -o ./solutions --checkpoint model.pt --threshold 0.37

Inference takes roughly 0.2 s per sentence pair on a laptop CPU including model load, and is far faster on a GPU. The GitHub repo additionally ships a Docker image with the TIRA entry point baked in.

In the browser

onnx/model_int8.onnx (144 MB) is the same network as a single-file ONNX graph, dynamically quantized to int8 per channel, with tokenizer.json next to it for transformers.js. It takes two inputs, input_ids and attention_mask (DeBERTa v1 has type_vocab_size = 0, so token_type_ids are unused), both int64 with shape [batch, seq], and returns one logit per pair — apply a sigmoid yourself. This is what the demo Space runs, with no backend.

Cost of the quantization, measured against this checkpoint on 300 held-out pairs:

mean |Δp|	max |Δp|	decisions flipped at τ = 0.37	macro F1
0.008	0.034	1.3%	0.7594 vs 0.7629

hf/export_onnx.py in the GitHub repo regenerates both files. Per-channel quantization matters here: per-tensor drifts four times further (mean |Δp| 0.038, 3.7% of decisions flipped).

Training

Trained on the hard tier with --corpus both: the raw training pairs plus their POSNoise-masked copies, 1,058,038 pairs in total, as a style-oriented augmentation.

backbone	microsoft/deberta-base
epochs	2 (best epoch: 2)
optimiser	AdamW, lr 2e-5, weight decay 0.01
schedule	linear, warm-up ratio 0.1
batch size	32 (eval 64)
max sequence length	128 tokens per pair
head	256 hidden, dropout 0.1
precision	fp32 (DeBERTa v1 does not run under fp16 autocast)
seed	42
cost	5.5 h on one GPU of the TU Darmstadt Lichtenberg cluster, 7.4 GB VRAM peak

The POSNoise-masked half of the training data (and the masked validation split) is regenerated from the official corpus with python -m maa.posnoise.posnoise_transformation in the GitHub repo — it is not redistributed here.

Run id cross_encoder_deberta_both_20260915_163055; the full results JSON behind every number above is in the GitHub repo under src/maa/job/logs/results/.

Limitations

• Trained on one tier. Only the hard tier (single-topic documents) was used. The easy and medium tiers have different positive rates, and the dev-tuned threshold does not transfer — re-tune per tier.
• Recall on changes is the weak spot. At τ = 0.5 it finds under half of the real changes (recall 0.4883) at precision 0.7328. Lowering the threshold to 0.37 trades precision for recall and is what the macro-F1 optimum wants; pick the threshold for your own cost ratio.
• Pairwise, not sequential. Each boundary is classified independently; the model has no document-level view and can produce implausible alternating segmentations. No smoothing, CRF or segment prior is applied.
• English, Reddit-style prose from the PAN corpus. Behaviour on other languages, domains or sentence lengths is untested. Sentences are truncated to 128 tokens per pair.
• Still partly topical. 22.3% of its headroom above the trivial baseline disappears when content words are masked, so part of the signal is topic rather than style.
• Not a plagiarism verdict. A high probability means "these two sentences read as if written by different people", on this corpus' distribution. It is not evidence about any real author, and should not be used to accuse anyone of anything.

Licence

The code in this repo is MIT. The weights are fine-tuned from microsoft/deberta-base (MIT) on the PAN 2026 MAWSA corpus, which Zenodo publishes under CC BY 4.0 (record 19068843) — redistribution and re-use are allowed with attribution, which the citation below provides. The few corpus sentences used as examples in the demo Space are credited there as well.

Citation

@misc{cheikhrouhou2026mawsa,
  title  = {Sentence-level author-change detection with topic-controlled evaluation},
  author = {Cheikhrouhou, Amine},
  year   = {2026},
  note   = {Praktikum report, Technische Universit\"at Darmstadt},
  url    = {https://github.com/aminechr54/multi-author-writing-style-analysis-praktikum}
}

POSNoise masking follows Halvani & Graner; the corpus is by Zangerle, Mayerl, Stein and Potthast (PAN@CLEF 2026).