minicpm5-1b-arcsum (sft-dropv2)

A MiniCPM5-1B fine-tune that reads a zh-TW meeting transcript in ~2,500-token chunks and curates a small external memory, then writes a single flowing zh-TW prose summary.

Read this first: this checkpoint did NOT pass its own ship gates. It clears 6 of 7. Under the project's spec — which is all-or-nothing — the recorded decision is "ship the map-reduce baseline", and this model is published as a measured negative result, not as a winner. Details below, including the two numbers that carry caveats. Nothing here is rounded in the model's favour.

What it does

The transcript is read chunk by chunk. The harness owns a two-slot external memory — ARC (a rolling one-sentence throughline, ≤80 tokens) and POINTS (≤16 short facts) — and re-renders it into every step's prompt. No conversation history crosses steps. The model emits only edit lines:

ARC: <replacement throughline>
ADD - <new point>
DROP «<prefix of an existing point>»
NOP

The harness applies them deterministically, including cap-overflow handling. A final SYNTHESIZE call turns the finished memory into prose.

The point of the external memory is the thing aggregate scores cannot show: a later chunk can overturn an earlier conclusion. Map-reduce structurally cannot do this — each window is summarised independently, so a decision reversed at minute 90 never reaches the summary of minute 10. That capability is what the G1 gate tests directly, and it passes.

Measured results (n = 20 held-out meetings, paired)

Against a fair map-reduce baseline: same model, same chunk size, same output contract — not a strawman.

gate	result
G1 revision probe	PASS	states the later decision, never the stale one
G2 faithfulness	PASS	8 inversions vs baseline's 18
G3 ROUGE-2	PASS	+0.055, p=0.000, wins 19/20
G3 ROUGE-L	PASS	+0.066, p=0.000, wins 19/20
G4 on-device budget	PASS	19.58 min vs a 20 min ceiling — projected
G3 ROUGE-1	FAIL	+0.056 (lower bound +0.035) but sign test 14/20, p=0.115

Why ROUGE-1 fails. Not effect size — that clears. The sign test misses: 14 of 20 meetings win, and 15 would give p=0.021. The losses are concentrated in long meetings (median 30 chunks, vs 12 for wins), where the model fixates — re-emitting an unchanged ARC while the transcript has moved on. Diagnosed, not guessed: only 1.6% of training steps sit at index 40+, while the correct behaviour there (NOP) is more common than early on, not less.

Two numbers that need their caveats

• G4 was never measured on the phone. It is arithmetic: a measured reading step (71s thin / 90s saturated, on the real device, same quantisation) × 14 steps, plus synthesis scaled linearly to this model's measured 317-char mean output. The margin is 2.1%, and thermal throttling over a sustained 20-minute run on a passively cooled phone is not modelled and acts in the failing direction.
• G2 rests on 18 paired meetings, not 20. Two baseline cases never scored even at a 14,000-token judge budget. Longer summaries are what exhaust the judge, so the excluded cases are plausibly the baseline's hardest — which flatters this model slightly.

Diagnostics (deliberately not gates)

	agent	baseline
coverage	0.982	0.993
density	3.26	4.05

Both measure extractiveness — how much text is copied verbatim. The agent is less extractive, which is the design goal (flowing abstractive prose), so a lower number here is expected rather than bad. They were briefly gated by mistake, which made shipping unreachable by construction: passing would have required the agent to copy more than map-reduce. Read them as shape descriptors, not scores.

Mean summary length: agent 317 chars, baseline 691, human reference 471. The agent wins ROUGE while producing summaries under half the baseline's length.

Files

file	use
MiniCPM5-1B.Q8_0.gguf	the evaluated artifact. Every number above was measured on this.
MiniCPM5-1B.Q4_K_M.gguf	smaller/faster, for interactive demos. Not what was evaluated — no gate number applies to it.

Usage

The edit-line protocol is not self-describing; the model expects the harness's exact prompts and a memory rendered fresh each step. Use the harness rather than prompting it directly:

llama-server -m MiniCPM5-1B.Q8_0.gguf --ctx-size 4096 --jinja --temp 0

Two serving details, both measured the hard way:

• Pin cache_prompt: false for anything you intend to report. llama.cpp's prompt cache changes generation: same model, same seed, same prompt returned 700 characters with the cache on and 167 with it off, each internally deterministic across three repeats. Results are reproducible given a cache state, not across runs.
• Use a repetition penalty (~1.1) on the prose call only, never the reading steps. Greedy decoding degenerates: one synthesis emitted the same sentence eight times. Reading steps emit a fixed op vocabulary, so a penalty there punishes the literal ADD/DROP/ARC tokens the format requires.

Training

Full fine-tune (not LoRA) of openbmb/MiniCPM5-1B, 3 epochs, lr 5e-5, on ~4.3k steps derived from 200 MeetingBank meetings machine-translated to zh-TW, with per-step targets produced by a Qwen3.8-27B teacher and replayed through the real harness before use.

The pool balance is load-bearing and easy to get wrong. An earlier checkpoint pushed the NOP share to 25.7% against the teacher's natural 38.2% — two rebalancing knobs compounding — and the model stopped emitting NOP at all, instead churning: DROP followed by a near-identical re-ADD, burning up to 45 of one 53-step meeting's steps on a single topic. A later attempt to fix the long-meeting weakness by oversampling late steps also regressed it (ROUGE-1 14/20 → 12/20), even though it demonstrably improved the meeting it targeted. Stable label shares did not imply stable behaviour.

Limitations

• zh-TW only. English is source material, never a product language.
• Long meetings. Beyond ~30 chunks the model fixates; this is the open weakness and the reason ROUGE-1 misses. It needs more long-meeting supervision, not reweighting — that was tried and rejected.
• Not a general chat model. It emits edit lines, and outside the harness's prompts its behaviour is undefined.
• Reference summaries are teacher-synthesised from machine-translated English meetings, so no score here is comparable to published MeetingBank numbers.