exp019_cr/README.md

exp019_cr — the capacity for distilled content retention (+ generalization)

When content is distilled rather than directly learned, how much transfers, how much is retained, and for how long? Sequel to exp016 and the exp014 memory-substrate studies; the weak-to-strong distillation cell made real (the teacher holds content the student lacks — true headroom on the content axis).

Instrument: N fact records \n@<key6>=<value12>\n (random alphanumeric — uncompletable from language statistics, so exact-match greedy completion IS retention), mixed into the wikitext byte stream at rate 0.5. Teacher = addr_msl64 bed model, 4000 steps, gated on its own recall. Channels (fresh student, 2000 steps): direct (ground truth; ceiling) | kd_facts (fact rows supervised ONLY by teacher logits — pure distilled content, α=1.0) | kd_general (KD on clean text only — leakage probe) | book_implant (teacher's trained codebook into a fresh student) | none (floor). Axes: N ∈ {64, 256, 1024}; retention re-measured after 1000 further clean-stream steps (interference). The 19b block swaps rote facts for rule-bearing content (value = fixed substitution cipher of the key; 256 train keys, 128 held out) plus prompt-format variants.

build_results.py re-asserts every claim below from results/ledger.jsonl (36 rows).

Findings — retention sweep

1. Capacity curve: teacher recall at 4k steps = 1.00 (N=64), 0.99 (N=256), 0.01–0.20 (N=1024) — the ~1.9M model holds ~256 facts near-perfectly and hits a cliff before 1024 (cliff edge seed-chaotic).
2. The distillation tax is zero-to-negative. Where the teacher knows the content, teacher logits alone transfer it at parity with ground truth (N=64: exact 1.0 = 1.0, both seeds) or better (N=256: 0.953 vs 0.871 s0; 0.859 vs 0.856 s1). Soft targets from a teacher with headroom are at least as content-efficient as the data itself. The cost is general modeling: kd students' clean bpb 4.0–4.3 vs direct 2.8–3.1 vs floor 2.47.
3. No logit leakage: KD on clean text transfers zero facts (0.000, 6/6 cells) — content does not cross without exposure at this scale.
4. Anchors carry no bytes: the trained codebook of a teacher that knows 256 facts, implanted into a fresh student, transfers none of them (0.000, both seeds). The codebook organizes addressing; content lives in the trunk.
5. Universal catastrophic forgetting: after 1000 clean-stream steps, every channel's recall is exactly 0.000 — including perfectly-learned direct students. At this scale nothing persists without pressure; persistence, not transfer, is the unsolved axis of the memory substrate.

Findings — 19b generalization block (rule content)

6. Rules are learned fragmentarily: teachers memorize the rule-pairs near-perfectly (train exact 0.98–1.00) but reach only 0.25–0.27 held-out byte accuracy (10× chance) with zero exact completions — character mappings absorbed, the full cipher never.
7. The logit channel beats direct learning on structured content, 2/2 seeds (train exact 0.758/0.844 vs 0.652/0.773) and transfers the partial rule at full fidelity — the dark-knowledge advantage, certified on rules.
8. Content is format-locked — variant-format prompts collapse recall to ~0 even on trained keys — but KD students are consistently less format-locked than direct students (variant byte 0.077–0.091 vs 0.000–0.024): soft targets bind content less rigidly to surface form.
9. Interference erases the rule too — structure forgets like instances.

Files

• exp019_content_retention.py — fact/rule corpora, streams, exact-match recall (with format variants), channel training, both runners, smoke.
• geolip_vitals.py / ar_differentiation_bed.py / exp014_genetic_distillation.py / read_codebook.py — this package's own harness copies. Standalone.
• repro.py, build_results.py, results/ledger.jsonl (36 rows), specimens/ (6 fact teachers + 2 rule teachers + 6 rule students).

Reproduce (from inside this folder)

pip install torch --index-url https://download.pytorch.org/whl/cu128
pip install pyarrow huggingface_hub
python repro.py            # CPU smoke
python repro.py --run      # retention sweep (GPU, ~3h)
python repro.py --run19b   # rule/generalization block (GPU, ~1h)
python build_results.py    # re-assert every claim from the ledger

Data lands in ./data (override with GEOLIP_DATA).

License: MIT · AbstractPhil + Claude Fable 5 · July 11, 2026