# exp019_cr — the capacity for distilled content retention (+ generalization)

When content is **distilled** rather than directly learned, how much transfers,
how much is retained, and for how long? Sequel to [exp016](../exp016_ct/) and
the [exp014](../exp014_gd/) memory-substrate studies; the weak-to-strong
distillation cell made real (the teacher holds content the student lacks —
true headroom on the content axis).

**Instrument**: N fact records `\n@<key6>=<value12>\n` (random alphanumeric —
uncompletable from language statistics, so exact-match greedy completion IS
retention), mixed into the wikitext byte stream at rate 0.5. Teacher =
addr_msl64 bed model, 4000 steps, gated on its own recall. Channels (fresh
student, 2000 steps): `direct` (ground truth; ceiling) | `kd_facts` (fact rows
supervised ONLY by teacher logits — pure distilled content, α=1.0) |
`kd_general` (KD on clean text only — leakage probe) | `book_implant`
(teacher's trained codebook into a fresh student) | `none` (floor). Axes:
N ∈ {64, 256, 1024}; retention re-measured after 1000 further clean-stream
steps (interference). The **19b block** swaps rote facts for **rule-bearing
content** (value = fixed substitution cipher of the key; 256 train keys, 128
held out) plus prompt-format variants.

`build_results.py` re-asserts every claim below from `results/ledger.jsonl`
(36 rows).

## Findings — retention sweep

1. **Capacity curve**: teacher recall at 4k steps = 1.00 (N=64), 0.99 (N=256),
   0.01–0.20 (N=1024) — the ~1.9M model holds ~256 facts near-perfectly and
   hits a cliff before 1024 (cliff edge seed-chaotic).
2. **The distillation tax is zero-to-negative.** Where the teacher knows the
   content, teacher logits alone transfer it at parity with ground truth
   (N=64: exact 1.0 = 1.0, both seeds) or better (N=256: 0.953 vs 0.871 s0;
   0.859 vs 0.856 s1). Soft targets from a teacher with headroom are at least
   as content-efficient as the data itself. The cost is general modeling:
   kd students' clean bpb 4.0–4.3 vs direct 2.8–3.1 vs floor 2.47.
3. **No logit leakage**: KD on clean text transfers zero facts (0.000, 6/6
   cells) — content does not cross without exposure at this scale.
4. **Anchors carry no bytes**: the trained codebook of a teacher that knows
   256 facts, implanted into a fresh student, transfers **none** of them
   (0.000, both seeds). The codebook organizes addressing; content lives in
   the trunk.
5. **Universal catastrophic forgetting**: after 1000 clean-stream steps, every
   channel's recall is exactly 0.000 — including perfectly-learned direct
   students. At this scale nothing persists without pressure; **persistence,
   not transfer, is the unsolved axis of the memory substrate.**

## Findings — 19b generalization block (rule content)

6. **Rules are learned fragmentarily**: teachers memorize the rule-pairs
   near-perfectly (train exact 0.98–1.00) but reach only ~0.25–0.27 held-out
   byte accuracy (~10× chance) with **zero** exact completions — character
   mappings absorbed, the full cipher never.
7. **The logit channel beats direct learning on structured content, 2/2
   seeds** (train exact 0.758/0.844 vs 0.652/0.773) and transfers the partial
   rule at full fidelity — the dark-knowledge advantage, certified on rules.
8. **Content is format-locked** — variant-format prompts collapse recall to
   ~0 even on trained keys — but **KD students are consistently less
   format-locked** than direct students (variant byte 0.077–0.091 vs
   0.000–0.024): soft targets bind content less rigidly to surface form.
9. Interference erases the rule too — structure forgets like instances.

## Files
- `exp019_content_retention.py` — fact/rule corpora, streams, exact-match
  recall (with format variants), channel training, both runners, smoke.
- `geolip_vitals.py` / `ar_differentiation_bed.py` /
  `exp014_genetic_distillation.py` / `read_codebook.py` — this package's own
  harness copies. Standalone.
- `repro.py`, `build_results.py`, `results/ledger.jsonl` (36 rows),
  `specimens/` (6 fact teachers + 2 rule teachers + 6 rule students).

## Reproduce (from inside this folder)
```bash
pip install torch --index-url https://download.pytorch.org/whl/cu128
pip install pyarrow huggingface_hub
python repro.py            # CPU smoke
python repro.py --run      # retention sweep (GPU, ~3h)
python repro.py --run19b   # rule/generalization block (GPU, ~1h)
python build_results.py    # re-assert every claim from the ledger
```
Data lands in `./data` (override with `GEOLIP_DATA`).

License: MIT · AbstractPhil + Claude Fable 5 · July 11, 2026