File size: 5,692 Bytes
76543b3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 | ---
language: zh
license: mit
tags:
- chinese
- white-box
- nlp
- cognitive
- text-correction
- interpretability
datasets:
- shibing624/chinese_text_correction
- MuCGEC
metrics:
- accuracy
- rouge-l
- cosine-similarity
model-index:
- name: V19 White-box Chinese Cognition Engine
results:
- task:
type: text-correction
dataset:
name: shibing624/chinese_text_correction + MuCGEC
type: public-combined
metrics:
- type: word-accuracy
value: 92.4
- type: rouge-l-f1
value: 93.2
---
# V19 White-box Chinese Cognition Engine
## Model Summary
V19 is a **fully interpretable** Chinese language understanding system with **4.7 million parameters**. It reads a Chinese sentence as a sequence of characters, builds word-level representations through a frozen char-to-word encoder (P1), routes information across sentences (P7), and decodes back to word sequences (P6) — all while maintaining traceable, auditable intermediate states.
Unlike transformer-based LLMs, every internal decision in V19 can be inspected:
- **P1**: Which words each character pair maps to
- **P7**: How words route across sentences (32-head attention with per-head gating)
- **Explore+Meta Gate**: Which decoding dimensions are active and why
- **P6**: How each output word is decoded from the sentence vector + position embedding
## Intended Use
- **Chinese text correction** (primary task)
- **Interpretability research**: study how linguistic attributes compose without black boxes
- **Education**: demonstrate NLP concepts with fully transparent architecture
- **Low-resource deployment**: 141MB GPU, runs on CPU at 71 sent/s
## Architecture
```
P1 (Char→Word, 96K frozen) → P7 (Router, 226K) → Explore+Meta (Gate, 101K) → P6 (Decoder, 4.37M)
```
### P1: Char→Word Encoder (frozen)
- Input: 2 consecutive characters
- Output: 128-dimensional word vector
- Cross-attention over 6,000-word vocabulary
- Batch encoding (50 words/batch) to control GPU memory
### P7: Cross-sentence Router
- 32 heads × 4 dimensions
- P5-style ±superposition for sent_vec (learnable positional weights, not mean pooling)
- Output: 256D sentence vector
### Explore + Meta Gate
- 12D loss vector → Explore network (128→256→256→tanh) → 256D control signal
- Meta: sigmoid(bias + signal) → 256D gate
- Gate modulates P6 encoder output dimension-wise
- Learns *when* to open/close dimensions without direct loss minimization
### P6: Sentence → Word Decoder (Position Embedding V6)
- Encoder: 256→256→256 (GELU)
- 128 independent extraction heads, each: `h * gate + pos_embed[i] → Linear(256,128) → word_i`
- Position embedding provides unique starting point per head — naturally prevents repetition collapse
- No rep_pen, no residual subtraction, no detach needed
## Training
| Config | Value |
|--------|-------|
| Optimizer | Adam (P6 lr=0.003, P7 lr=0.0045, Gate lr=0.006) |
| Loss | `1.0 - mean(cosine_similarity(pred, true))` |
| Epochs | 1000 |
| Batch | Full dataset per epoch (41,909 pairs) |
| GPU | RTX 5070 |
| Memory | ~300MB (training), 141MB (inference) |
## Evaluation
### V18 (875K params, 16 heads)
| Metric | Score |
|--------|-------|
| Word Accuracy | 92.4% |
| Exact Match | 76.3% |
| Rouge-L F1 | 93.2 |
| Per-word Cosine Mean | 0.96 |
| Inference | 14ms/sentence |
### V19 (4.7M params, 128 heads) — in training
| Metric | Epoch 1 | Target |
|--------|---------|--------|
| Word Accuracy | 43.5% | >95% |
| Per-word Cosine | 0.73 | >0.97 |
## Key Innovations
### Position Embedding V6 (Anti-collapse)
After 5 failed approaches to prevent the P6 decoder from outputting the same word repeatedly (rep_pen, residual extraction, weight transpose inversion, orthogonal init, cos_loss margin), the final solution was the simplest:
```python
for i in range(max_words):
hi = h + self.pos_embed[i] # unique starting point per head
w = self.extract[i](hi)
```
No rep_pen. No residuals. No detach. Just position diversity.
### Explore→Meta Gate
Instead of directly minimizing loss (which causes gates to converge to all-open or all-closed), the gate is trained *indirectly*:
1. Loss flows into Explore network → produces 256D signal
2. Meta applies learned bias + sigmoid → 256D gate
3. Gate modulates P6 encoder → affects word predictions
4. Gate quality is measured by per-head prediction accuracy, not total loss
This prevents the "gate symmetry lock" (all dims identical, std=0.0001) that plagued early versions.
## Data
| Source | Pairs | License |
|--------|-------|---------|
| [shibing624/chinese_text_correction](https://huggingface.co/datasets/shibing624/chinese_text_correction) | 53,298 | Apache 2.0 |
| [MuCGEC](https://github.com/HillZhang1999/MuCGEC) | 1,038 | CC BY 4.0 |
After sentence splitting: **52,387 pairs**
Split: train 41,909 (80%) / test 5,238 (10%) / exam 5,240 (10%)
## Limitations
- **Chinese only**: Character set limited to 6,164 unique characters from training data
- **Sentence length**: Max 128 characters (configurable, but untested beyond)
- **No multilingual support**: Architecture assumes CJK character structure
- **Training data bias**: Primarily news/law/medical domains from text correction dataset
- **V19 training incomplete**: 1000-epoch training in progress; current model may be suboptimal
## Citation
```bibtex
@misc{wei2026v19,
title={V19: A White-box Chinese Cognition Engine},
author={Wei, Jinqi},
year={2026},
howpublished={\url{https://github.com/Xuan-yi-yan/V18-cognitive-architecture}},
}
```
## Contact
GitHub: [@Xuan-yi-yan](https://github.com/Xuan-yi-yan)
|