| --- |
| license: mit |
| tags: |
| - nanochat |
| - gpt |
| - language-model |
| - custom-trained |
| library_name: nanochat |
| pipeline_tag: text-generation |
| --- |
| |
| # nanochat-keith-mid |
|
|
| This is a custom-trained nanochat model uploaded from checkpoint: **mid/d20** at step **814**. |
|
|
| ## Model Details |
|
|
| - **Architecture**: GPT-style transformer |
| - **Training Stage**: MID |
| - **Model Tag**: d20 |
| - **Training Step**: 814 |
|
|
| ### Model Configuration |
|
|
| ```json |
| { |
| "sequence_len": 2048, |
| "vocab_size": 65536, |
| "n_layer": 20, |
| "n_head": 10, |
| "n_kv_head": 10, |
| "n_embd": 1280 |
| } |
| ``` |
|
|
| ### Training Configuration |
|
|
| ```json |
| { |
| "run": "dummy", |
| "device_type": "", |
| "dtype": "bfloat16", |
| "num_iterations": -1, |
| "max_seq_len": 2048, |
| "device_batch_size": 32, |
| "unembedding_lr": 0.004, |
| "embedding_lr": 0.2, |
| "matrix_lr": 0.02, |
| "init_lr_frac": 1.0, |
| "weight_decay": 0.0, |
| "eval_every": 150, |
| "eval_tokens": 10485760, |
| "total_batch_size": 524288, |
| "dry_run": 0 |
| } |
| ``` |
|
|
| ## Performance Metrics |
|
|
| - **Validation BPB**: 0.43976500298865956 |
| - **Validation Loss**: N/A |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| from nanochat.checkpoint_manager import build_model |
| from nanochat.tokenizer import get_tokenizer |
| from nanochat.engine import Engine |
| |
| # Download and load the model |
| device = "cuda" if torch.cuda.is_available() else "cpu" |
| |
| # Load model (you'll need to download the checkpoint files) |
| checkpoint_dir = "./checkpoint" # directory with model files |
| step = 814 |
| model, tokenizer, meta = build_model(checkpoint_dir, step, device, phase="eval") |
| |
| # Create engine for generation |
| engine = Engine(model, tokenizer) |
| |
| # Generate text |
| prompt = "Hello, how are you?" |
| tokens = tokenizer(prompt, prepend="<|bos|>") |
| samples, _ = engine.generate_batch(tokens, num_samples=1, max_tokens=100, temperature=0.8) |
| print(tokenizer.decode(samples[0])) |
| ``` |
|
|
| ## Training Details |
|
|
| This model was trained using the [nanochat](https://github.com/karpathy/nanochat) framework by Andrej Karpathy. |
|
|
| ### Training Data |
|
|
| - Custom SFT data: keith_identity.jsonl, keith_reputation.jsonl |
| - Base datasets: SmolTalk, MMLU, GSM8K, ARC, SpellingBee |
| - Training stage: mid |
|
|
| ### Computational Requirements |
|
|
| Training was performed following the nanochat speedrun pipeline. |
|
|
| ## Citation |
|
|
| If you use this model, please cite the original nanochat project: |
|
|
| ``` |
| @misc{nanochat, |
| author = {Karpathy, Andrej}, |
| title = {nanochat: A minimalistic ChatGPT clone}, |
| year = {2024}, |
| publisher = {GitHub}, |
| url = {https://github.com/karpathy/nanochat} |
| } |
| ``` |
|
|
| ## License |
|
|
| This model follows the MIT license of the nanochat project. |
|
|
|
|
| ## Checkpoint Lineage and Retention |
|
|
| This repository is the **MID** checkpoint in the private lineage: |
|
|
| 1. [`keithtyser/nanochat-keith`](https://huggingface.co/keithtyser/nanochat-keith) — base checkpoint |
| 2. [`keithtyser/nanochat-keith-mid`](https://huggingface.co/keithtyser/nanochat-keith-mid) — intermediate checkpoint |
| 3. [`keithtyser/nanochat-keith-sft`](https://huggingface.co/keithtyser/nanochat-keith-sft) — supervised-fine-tuned checkpoint |
|
|
| Keep all three while reproducibility, regression comparison, rollback, or training resume matters. Stage-specific data claims in the older card text should be treated as provisional unless they are backed by a dataset manifest and training log; the datasets themselves are not bundled here. |
|
|
