---
license: mit
tags:
- nanochat
- gpt
- language-model
- custom-trained
library_name: nanochat
pipeline_tag: text-generation
---

# nanochat-keith-mid

This is a custom-trained nanochat model uploaded from checkpoint: **mid/d20** at step **814**.

## Model Details

- **Architecture**: GPT-style transformer
- **Training Stage**: MID
- **Model Tag**: d20
- **Training Step**: 814

### Model Configuration

```json
{
  "sequence_len": 2048,
  "vocab_size": 65536,
  "n_layer": 20,
  "n_head": 10,
  "n_kv_head": 10,
  "n_embd": 1280
}
```

### Training Configuration

```json
{
  "run": "dummy",
  "device_type": "",
  "dtype": "bfloat16",
  "num_iterations": -1,
  "max_seq_len": 2048,
  "device_batch_size": 32,
  "unembedding_lr": 0.004,
  "embedding_lr": 0.2,
  "matrix_lr": 0.02,
  "init_lr_frac": 1.0,
  "weight_decay": 0.0,
  "eval_every": 150,
  "eval_tokens": 10485760,
  "total_batch_size": 524288,
  "dry_run": 0
}
```

## Performance Metrics

- **Validation BPB**: 0.43976500298865956
- **Validation Loss**: N/A

## Usage

```python
import torch
from nanochat.checkpoint_manager import build_model
from nanochat.tokenizer import get_tokenizer
from nanochat.engine import Engine

# Download and load the model
device = "cuda" if torch.cuda.is_available() else "cpu"

# Load model (you'll need to download the checkpoint files)
checkpoint_dir = "./checkpoint"  # directory with model files
step = 814
model, tokenizer, meta = build_model(checkpoint_dir, step, device, phase="eval")

# Create engine for generation
engine = Engine(model, tokenizer)

# Generate text
prompt = "Hello, how are you?"
tokens = tokenizer(prompt, prepend="<|bos|>")
samples, _ = engine.generate_batch(tokens, num_samples=1, max_tokens=100, temperature=0.8)
print(tokenizer.decode(samples[0]))
```

## Training Details

This model was trained using the [nanochat](https://github.com/karpathy/nanochat) framework by Andrej Karpathy.

### Training Data

- Custom SFT data: keith_identity.jsonl, keith_reputation.jsonl
- Base datasets: SmolTalk, MMLU, GSM8K, ARC, SpellingBee
- Training stage: mid

### Computational Requirements

Training was performed following the nanochat speedrun pipeline.

## Citation

If you use this model, please cite the original nanochat project:

```
@misc{nanochat,
  author = {Karpathy, Andrej},
  title = {nanochat: A minimalistic ChatGPT clone},
  year = {2024},
  publisher = {GitHub},
  url = {https://github.com/karpathy/nanochat}
}
```

## License

This model follows the MIT license of the nanochat project.


## Checkpoint Lineage and Retention

This repository is the **MID** checkpoint in the private lineage:

1. [`keithtyser/nanochat-keith`](https://huggingface.co/keithtyser/nanochat-keith) — base checkpoint
2. [`keithtyser/nanochat-keith-mid`](https://huggingface.co/keithtyser/nanochat-keith-mid) — intermediate checkpoint
3. [`keithtyser/nanochat-keith-sft`](https://huggingface.co/keithtyser/nanochat-keith-sft) — supervised-fine-tuned checkpoint

Keep all three while reproducibility, regression comparison, rollback, or training resume matters. Stage-specific data claims in the older card text should be treated as provisional unless they are backed by a dataset manifest and training log; the datasets themselves are not bundled here.