---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- brujula
- causal-lm
- mla
- rope
- muon
- consumer-gpu
- model-growth
- gstack
datasets:
- HuggingFaceFW/fineweb-edu
---

# Brújula-18M (G_stack)

A **18M-parameter** decoder-only LM created by **depth-growing [Brújula-15M](https://huggingface.co/Sakatepon/Brujula-15M)**
with the **G_stack** operator (Du et al., *"Stacking Your Transformers"*, NeurIPS 2024): copy
the trained 15M's 4 layers into an **8-layer** model, then continue pre-training. The whole
thing runs on a single consumer GPU (Intel Arc B580). *Brújula* ("compass" in Spanish) uses a
minimal DeepSeek-style architecture (MLA + RoPE + SquaredReLU, tied embeddings, Muon).

The result: **depth-doubling the 15M champion nearly halved perplexity** on both metrics, for
~3h of extra local compute.

## Results

Perplexity (lower is better), fixed local harness at context length 1024:

| Model | FineWeb-Edu val PPL | WikiText-103 PPL |
|---|---|---|
| Brújula-15M (the base, 4 layers) | 78.05 | 190.74 |
| **Brújula-18M (this model, 8 layers)** | **46.26** | **108.72** |
| **improvement** | **−41%** | **−43%** |

Honest note: the gain comes from **added depth + warm-start** (the grown model effectively saw
~2× the cumulative tokens of the base), not depth alone — no from-scratch-8-layer control was
run. Either way, it's the best sub-50M model in the family.

### The Brújula family

| Model | Params | FineWeb val | WikiText | Notes |
|---|---|---|---|---|
| [Brújula-15M](https://huggingface.co/Sakatepon/Brujula-15M) | 15.5M | 78.05 | 190.74 | tiny champion, from scratch on one Arc B580 |
| **Brújula-18M** | **18M** | **46.26** | **108.72** | this model — Brújula-15M G_stack-grown (4→8 layers) |
| [Brújula-150M](https://huggingface.co/Sakatepon/Brujula-150M) | 153.6M | 21.44 | 36.08 | the flagship |

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Sakatepon/Brujula-18M"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True).eval()

ids = tok("The mitochondria is the", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=64, do_sample=True, temperature=0.8, top_p=0.95, repetition_penalty=1.2)
print(tok.decode(out[0], skip_special_tokens=True))
```

Small base model — use **sampling** + a **continuation cue**; greedy tends to repetition-loop.

## Architecture

| | |
|---|---|
| Type | decoder-only, causal LM |
| Hidden / Layers / Heads | `n_embd=256` / `n_layer=8` (grown from 4 via G_stack) / `n_head=4` |
| Context length | 1024 |
| Attention | Multi-head Latent Attention (MLA), kv-compress 32 / q-compress 64 |
| Position / FFN / Norm | RoPE / SquaredReLU / RMSNorm (pre-norm), tied embeddings |
| Vocab | 50257 (GPT-2 BPE) |
| Unique params | 18.0M |

## How it was made (G_stack)

1. Train Brújula-15M from scratch (4 layers).
2. **Stack**: copy the 4 trained layers into an 8-layer model (`[0,1,2,3] → [0,1,2,3,0,1,2,3]`); keep the embedding / final-norm / tied head.
3. **Continue pre-training** 1 epoch on FineWeb-Edu (~1.4B tokens), batch 32, peak LR 1.2e-3, bf16, ~3h14m on one Intel Arc B580.

The post-stack loss spikes (copy-init isn't function-preserving), then recovers and surpasses
the 15M base within the first ~12% of the epoch — consistent with G_stack's finding that
violating function preservation is fine and even preferable.

## Limitations

- Base completion model — not instruction-tuned, no safety tuning.
- English only, educational-web distribution (FineWeb-Edu); weaker out-of-distribution.
- ~18M params: plausible prose, unreliable facts; best on cued, definitional prompts.
- Short context (1024); no KV-cache in this reference implementation.

## License & attribution

- **Model + code:** Apache-2.0. **Training data:** FineWeb-Edu ([ODC-BY](https://opendatacommons.org/licenses/by/1-0/)).
- Methods: G_stack (Du et al., 2024), DeepSeek-V2 (MLA), Muon, Primer (SquaredReLU), GPT-2 (BPE).