bgub/maccy-96m-16k-base

Maccy 96M (61M active), 16K vocabulary

Maccy is a compact, from-scratch base language model trained on Apple Silicon. It has 96,359,193 total parameters and activates approximately 60,527,385 parameters per token through top-2 routing across 4 SwiGLU experts.

This is a base completion model, not a chat or instruction-following model.

Architecture

Property	Value
Total parameters	96.4M
Active parameters per token	60.5M
Layers	12
Model width	576
Sequence mixers	9 KDA, 3 MLA
Channel mixers	4-expert sparse MoE, top-2 routing
Context length	1,024 tokens
Vocabulary	16,000 byte-level BPE tokens

Maccy combines Kimi Delta Attention (KDA), Multi-head Latent Attention (MLA), and a sparse mixture of experts. Input and output embeddings are tied.

Usage

The repository includes a portable Transformers reference implementation built for Transformers 5.14 or newer. Because Maccy is a custom architecture, loading the model requires trust_remote_code=True.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "bgub/maccy-96m-16k-base"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.float32,
)

inputs = tokenizer("Once upon a time", return_tensors="pt")
output = model.generate(
    **inputs,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.8,
    top_k=50,
    use_cache=False,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

For the optimized Apple-Silicon kernels and training code, see https://github.com/bgub/mokka.

Training

• Training data: Karpathy's shuffled ClimbMix repack, derived from NVIDIA Nemotron-ClimbMix
• Inherited pretraining: 2,120,089,600 tokens with the original vocabulary
• Vocabulary-recovery pretraining: 196,608,000 tokens
• Total parameter exposure: 2,316,697,600 tokens
• Recovery optimizer steps: 6,000
• Training context: 1,024 tokens
• Effective batch: 32 sequences / 32,768 tokens per optimizer step
• Precision: bfloat16 activations with float32 master weights

The 16K tokenizer retains the first 15,991 mergeable tokens from Maccy's original 32K byte-level BPE, then places the nine special tokens directly after them. This preserves all retained ordinary-token IDs and complete UTF-8 byte fallback. The resized model was recovered with continued pretraining on the same corpus.

NVIDIA's source dataset card designates ClimbMix for research and development under CC BY-NC 4.0. Users are responsible for reviewing both the source-dataset terms and this model's license before use.

Evaluation

The table below recomputes bits per byte (BPB) with the same harness and 256 KiB of UTF-8 text per corpus. Models use their native tokenizers, float32 inference, a common 1,024-token context, and a 512-token sliding stride. Lower is better.

Model	Parameters (total / active)	Pretraining tokens	ClimbMix validation	WikiText-103 test	enwik8 test	FineWeb-Edu sample
Maccy 96M 16K (this model)	96.4M / 60.5M active	2.32B	0.9982	1.3538	1.5222	1.1068
Maccy 106M 32K	106.0M / 70.2M active	2.12B	0.9916	1.3377	1.5251	1.0979
NanoWhale 100M	110.4M / 100.5M active	2.6B	1.2516	1.4063	1.8468	1.1641
Pythia 70M	70.4M / 70.4M active	299.9B	1.0815	1.2441	1.2157	1.1168
GPT-2 Small	124.4M / 124.4M active	Not disclosed	0.9668	1.0499	1.1907	0.9897
SmolLM2 135M	134.5M / 134.5M active	2T	0.8119	0.9287	0.8482	0.8421

ClimbMix favors Maccy. FineWeb-Edu favors NanoWhale and SmolLM2 and is not claimed to be held out from them. Possible WikiText-103 and enwik8 overlap for public reference models is unknown. Training budgets differ enormously, so this is a checkpoint comparison, not a controlled architecture comparison. Corpus hashes, model revisions, loss sums, and the reproduction script are in the source repository under benchmarks/results/reference-bpb-v1.

Limitations

• This checkpoint has not been post-trained for conversation or instruction following.
• The 1,024-token context is short by modern standards.
• Code, mathematics, factual reliability, and long-form coherence are limited.
• The portable Transformers implementation does not yet include a recurrent generation cache and is slower than Mokka's native Metal implementation.
• Training data may contain errors, biases, and objectionable material that the model can reproduce.