DedeProGames/Kiyo-135M-0960

Kiyo-135M-0960 is an updated checkpoint of Kiyo-135M, continuing pretraining from 200B to 1,000,000,000,000 (1T) tokens on the same data mixture: FineWeb-Edu, DCLM-Baseline, FineMath, and Stack-v3-train.

The model follows the SmolLM2-135M architecture: a Llama-style decoder with grouped query attention, RMSNorm, SwiGLU MLPs, and tied input/output embeddings. SmolLM2 was chosen as a base architecture because it is specifically tuned for small-scale pretraining efficiency — narrow hidden size with a comparatively deep layer stack, and a large 49k-token vocabulary that keeps sequence lengths short for a model this size. Kiyo-135M-0960 uses the same architecture and initialization lineage as Kiyo-135M, extended with additional pretraining on the same data mixture.

Model Details

Field	Value
Parameters	134,515,008
Architecture	Llama-style decoder (SmolLM2 architecture)
Layers	30
Hidden size	576
Intermediate size	1,536
Attention heads	9
KV heads	3
Attention type	Grouped query attention
Activation	SwiGLU
Normalization	RMSNorm
Positional encoding	RoPE (theta 100,000)
Vocabulary size	49,152
Context length	8,192
Embeddings	Tied input/output embeddings
Training tokens	1,000,000,000,000
Weight format	safetensors

Training Data

Source	Domain
FineWeb-Edu	General web text, education-filtered
DCLM-Baseline	General web text, high-quality filtered
FineMath	Mathematical reasoning
Stack-v3-train	Source code

Benchmarks

Self-reported results from the official BananaMind Base Bench 1.1 script, all measured with the same runner, dtype (bfloat16) and GPU.

Model	Params	Overall Elo
Kiyo-135M-0960	134.5M	1,139
Kiyo-135M	134.5M	1,126
BananaMind-2-Pro	139.0M	1,124
Rose-Pro	151.3M	1,105
GPT-2	124M	990

Figures for Kiyo-135M, BananaMind-2-Pro, Rose-Pro, and GPT-2 are as self-reported on their own model cards, all against the same BananaMind Base Bench 1.1 suite.

Detailed Kiyo-135M-0960 result

Category	Accuracy	z vs. chance	Elo	Significant
Language completion	98.0%	+11.92	1,468	*
Code completion	88.0%	+10.29	1,453	*
World knowledge	82.0%	+9.31	1,177	*
Commonsense	80.0%	+8.98	1,178	*
Logical reasoning	50.0%	+4.08	1,061	*
Context tracking	52.0%	+4.41	1,002	*
Quantitative	28.0%	+0.49	889

* = passes 1.96σ vs. chance; n=50 per category

By difficulty

Difficulty	Accuracy
Easy	76.1%
Medium	65.0%
Hard	63.8%

Summary

Metric	Value
Parameters	134,515,008
Overall Elo	1,139
Chance floor	805
Above chance floor	+334
Raw accuracy	68.3%

Scores are self-evaluated and may vary with the benchmark revision, Transformers version, dtype, hardware, and generation settings.

Usage

pip install -U transformers safetensors torch

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "DedeProGames/Kiyo-135M-0960"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,
).cuda().eval()

prompt = "The meaning of life is "
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)

with torch.no_grad():
    output = model.generate(
        input_ids=input_ids,
        max_new_tokens=64,
        do_sample=False,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

This is a base model, not instruction-tuned — it continues text rather than following instructions. At 135M parameters it produces fluent, well-structured text and is strong on language completion and code, but accuracy drops on quantitative and context-tracking tasks. It can generate incorrect facts and should not be used for high-stakes decisions without verification. Keep a finite generation limit to avoid repetition or drift on long outputs.

License

Apache 2.0