TwIL-8B-MLX-slq-4.5bpw

SLQ 4.5 bpw mixed-precision MLX quant of TwIL-8B (balanced variant), an 8.79B Granite reasoning model for math, code, and instruction following.

Model details

Base: webAI-Official/TwIL-8B (balanced), Granite dense decoder, 40 layers, hidden 4096, vocab 100,352
Quantization: SLQ mixed precision, target 4.50 bpw, achieved 4.4979 bpw, group size 128
Quantized tensors: 280 linear projections (7,969,177,600 params). Embeddings and lm_head stay BF16
Size: 5.937 GiB, vs 16.38 GiB BF16 (2.76x smaller)
Calibration: FineWeb-Edu, 16,384 tokens x 2048. Bitwidths measured at 4, 5, 6, 8. The 2- and 3-bit points were dropped after a smoke run showed they were never allocated at this budget
Stage 3 search was heuristic-only. At 100k vocab the full validation loop does not fit in 64 GB, so we validated with zero-shot recovery instead

Bitwidth distribution

Bitwidth	Layers
8-bit	1
6-bit	81
5-bit	114
4-bit	84

Usage

No custom code required. Recent mlx-lm infers per-layer bitwidths from tensor geometry.

from mlx_lm import load, generate

model, tokenizer = load("hermitdave/TwIL-8B-MLX-slq-4.5bpw")
text = generate(model, tokenizer, "Hello, my name is", max_tokens=100)

Requires mlx-lm 0.32.2 or newer. slq_bits in config.json is provenance metadata only, the loader does not read it.

Validation

Zero-shot multiple-choice, BF16 vs this checkpoint, 100 examples per task.

Task	BF16	SLQ	Recovery
arc_challenge	0.5100	0.5200	1.0196
arc_easy	0.8000	0.7500	0.9375
hellaswag	0.4900	0.4800	0.9796
winogrande	0.5500	0.5700	1.0364
mmlu	0.4400	0.5000	1.1364
Mean			1.0219

Task-lossless threshold is recovery >= 0.99. This checkpoint clears it on these generic tasks.

Performance

Tokens per second, same prompt (The capital of France is), 128 generated tokens, stock mlx-lm, M3 Max 64 GB.

Model	TPS
BF16	19.3
SLQ 4.5 bpw	53.9

2.79x faster decode, consistent with the 2.76x size reduction.

Two honest caveats. The sample is 100 per task, not the full sets, so the confidence intervals are wide. And these tasks do not measure what TwIL was built for. Math and code reasoning were not benchmarked here, so treat the headline number as a fidelity check, not a claim about contest performance.

The source model carries the webAI Non-Commercial License, and this quant inherits it. Local use is fine; do not use it commercially.

Methodology

SLQ (Statistically Lossless Quantization) uses Hessian-weighted per-layer error estimation to assign bitwidths: GPTQ quantize each layer at candidate bitwidths, build a relative-error database, greedily allocate bits against the bpw budget, then pack into MLX QuantizedLinear tensors.