Add model card with architecture details + quick start
Browse files
README.md
ADDED
|
@@ -0,0 +1,169 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
- fr
|
| 6 |
+
- de
|
| 7 |
+
- es
|
| 8 |
+
- it
|
| 9 |
+
- pt
|
| 10 |
+
- nl
|
| 11 |
+
- hi
|
| 12 |
+
pipeline_tag: automatic-speech-recognition
|
| 13 |
+
tags:
|
| 14 |
+
- audio
|
| 15 |
+
- speech-recognition
|
| 16 |
+
- transcription
|
| 17 |
+
- ggml
|
| 18 |
+
- gguf
|
| 19 |
+
- voxtral
|
| 20 |
+
- mistral
|
| 21 |
+
- whisper
|
| 22 |
+
- speech-llm
|
| 23 |
+
- multilingual
|
| 24 |
+
library_name: ggml
|
| 25 |
+
base_model: mistralai/Voxtral-Mini-3B-2507
|
| 26 |
+
---
|
| 27 |
+
|
| 28 |
+
# Voxtral-Mini-3B-2507 β GGUF
|
| 29 |
+
|
| 30 |
+
GGUF / ggml conversions of [`mistralai/Voxtral-Mini-3B-2507`](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507) for use with the `voxtral-main` CLI from **[CrispStrobe/CrispASR](https://github.com/CrispStrobe/CrispASR)**.
|
| 31 |
+
|
| 32 |
+
Voxtral Mini is Mistral's **3B-parameter speech-LLM** β an enhancement of [Ministral 3B](https://mistral.ai/news/ministraux) with state-of-the-art audio input capabilities while retaining best-in-class text performance. It excels at speech transcription, translation, and audio understanding.
|
| 33 |
+
|
| 34 |
+
- **8 languages** (English, French, German, Spanish, Italian, Portuguese, Dutch, Hindi) with automatic language detection
|
| 35 |
+
- **Built-in audio Q&A and summarization** β ask questions about audio content directly
|
| 36 |
+
- **Function calling from voice** β trigger backend functions based on spoken intents
|
| 37 |
+
- **Long-form context** β up to 30 minutes of audio for transcription, 40 minutes for understanding
|
| 38 |
+
- **Natively multilingual** with state-of-the-art WER across the world's most widely used languages
|
| 39 |
+
- **Highly capable at text** β retains the text understanding capabilities of its Ministral 3B backbone
|
| 40 |
+
- **Apache-2.0** licence
|
| 41 |
+
|
| 42 |
+
## Files
|
| 43 |
+
|
| 44 |
+
| File | Size | Notes |
|
| 45 |
+
| --- | ---: | --- |
|
| 46 |
+
| `voxtral-mini-3b-2507-q4_k.gguf` | 2.5 GB | **Q4_K β recommended default** |
|
| 47 |
+
| `voxtral-mini-3b-2507-q8_0.gguf` | 5.0 GB | Q8_0, near-lossless |
|
| 48 |
+
|
| 49 |
+
Both quantisations produce the correct transcript on `samples/jfk.wav`:
|
| 50 |
+
> And so, my fellow Americans, ask not what your country can do for you, ask what you can do for your country.
|
| 51 |
+
|
| 52 |
+
The mel filterbank from `WhisperFeatureExtractor` and the Tekken tokenizer vocab are **baked into the GGUF**, so the C++ runtime computes everything natively β no Python/torch/librosa at inference time.
|
| 53 |
+
|
| 54 |
+
## Quick Start
|
| 55 |
+
|
| 56 |
+
```bash
|
| 57 |
+
# 1. Build the runtime
|
| 58 |
+
git clone https://github.com/CrispStrobe/CrispASR
|
| 59 |
+
cd CrispASR
|
| 60 |
+
cmake -B build -DCMAKE_BUILD_TYPE=Release
|
| 61 |
+
cmake --build build -j$(nproc) --target voxtral-main
|
| 62 |
+
|
| 63 |
+
# 2. Download a quantisation
|
| 64 |
+
huggingface-cli download cstr/voxtral-mini-3b-2507-GGUF \
|
| 65 |
+
voxtral-mini-3b-2507-q4_k.gguf --local-dir .
|
| 66 |
+
|
| 67 |
+
# 3. Transcribe
|
| 68 |
+
./build/bin/voxtral-main \
|
| 69 |
+
-m voxtral-mini-3b-2507-q4_k.gguf \
|
| 70 |
+
-f audio.wav -t 8 -l en
|
| 71 |
+
```
|
| 72 |
+
|
| 73 |
+
Audio must be 16 kHz mono 16-bit PCM WAV. Pre-convert with:
|
| 74 |
+
```bash
|
| 75 |
+
ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav
|
| 76 |
+
```
|
| 77 |
+
|
| 78 |
+
### Language selection
|
| 79 |
+
|
| 80 |
+
Use `-l LANG` with a two-letter code:
|
| 81 |
+
|
| 82 |
+
| Flag | Language |
|
| 83 |
+
| --- | --- |
|
| 84 |
+
| `-l en` | English (default) |
|
| 85 |
+
| `-l de` | German |
|
| 86 |
+
| `-l fr` | French |
|
| 87 |
+
| `-l es` | Spanish |
|
| 88 |
+
| `-l it` | Italian |
|
| 89 |
+
| `-l pt` | Portuguese |
|
| 90 |
+
| `-l nl` | Dutch |
|
| 91 |
+
| `-l hi` | Hindi |
|
| 92 |
+
|
| 93 |
+
## Performance
|
| 94 |
+
|
| 95 |
+
Measured on `samples/jfk.wav` (11 seconds), 4-core CPU:
|
| 96 |
+
|
| 97 |
+
| Variant | Mel | Encoder | Prefill | Decode/tok | **Total** |
|
| 98 |
+
| --- | ---: | ---: | ---: | ---: | ---: |
|
| 99 |
+
| F16 (8.8 GB) | 264 ms | 48.7 s | 78.4 s | 1134 ms | 157 s |
|
| 100 |
+
| **Q4_K (2.5 GB)** | 246 ms | 32.7 s | **30.8 s** | **242 ms** | **70 s** |
|
| 101 |
+
|
| 102 |
+
Q4_K gives a **2.2Γ speedup** over F16 while producing identical transcripts. The 3B model is larger than the Qwen3-ASR 0.6B β for fastest CPU inference on short clips, Qwen3-ASR Q4_K (6.6s for 11s audio) is faster; Voxtral's advantage is the richer capabilities (audio understanding, function calling, text Q&A) and superior multilingual WER.
|
| 103 |
+
|
| 104 |
+
## Architecture
|
| 105 |
+
|
| 106 |
+
Voxtral-Mini-3B is a three-module speech-LLM:
|
| 107 |
+
|
| 108 |
+
| Component | Details |
|
| 109 |
+
| --- | --- |
|
| 110 |
+
| **Audio encoder** | 32-layer Whisper-large-v3 encoder: d=1280, 20 heads, head_dim=64, FFN=5120, 128 mels, **learned absolute positional embedding** (1500, 1280). Conv1d front-end: conv1(128β1280, k=3, **stride=1**, pad=1) + GELU β conv2(1280β1280, k=3, stride=2, pad=1) + GELU. Note: conv1 stride is 1 (not 2 like standard Whisper), so only conv2 does temporal downsampling (2Γ). 3000 mel frames β 3000 β 1500 encoder frames. |
|
| 111 |
+
| **Projector** | Stack-4-frames + 2Γ Linear: reshape (1500, 1280) β (375, 5120), then Linear(5120β3072) β GELU β Linear(3072β3072). 4Γ temporal downsampling: 50 fps Whisper output β **12.5 fps audio embeddings** matching the documented frame rate. |
|
| 112 |
+
| **LLM** | 30-layer Llama 3 / Ministral 3B: d=3072, **32 Q heads / 8 KV heads (GQA, ratio 4)**, head_dim=128, FFN=8192, SwiGLU, RMSNorm, NEOX-style RoPE ΞΈ=1e8, vocab=131072, max_pos=131072. **No biases** anywhere. **No Q-norm/K-norm** (unlike Qwen3-ASR's Qwen3 backbone). |
|
| 113 |
+
| **Tokenizer** | Mistral Tekken (tiktoken-style rank BPE, 150k vocab entries + 1000 special tokens). Stored in the GGUF as a binary blob. |
|
| 114 |
+
| **Audio injection** | `audio_token_id=24` placeholder in the `[INST]` prompt; the LLM input embeddings at those positions get replaced with the projector output frames. |
|
| 115 |
+
| **Parameters** | ~3B total |
|
| 116 |
+
|
| 117 |
+
### Transcription prompt format
|
| 118 |
+
|
| 119 |
+
```
|
| 120 |
+
<s> [INST] [BEGIN_AUDIO] <audio_pad>Γ375 [/INST] lang:en [TRANSCRIBE]
|
| 121 |
+
```
|
| 122 |
+
|
| 123 |
+
Token IDs: `[1, 3, 25, 24Γ375, 4, 9909, 1058, <lang_id>, 34]`
|
| 124 |
+
|
| 125 |
+
### Key differences from standard Whisper
|
| 126 |
+
|
| 127 |
+
1. **Conv1 stride is 1** (Whisper uses stride 2). This means the conv front-end only does 2Γ temporal reduction (just conv2), not 4Γ. 3000 mel frames β 1500 encoder frames (vs Whisper's 750).
|
| 128 |
+
2. **K-proj has no bias** in the encoder's self-attention (Whisper quirk preserved from the Whisper-large-v3 weights).
|
| 129 |
+
3. The encoder output is **not** consumed by a Whisper decoder β it's fed through a 4-frame-stack projector into a general-purpose Llama 3 LLM that generates the transcript (or any other text response) autoregressively.
|
| 130 |
+
|
| 131 |
+
## Implementation notes
|
| 132 |
+
|
| 133 |
+
The C++ runtime is verified against the PyTorch reference (bf16) at every architectural boundary:
|
| 134 |
+
|
| 135 |
+
| Stage | Diff metric | Result |
|
| 136 |
+
| --- | --- | --- |
|
| 137 |
+
| LLM forward (30 layers, text-only) | cosine sim at last position | 0.999973, **top-5 5/5 match** |
|
| 138 |
+
| Audio encoder + projector (32 layers + stack-4) | per-row cosine sim vs `proj2_out.npy` | mean 0.998, min 0.870 (bf16 ref precision) |
|
| 139 |
+
| End-to-end transcription on jfk.wav | generated token sequence | **Correct transcript** |
|
| 140 |
+
|
| 141 |
+
The 0.87 min cosine sim on the encoder is from the bf16 reference precision (7-bit mantissa) vs F16 GGUF weights (10-bit) with F32 compute in C++. An F32 reference would give tighter numbers β the end-to-end transcript is the real correctness test and it passes.
|
| 142 |
+
|
| 143 |
+
### Bugs found during the port
|
| 144 |
+
|
| 145 |
+
1. **`ggml_conv_1d` output layout**: returns `(OL, OC, N)` not `(OC, OL)`. Bias needs `(1, OC, 1)` reshape to broadcast over time+batch.
|
| 146 |
+
2. **Post-conv transpose**: `ggml_conv_1d` puts time on `ne[0]`, but LayerNorm needs feature dim on `ne[0]`. Fixed by reshape+transpose to `(d, T_enc)`.
|
| 147 |
+
3. **GELU approximation**: `ggml_gelu` (tanh approx) β `ggml_gelu_erf` (exact) for correctness matching.
|
| 148 |
+
4. **Tekken vocab blob storage**: gguf-py's `add_array` with Python int lists stores as INT32, corrupting the uint8 byte stream. Fixed by storing as a 1D F32 tensor.
|
| 149 |
+
|
| 150 |
+
## How this was made
|
| 151 |
+
|
| 152 |
+
1. HF safetensors converted to GGUF F16 by [`models/convert-voxtral-to-gguf.py`](https://github.com/CrispStrobe/CrispASR/blob/main/models/convert-voxtral-to-gguf.py). All 765 tensors (762 model + mel_filters + mel_window + Tekken vocab blob) map cleanly.
|
| 153 |
+
2. Quantised variants produced by [`cohere-quantize`](https://github.com/CrispStrobe/CrispASR/blob/main/examples/cohere-main/cohere-quantize.cpp) with the Q4_0 fallback for 1280-wide audio encoder tensors (1280 % 256 β 0 for Q4_K, same situation as Qwen3-ASR).
|
| 154 |
+
3. Inference implemented in [`src/voxtral.{h,cpp}`](https://github.com/CrispStrobe/CrispASR/blob/main/src/voxtral.cpp) (~1300 LOC): encoder and LLM each run as one ggml graph, with a persistent F16 KV cache `(head_dim, max_ctx, n_kv_heads, n_layers)` shared between prefill and per-token decode steps. Flash attention (`ggml_flash_attn_ext`) used on both prefill (F16 causal mask) and decode (no mask) paths.
|
| 155 |
+
|
| 156 |
+
## Related
|
| 157 |
+
|
| 158 |
+
- **Original model**: [`mistralai/Voxtral-Mini-3B-2507`](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507) (Apache-2.0)
|
| 159 |
+
- **C++ runtime**: [CrispStrobe/CrispASR](https://github.com/CrispStrobe/CrispASR)
|
| 160 |
+
- **Research paper**: [arxiv.org/abs/2507.13264](https://arxiv.org/abs/2507.13264)
|
| 161 |
+
- Sister releases in the same family:
|
| 162 |
+
- [`cstr/qwen3-asr-0.6b-GGUF`](https://huggingface.co/cstr/qwen3-asr-0.6b-GGUF) β Qwen3-ASR 0.6B (faster, 30 languages + Chinese dialects)
|
| 163 |
+
- [`cstr/parakeet-tdt-0.6b-v3-GGUF`](https://huggingface.co/cstr/parakeet-tdt-0.6b-v3-GGUF) β Parakeet TDT 600M (free word timestamps)
|
| 164 |
+
- [`cstr/canary-1b-v2-GGUF`](https://huggingface.co/cstr/canary-1b-v2-GGUF) β Canary 978M (speech translation)
|
| 165 |
+
- [`cstr/cohere-transcribe-03-2026-GGUF`](https://huggingface.co/cstr/cohere-transcribe-03-2026-GGUF) β Cohere Transcribe 2B (lowest English WER)
|
| 166 |
+
|
| 167 |
+
## License
|
| 168 |
+
|
| 169 |
+
Apache-2.0, inherited from the base model.
|