cstr commited on
Commit
7a6ffdc
Β·
verified Β·
1 Parent(s): fc47e59

Add model card with architecture details + quick start

Browse files
Files changed (1) hide show
  1. README.md +169 -0
README.md ADDED
@@ -0,0 +1,169 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - en
5
+ - fr
6
+ - de
7
+ - es
8
+ - it
9
+ - pt
10
+ - nl
11
+ - hi
12
+ pipeline_tag: automatic-speech-recognition
13
+ tags:
14
+ - audio
15
+ - speech-recognition
16
+ - transcription
17
+ - ggml
18
+ - gguf
19
+ - voxtral
20
+ - mistral
21
+ - whisper
22
+ - speech-llm
23
+ - multilingual
24
+ library_name: ggml
25
+ base_model: mistralai/Voxtral-Mini-3B-2507
26
+ ---
27
+
28
+ # Voxtral-Mini-3B-2507 β€” GGUF
29
+
30
+ GGUF / ggml conversions of [`mistralai/Voxtral-Mini-3B-2507`](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507) for use with the `voxtral-main` CLI from **[CrispStrobe/CrispASR](https://github.com/CrispStrobe/CrispASR)**.
31
+
32
+ Voxtral Mini is Mistral's **3B-parameter speech-LLM** β€” an enhancement of [Ministral 3B](https://mistral.ai/news/ministraux) with state-of-the-art audio input capabilities while retaining best-in-class text performance. It excels at speech transcription, translation, and audio understanding.
33
+
34
+ - **8 languages** (English, French, German, Spanish, Italian, Portuguese, Dutch, Hindi) with automatic language detection
35
+ - **Built-in audio Q&A and summarization** β€” ask questions about audio content directly
36
+ - **Function calling from voice** β€” trigger backend functions based on spoken intents
37
+ - **Long-form context** β€” up to 30 minutes of audio for transcription, 40 minutes for understanding
38
+ - **Natively multilingual** with state-of-the-art WER across the world's most widely used languages
39
+ - **Highly capable at text** β€” retains the text understanding capabilities of its Ministral 3B backbone
40
+ - **Apache-2.0** licence
41
+
42
+ ## Files
43
+
44
+ | File | Size | Notes |
45
+ | --- | ---: | --- |
46
+ | `voxtral-mini-3b-2507-q4_k.gguf` | 2.5 GB | **Q4_K β€” recommended default** |
47
+ | `voxtral-mini-3b-2507-q8_0.gguf` | 5.0 GB | Q8_0, near-lossless |
48
+
49
+ Both quantisations produce the correct transcript on `samples/jfk.wav`:
50
+ > And so, my fellow Americans, ask not what your country can do for you, ask what you can do for your country.
51
+
52
+ The mel filterbank from `WhisperFeatureExtractor` and the Tekken tokenizer vocab are **baked into the GGUF**, so the C++ runtime computes everything natively β€” no Python/torch/librosa at inference time.
53
+
54
+ ## Quick Start
55
+
56
+ ```bash
57
+ # 1. Build the runtime
58
+ git clone https://github.com/CrispStrobe/CrispASR
59
+ cd CrispASR
60
+ cmake -B build -DCMAKE_BUILD_TYPE=Release
61
+ cmake --build build -j$(nproc) --target voxtral-main
62
+
63
+ # 2. Download a quantisation
64
+ huggingface-cli download cstr/voxtral-mini-3b-2507-GGUF \
65
+ voxtral-mini-3b-2507-q4_k.gguf --local-dir .
66
+
67
+ # 3. Transcribe
68
+ ./build/bin/voxtral-main \
69
+ -m voxtral-mini-3b-2507-q4_k.gguf \
70
+ -f audio.wav -t 8 -l en
71
+ ```
72
+
73
+ Audio must be 16 kHz mono 16-bit PCM WAV. Pre-convert with:
74
+ ```bash
75
+ ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav
76
+ ```
77
+
78
+ ### Language selection
79
+
80
+ Use `-l LANG` with a two-letter code:
81
+
82
+ | Flag | Language |
83
+ | --- | --- |
84
+ | `-l en` | English (default) |
85
+ | `-l de` | German |
86
+ | `-l fr` | French |
87
+ | `-l es` | Spanish |
88
+ | `-l it` | Italian |
89
+ | `-l pt` | Portuguese |
90
+ | `-l nl` | Dutch |
91
+ | `-l hi` | Hindi |
92
+
93
+ ## Performance
94
+
95
+ Measured on `samples/jfk.wav` (11 seconds), 4-core CPU:
96
+
97
+ | Variant | Mel | Encoder | Prefill | Decode/tok | **Total** |
98
+ | --- | ---: | ---: | ---: | ---: | ---: |
99
+ | F16 (8.8 GB) | 264 ms | 48.7 s | 78.4 s | 1134 ms | 157 s |
100
+ | **Q4_K (2.5 GB)** | 246 ms | 32.7 s | **30.8 s** | **242 ms** | **70 s** |
101
+
102
+ Q4_K gives a **2.2Γ— speedup** over F16 while producing identical transcripts. The 3B model is larger than the Qwen3-ASR 0.6B β€” for fastest CPU inference on short clips, Qwen3-ASR Q4_K (6.6s for 11s audio) is faster; Voxtral's advantage is the richer capabilities (audio understanding, function calling, text Q&A) and superior multilingual WER.
103
+
104
+ ## Architecture
105
+
106
+ Voxtral-Mini-3B is a three-module speech-LLM:
107
+
108
+ | Component | Details |
109
+ | --- | --- |
110
+ | **Audio encoder** | 32-layer Whisper-large-v3 encoder: d=1280, 20 heads, head_dim=64, FFN=5120, 128 mels, **learned absolute positional embedding** (1500, 1280). Conv1d front-end: conv1(128β†’1280, k=3, **stride=1**, pad=1) + GELU β†’ conv2(1280β†’1280, k=3, stride=2, pad=1) + GELU. Note: conv1 stride is 1 (not 2 like standard Whisper), so only conv2 does temporal downsampling (2Γ—). 3000 mel frames β†’ 3000 β†’ 1500 encoder frames. |
111
+ | **Projector** | Stack-4-frames + 2Γ— Linear: reshape (1500, 1280) β†’ (375, 5120), then Linear(5120β†’3072) β†’ GELU β†’ Linear(3072β†’3072). 4Γ— temporal downsampling: 50 fps Whisper output β†’ **12.5 fps audio embeddings** matching the documented frame rate. |
112
+ | **LLM** | 30-layer Llama 3 / Ministral 3B: d=3072, **32 Q heads / 8 KV heads (GQA, ratio 4)**, head_dim=128, FFN=8192, SwiGLU, RMSNorm, NEOX-style RoPE ΞΈ=1e8, vocab=131072, max_pos=131072. **No biases** anywhere. **No Q-norm/K-norm** (unlike Qwen3-ASR's Qwen3 backbone). |
113
+ | **Tokenizer** | Mistral Tekken (tiktoken-style rank BPE, 150k vocab entries + 1000 special tokens). Stored in the GGUF as a binary blob. |
114
+ | **Audio injection** | `audio_token_id=24` placeholder in the `[INST]` prompt; the LLM input embeddings at those positions get replaced with the projector output frames. |
115
+ | **Parameters** | ~3B total |
116
+
117
+ ### Transcription prompt format
118
+
119
+ ```
120
+ <s> [INST] [BEGIN_AUDIO] <audio_pad>Γ—375 [/INST] lang:en [TRANSCRIBE]
121
+ ```
122
+
123
+ Token IDs: `[1, 3, 25, 24Γ—375, 4, 9909, 1058, <lang_id>, 34]`
124
+
125
+ ### Key differences from standard Whisper
126
+
127
+ 1. **Conv1 stride is 1** (Whisper uses stride 2). This means the conv front-end only does 2Γ— temporal reduction (just conv2), not 4Γ—. 3000 mel frames β†’ 1500 encoder frames (vs Whisper's 750).
128
+ 2. **K-proj has no bias** in the encoder's self-attention (Whisper quirk preserved from the Whisper-large-v3 weights).
129
+ 3. The encoder output is **not** consumed by a Whisper decoder β€” it's fed through a 4-frame-stack projector into a general-purpose Llama 3 LLM that generates the transcript (or any other text response) autoregressively.
130
+
131
+ ## Implementation notes
132
+
133
+ The C++ runtime is verified against the PyTorch reference (bf16) at every architectural boundary:
134
+
135
+ | Stage | Diff metric | Result |
136
+ | --- | --- | --- |
137
+ | LLM forward (30 layers, text-only) | cosine sim at last position | 0.999973, **top-5 5/5 match** |
138
+ | Audio encoder + projector (32 layers + stack-4) | per-row cosine sim vs `proj2_out.npy` | mean 0.998, min 0.870 (bf16 ref precision) |
139
+ | End-to-end transcription on jfk.wav | generated token sequence | **Correct transcript** |
140
+
141
+ The 0.87 min cosine sim on the encoder is from the bf16 reference precision (7-bit mantissa) vs F16 GGUF weights (10-bit) with F32 compute in C++. An F32 reference would give tighter numbers β€” the end-to-end transcript is the real correctness test and it passes.
142
+
143
+ ### Bugs found during the port
144
+
145
+ 1. **`ggml_conv_1d` output layout**: returns `(OL, OC, N)` not `(OC, OL)`. Bias needs `(1, OC, 1)` reshape to broadcast over time+batch.
146
+ 2. **Post-conv transpose**: `ggml_conv_1d` puts time on `ne[0]`, but LayerNorm needs feature dim on `ne[0]`. Fixed by reshape+transpose to `(d, T_enc)`.
147
+ 3. **GELU approximation**: `ggml_gelu` (tanh approx) β†’ `ggml_gelu_erf` (exact) for correctness matching.
148
+ 4. **Tekken vocab blob storage**: gguf-py's `add_array` with Python int lists stores as INT32, corrupting the uint8 byte stream. Fixed by storing as a 1D F32 tensor.
149
+
150
+ ## How this was made
151
+
152
+ 1. HF safetensors converted to GGUF F16 by [`models/convert-voxtral-to-gguf.py`](https://github.com/CrispStrobe/CrispASR/blob/main/models/convert-voxtral-to-gguf.py). All 765 tensors (762 model + mel_filters + mel_window + Tekken vocab blob) map cleanly.
153
+ 2. Quantised variants produced by [`cohere-quantize`](https://github.com/CrispStrobe/CrispASR/blob/main/examples/cohere-main/cohere-quantize.cpp) with the Q4_0 fallback for 1280-wide audio encoder tensors (1280 % 256 β‰  0 for Q4_K, same situation as Qwen3-ASR).
154
+ 3. Inference implemented in [`src/voxtral.{h,cpp}`](https://github.com/CrispStrobe/CrispASR/blob/main/src/voxtral.cpp) (~1300 LOC): encoder and LLM each run as one ggml graph, with a persistent F16 KV cache `(head_dim, max_ctx, n_kv_heads, n_layers)` shared between prefill and per-token decode steps. Flash attention (`ggml_flash_attn_ext`) used on both prefill (F16 causal mask) and decode (no mask) paths.
155
+
156
+ ## Related
157
+
158
+ - **Original model**: [`mistralai/Voxtral-Mini-3B-2507`](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507) (Apache-2.0)
159
+ - **C++ runtime**: [CrispStrobe/CrispASR](https://github.com/CrispStrobe/CrispASR)
160
+ - **Research paper**: [arxiv.org/abs/2507.13264](https://arxiv.org/abs/2507.13264)
161
+ - Sister releases in the same family:
162
+ - [`cstr/qwen3-asr-0.6b-GGUF`](https://huggingface.co/cstr/qwen3-asr-0.6b-GGUF) β€” Qwen3-ASR 0.6B (faster, 30 languages + Chinese dialects)
163
+ - [`cstr/parakeet-tdt-0.6b-v3-GGUF`](https://huggingface.co/cstr/parakeet-tdt-0.6b-v3-GGUF) β€” Parakeet TDT 600M (free word timestamps)
164
+ - [`cstr/canary-1b-v2-GGUF`](https://huggingface.co/cstr/canary-1b-v2-GGUF) β€” Canary 978M (speech translation)
165
+ - [`cstr/cohere-transcribe-03-2026-GGUF`](https://huggingface.co/cstr/cohere-transcribe-03-2026-GGUF) β€” Cohere Transcribe 2B (lowest English WER)
166
+
167
+ ## License
168
+
169
+ Apache-2.0, inherited from the base model.