primitive-ai/Qwen3.8-Flash-Next-NVFP4

The 180B Flash-Next on one GPU.

Qwen3.8-Flash-Next is 360 GB in BF16 and needs two data-center GPUs in FP8.
This build serves it on one 96 GB Blackwell: 88.8 GiB of VRAM, the 51B n-gram table in host RAM, and no patched runtime.

---

Why this quant

• 🖥️ One 96 GB GPU. 88,828 MiB of VRAM at serve: NVFP4 experts plus the BF16 tail. The n-gram table lives in host RAM (~100 GB) with async prefetch. No other public build of this model can use the offload path at all.
• 🎯 92.2 knowledge on a 1,170-item, 9-benchmark suite (n=2 runs: 92.1, 92.2), 99.4% finished, zero request errors. GSM8K 98.0, MMLU-Pro 90.0.
• 🔧 84.6 call / 56.7 abstain on the 200-item tool-calling suite (n=3). It makes correct calls at the same rate as much smaller strong models and never declines to call — the abstention items are what pull a pooled number down, which is why both halves are printed.
• ⚡ 74.4 tok/s single-stream (12.2 ms/token), 483.8 tok/s at concurrency 32, measured prefix-cache-free with distinct seeds; the two seeds agreed within 0.04%.
• 🔀 MTP speculative decoding preserved. All 31 MTP tensors byte-identical to the source; {"method":"mtp","num_speculative_tokens":3} works as Qwen documents.
• 🧩 Stock image, no patches. vllm/vllm-openai:qwen38-flash-next exactly as published. The serve block below carries the two flags that make single-GPU work — without them the server hangs silently or times out.

---

Serve it

docker run --gpus all --ipc=host -p 8000:8000 \
  -e VLLM_PLE_CPU_OFFLOAD=1 -e VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800 \
  vllm/vllm-openai:qwen38-flash-next \
  --model primitive-ai/Qwen3.8-Flash-Next-NVFP4 \
  --distributed-executor-backend mp \
  --gpu-memory-utilization 0.92 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Two flags are load-bearing on a single GPU. --distributed-executor-backend mp: the default single-GPU executor never starts the n-gram offload worker, and the first forward waits on it forever — the server looks healthy and hangs. VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800: the worker loads a 95 GB table before serving and the 600 s default can expire first. Host needs about 100 GB of free RAM. The reasoning parser and tool-call parser in this block are validated on this exact checkpoint: thinking lands in reasoning with no markup in content, and tool calls arrive as structured tool_calls with valid JSON arguments and finish_reason: tool_calls.

---

Not enough host RAM? Put the table on NVMe

The serve command above wants ~100 GB of free host RAM for the n-gram table. With fast local storage you can skip that: this repo ships a one-file overlay (worker_image_disk.py) that maps the table from a file instead. First boot writes 95.4 GB into the store directory; every later boot maps it instantly and skips the table's checkpoint reads.

hf download primitive-ai/Qwen3.8-Flash-Next-NVFP4 worker_image_disk.py --local-dir .
mkdir -p pledisk_store

docker run --gpus all --ipc=host -p 8000:8000 \
  -v $PWD/worker_image_disk.py:/usr/local/lib/python3.12/dist-packages/vllm/v1/ple_offload/worker.py:ro \
  -v $PWD/pledisk_store:/pledisk_store \
  -e VLLM_PLE_DISK_OFFLOAD_DIR=/pledisk_store \
  -e VLLM_PLE_CPU_OFFLOAD=1 -e VLLM_PLE_OFFLOAD_READY_TIMEOUT=3600 \
  vllm/vllm-openai:qwen38-flash-next \
  --model primitive-ai/Qwen3.8-Flash-Next-NVFP4 \
  --distributed-executor-backend mp \
  --gpu-memory-utilization 0.92 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Measured on the mixed sibling — both repos ship byte-identical BF16 tables and the same worker path, so the disk behavior transfers; absolute tok/s columns are the sibling's. 8K in / 512 out, prefix-cache-free, two seeds per cell (shown a / b):

config	boot	tok/s @ 1	tok/s @ 32	median TTFT @ 1
table in RAM (command above)	302 s	84.5 / 84.4	516.8 / 523.6	569 / 573 ms
disk, container capped to 48 GB RAM — recommended	263 s	79.4 / 76.8	427.0 / 435.8	571 / 573 ms
disk, uncapped 176 GB host	303–344 s	50.4–62.1	196.7–396.7	1.8–2.9 s
disk, cold page cache	404 s	40.8 / 37.0	134.2 / 290.3	4.6 / 5.2 s
disk, first boot (writes the file)	504 s	—	—	—

Net cost of the disk path, run capped: −8% single-stream, −17% at concurrency 32, TTFT parity with the RAM baseline. The counterintuitive row is the uncapped one, and it reproduces across two boots and four seeds: without a container memory cap, the boot's own 172 GB checkpoint streaming flows through the global page cache and evicts the table it is about to need, so gathers fault back to NVMe mid-decode. A memory cap makes reclaim cgroup-local — the container's checkpoint reads can only evict the container's own cache, and the table stays resident. So on the disk path, always cap the serving container (48 GB is what we validated; --memory 48g --memory-swap 48g).

Accuracy is unaffected — the mapping serves the same bytes. Inside the 48 GB cap the 200-item tool-calling suite scored 78.5 with zero request errors and zero truncations (repeat spread on this suite: 78.0–80.5), and the generation-sanity gate passed on the first-boot and capped configurations. Cold cache is a floor, not a steady state: the two cold @ 32 runs went 134 → 290 tok/s back to back as the cache refilled. Boot times share one caveat: all were measured with the checkpoint at least partially page-cache-resident; a truly cold first read of the 172 GB weights adds its own disk time to any of them.

The overlay targets this exact image. The same change is a draft PR to vLLM — vllm-project/vllm#54070, branch feat/ple-disk-offload — stacked on the PLE CPU-offload PR (vllm-project/vllm#53899).

---

Quantized PLE tables: 49 / 32 / 28.8 GB instead of 95 GB

The table itself also quantizes well. We publish it in FP8 per-row (49 GB), INT4 group-16 (32 GB), and NVFP4-style e2m1 group-16 (28.8 GB), served memory-mapped by a two-file overlay — host RAM cost becomes page cache only, no container cap needed. Accuracy holds on both suites for all three (knowledge 92.2 / 92.9 / 92.2 vs 92.2 BF16; tool-calling n=3 means 77.7 / 78.2 / 78.7 vs 79.2, one ±1.5 band), throughput lands within 5–6% of the in-RAM BF16 baseline, and MTP keeps most of its speed-up (129.6 tok/s single-stream with the INT4 table, 128.8 with NVFP4, vs 142.6 in-RAM). Validated end to end inside a 48 GB container: tool-calling 80.5, 79.4 / 486 tok/s — a 64 GB-RAM host serves this model. Tables, overlay files, serve command, format spec, and the full measurement table: primitive-ai/Qwen3.8-Flash-Next-PLE-quant.

---

Speculative decoding (MTP)

The MTP tensors are preserved byte-identical, so vLLM's built-in draft path works — add:

  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Real-prompt A/B (measured on the mixed sibling), single stream, thinking on (60-item subset of our eval, decode rate = output tokens over wall time — not comparable to the bench-serve numbers above, and measured on real prompts because random-token benches overstate speculative gains):

speculative config	decode tok/s	strict score
none	91.2	91.7
num_speculative_tokens: 1	does not boot	—
num_speculative_tokens: 2	133.2	93.3
num_speculative_tokens: 3	142.6 (+56%)	93.3
num_speculative_tokens: 3, table on NVMe	77.5–82.3	93.3

Score differences are single-run noise on 60 items; the speed difference reproduces. num_speculative_tokens: 1 hangs this image at startup — the engine core blocks in shm_broadcast before the API server binds, reproduced on three boots (one waited 80 minutes). Use 3.

---

The other 4-bit quants, measured on the same box

Every serious vLLM-lane quant of this model is experts-only 4-bit. We ran the four public ones under our exact protocol (one RTX PRO 6000, 200-item tool-calling suite, cache-free 8K/512 bench, table in host RAM):

build	expert format	tok/s @ 1	tok/s @ 32	TTFT @ 1	tool-calling
our mixed build	NVFP4 g16 + FP8 attention	84.4	520	570 ms	79.2 (n=3)
Intel AutoRound	INT4 g128 sym, iters=200	82.6	482	683 ms	80.5 (n=1)
lvkaokao RTN	INT4 g128 sym, RTN	82.4	481	683 ms	80.0 (n=1)
cyankiwi AWQ	INT4 g32 asym, AWQ	81.6	422	680 ms	78.0 (n=1)
wtdcode AWQ	INT4 g128 sym, AWQ	82.6	484	682 ms	76.0 (n=1)

Single tool-calling runs carry a ±1.5 spread, so the middle of this column is one band; the AWQ-vs-RTN-family gap at the edges is larger than that. Two patterns hold regardless: every 4-bit expert format decodes at ~82 tok/s single-stream with ~680 ms TTFT — the speed difference in the top row is the FP8 attention, not the expert format — and the W4A16 builds run on pre-Blackwell GPUs, which NVFP4 does not serve natively. All four keep the n-gram table in BF16, so the quantized-table sidecars apply to them as well.

MTP and the disk-backed BF16 table do not combine well: speculation multiplies table-gather traffic, the working set outgrows the page cache, and the +56% collapses to roughly the no-speculation rate (capped 77.5, uncapped 82.3). With the BF16 table in RAM, use MTP; on a low-RAM host, pair MTP with the INT4 quantized table instead, which keeps 129.6 tok/s.

---

Measured

1,370 items across fourteen public benchmarks. A 1,170-item knowledge suite (MMLU-Pro, ARC-Challenge, HellaSwag, WinoGrande, CommonsenseQA, BoolQ, OpenBookQA, GSM8K, MATH-500) and a 200-item tool-calling suite (BFCL v4, xLAM/APIGen, ToolACE, Glaive v2, nvidia When2Call), under one fixed protocol: temperature 0.6 / top_p 0.95 / top_k 20, thinking forced on, a 16,384-token budget, no reasoning parser, the last ANSWER: scored. Concurrency 32 on the same single RTX PRO 6000 Blackwell the fit numbers come from. Auto-scored, no LLM judge.

build	size	overall	knowledge	call	abstain	runs k/a	finished	out/answer	tok/s @ 32	tok/s @ 1
this repo	186 GB	90.2	92.2	84.6	56.7	2/3	99.4%	664 tok	483.8	74.4

overall is one number over both suites: the 1,170 knowledge and 200 tool-calling items pooled as 1,370, weighted 85.4% and 14.6% by item count. call is accuracy on the 160 tool-calling items that require a call; abstain is the 40 whose correct action is to call nothing — they are never pooled into one number on our cards, because a model can be strong at one and weak at the other, and this one is exactly that: solid call accuracy, zero abstentions in the system-prompt tool format. Through the native tools= API it does abstain (validated above), so weight the two columns by how your application passes tools.

Same-box numbers for the community W4A16 builds are in the comparison table further up; the section below covers what single-GPU serving requires of a checkpoint.

Knowledge is a mean of 2 runs (92.1, 92.2); tool-calling of 3 (78.5, 80.0, 78.5 pooled). Throughput is prefix-cache-free — --random-prefix-len 0, a distinct seed per run, warm-up seeded apart — and two seeds agreed within 0.04%. On our other models this suite's repeat spread runs to ±0.5 on knowledge and ±1.5 on tool calling; treat gaps inside that as ties.

---

Single-GPU serving: what a checkpoint needs

VLLM_PLE_CPU_OFFLOAD builds the CPU-side n-gram table as plain BF16, so serving on one 96 GB card needs a checkpoint that ships the table in BF16. This build does, and so do the community W4A16 builds measured above — all of them serve on a single card with ~100 GB of host RAM (or far less with the quantized-table sidecars: 49 / 32 / 28.8 GB, page-cache only). Checkpoints that store the table as FP8 with a scale tensor — the official FP8 release and the NVFP4 conversions derived from it — need multi-GPU (TP2 and up) or a runtime that dequantizes the table at load. GGUF and MLX conversions target other runtimes and do not serve on vLLM.

---

What's quantized to what

tensors	format
all 48 layers' routed experts (gate/up/down_proj, 120.8B params)	NVFP4 (group 16)
n-gram embedding table (51.2B, 128 shards)	BF16, pre-scaled — the vLLM offload worker loads no other format
attention, GDN linear-attention, shared experts, routers, MTP, vision, embeddings, norms	BF16, byte-identical to the source

Weights-only round-to-nearest, no calibration. The n-gram table carries the FP8 release's values, materialized in BF16 — the same numbers every runtime materializes at load.

model	shape	size	overall	knowledge	call	abstain
Laguna-XS-2.1	31 B MoE	19.3 GiB	81.7	83.8	68.4	73.5
Nemotron-3.5-Lightning-30B-A3B	30 B MoE+Mamba	19.2 GiB	87.1	87.9	85.4	70.5
Ornith-1.5-35B-A3B	35 B MoE	22.6 GiB	88.7	91.7	74.4	60.0
Muse-Glimmer-30B	30 B MoE	20.4 GiB	86.6	88.8	78.6	54.5
Qwen3.8-27B	27 B dense	20.7 GiB	88.8	90.4	85.5	54.5
Laguna-S-2.1	110 B MoE	64.0 GiB	84.3	87.1	64.6	81.0
Qwen3.8-Flash-Next	180 B MoE (6 B active)	185.8 GB	90.3	92.2	85.0	56.7

model	shape	size	overall	knowledge	call	abstain
Laguna-XS-2.1	31 B MoE	19.3 GiB	81.7	83.8	68.4	73.5
Nemotron-3.5-Lightning-30B-A3B	30 B MoE+Mamba	19.2 GiB	87.1	87.9	85.4	70.5
Ornith-1.5-35B-A3B	35 B MoE	22.6 GiB	88.7	91.7	74.4	60.0
Muse-Glimmer-30B	30 B MoE	20.4 GiB	86.6	88.8	78.6	54.5
Qwen3.8-27B	27 B dense	20.7 GiB	88.8	90.4	85.5	54.5
Laguna-S-2.1	110 B MoE	64.0 GiB	84.3	87.1	64.6	81.0
Qwen3.8-Flash-Next	180 B MoE (6 B active)	183.7 GB	90.3	92.2	84.8	56.7

---

_{primitive · more models · inference economics for production LLM systems}