WaveCut/Qwen3.8-Flash-Next-ngram-table-NInfer-v3

Qwen3.8-Flash-Next n-gram table, NInfer v3 artifact

The n-gram table of Qwen3.8-Flash-Next's per-layer embedding (PLE), as a NInfer v3 artifact for NInfer-all, the master branch of iamwavecut/ninfer-all. Every NInfer conversion of Qwen3.8-Flash-Next published without its table reads its rows from this file:

model	artifact
Qwen3.8-Flash-Next GSQ-RCO Q2_0	WaveCut/Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-NInfer-v3
Qwen3.8-Flash-Next GSQ-RCO IQ3_S	WaveCut/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-NInfer-v3
Qwen3.8-Flash-Next Coder GSQ-RCO IQ1_M	WaveCut/Qwen3.8-Flash-Next-GSQ-RCO-Coder-IQ1_M-NInfer-v3

Stock NInfer builds cannot read it: the model family and the gguf_* formats exist only in that line; build master from commit 82f120de on.

What is inside

The table holds 320,001,536 rows of 160 values (51.2B parameters), 16 hash heads over the 2- and 3-token n-grams of the context, each head's rows at its own offset. Every token reads the 16 rows its n-grams address. The rows are ISTA-DASLab's IQ4_NL blocks (per_layer_token_embd of their GGUF releases, 90 bytes a row), copied byte for byte: the second shard of every GSQ-RCO release, the Coder build's included, is the same file.

One file, Qwen3.8-Flash-Next-ngram-table-IQ4_NL-ninfer-v3.ninfer, of 28,905,035,512 bytes (26.92 GiB), next to its conversion report, SHA256SUMS, NOTICE and LICENSE. Its ngram component records the hash constants the rows were written for, their format and the SHA-256 of the rows, dd55c28902f38cd88134b2a569c51282c5ffce30080487e1a645740115c56cc3. A model names the table it needs by those, and NInfer refuses a table that differs. Original row conversion command, before profile attachment:

python3 -m tools.convert --model Qwen3.8-Flash-Next \
  --recipe qwen3_8_flash_next_gguf --components ngram \
  --source ngram=Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-00002-of-00002.gguf \
  --device cpu --rows-per-chunk 65536 \
  --name qwen3.8-flash-next-ngram-table --out Qwen3.8-Flash-Next-ngram-table-IQ4_NL-ninfer-v3.ninfer

--model needs only the configuration files of Qwen/Qwen3.8-Flash-Next.

Running

Download it next to a model and pass it with --ngram-table:

hf download WaveCut/Qwen3.8-Flash-Next-ngram-table-NInfer-v3 \
  Qwen3.8-Flash-Next-ngram-table-IQ4_NL-ninfer-v3.ninfer --local-dir models
hf download WaveCut/Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-NInfer-v3 \
  Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-ninfer-v3.ninfer --local-dir models
ninfer-serve models/Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-ninfer-v3.ninfer \
  --ngram-table models/Qwen3.8-Flash-Next-ngram-table-IQ4_NL-ninfer-v3.ninfer \
  --model-id qwen3.8-flash-next --expert-residency host --max-context 32768

The Docker image takes the same flags after serve, with both files under /models:

docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
  -v "$PWD/models:/models" -v ninfer-cache:/cache \
  ghcr.io/iamwavecut/ninfer-all serve /models/Qwen3.8-Flash-Next-GSQ-RCO-Q2_0-ninfer-v3.ninfer \
  --ngram-table /models/Qwen3.8-Flash-Next-ngram-table-IQ4_NL-ninfer-v3.ninfer \
  --model-id qwen3.8-flash-next --expert-residency host --max-context 32768

With the default disk residency, nothing reads all table rows at load: each token reads its 16 rows (1.4 KB) from the file through the OS page cache, so the table can sit on any disk. --ngram-residency ram loads all of it into RAM instead.

A model started without its table is refused. --no-ngram-table overrides that, and the model then runs without its per-layer embedding: a non-standard, experimental mode, warned about at startup, with no practical use. The model was trained with the table: without it the Q2_0 release's WikiText-2 perplexity nearly doubles, from 2.66 to 5.01.

Optional broad n-gram profile

This update adds a 104,893,176-byte row ranking as an embedded resource and as broad-v1.hot. The container grows by 104,893,176 bytes (0.364%). Every pre-existing object passed exact byte readback; the IQ4 row bytes, quantization and row digest are unchanged. This profile selects cached rows; it does not change model weights or establish answer-quality improvements.

The deterministic corpus uses four pinned Common Corpus shards and one pinned github-code-clean shard. It covers prose, web, press, science and code without preferred languages. Equal per-group token ceilings limit dominance, and source documents or repositories separate training from held-out text. profile-corpus.json records revisions, sampling and limitations. Language labels are automatic; this finite sample is not a complete language inventory or an estimate of user traffic. Exact snippet deduplication does not eliminate all near-duplicates.

The profile contains 26,223,284 ranked rows from 3,711,509 training tokens. The held-out set contains 1,103,918 tokens and 445 domain/language groups. NumPy and the native C++ profiler produced byte-identical profiles.

RAM budget, MiB	held-out row hits
256	33.79%
512	38.49%
1024	42.92%
2048	48.48%
4096	49.64%
8192	49.64%
16384	49.64%
32768	49.64%

Across 239 groups with at least 2,048 held-out tokens, the minimum / median / maximum hit rates are 6.04% / 32.85% / 90.64% at 256 MiB and 20.54% / 49.38% / 95.47% at 4 GiB. Japanese books have the lowest measured coverage: 6.04% and 20.54%, respectively. All groups, including sparse ones, remain in profile-heldout.json.

On RTX 3090, disk residency, the embedded profile and the explicit profile produced identical tokens for two prompts, two repetitions and 32 output tokens, with MTP K=4 and int8 KV. Both profile modes served rows from the configured 256 MiB RAM cache. The Engine report retains startup time, every sample and memory counters. Cache state was uncontrolled and artifact transfers overlapped these checks, so this is functional evidence, not an isolated speedup. Caching remains opt-in; earlier cache tests showed 10–11% slower first requests.

Public commit 64492cb14 accepts an explicit profile:

hf download WaveCut/Qwen3.8-Flash-Next-ngram-table-NInfer-v3 broad-v1.hot --local-dir models
# Add these options to the serving command above:
--ngram-residency ram-hot --ngram-hot-profile models/broad-v1.hot --ngram-ram-mib 256

Automatic loading of the embedded profile is available in NInfer master at ac94a1b610f6 and was qualified on the development snapshot recorded in profile-engine.json. Use --ngram-residency ram-hot --ngram-ram-mib 256 to select it. Older commit 64492cb14 requires the explicit --ngram-hot-profile option above. Default disk operation does not use the profile.

Credits and license

The table comes from ISTA-DASLab's Qwen3.8-Flash-Next GSQ-RCO GGUFs, which quantize Qwen/Qwen3.8-Flash-Next. The model is released under the Qwen Community License 1.0, reproduced in LICENSE, and its conditions apply to this file; ISTA-DASLab publish their GGUFs under Apache-2.0. Attribution notices are collected in NOTICE. This repository only re-packs those rows into NInfer's container.