shadowpa0327/Qwen3.8-Flash-Next-DSpark

Qwen3.8-Flash-Next DSpark draft

A DSpark speculative-decoding draft for Qwen3.8-Flash-Next, trained with SpecForge against nvidia/Qwen3.8-Flash-Next-NVFP4.

This revision adds coding-agent training. The general chat draft was fine-tuned on Qwen3.8-Flash-Next's own coding-agent sessions from nvidia/Open-SWE-Traces, up to 32K tokens per session. Results on held-out agent sessions with 0-128K tokens of context:

• It accepts 37% more tokens per verify step than before this training (3.70 vs 2.69).
• It still accepts 13% fewer than the model's built-in MTP head at 7 steps (4.24). Most of the gap is beyond 32K tokens of context.
• Below 32K context it decodes as fast as or up to 16% faster than the best MTP setting at batch 1.

On chat, math and code prompts it accepts 4% less than before (4.55 vs 4.75).

These numbers come from step 1,600 of the coding-agent run. This upload is step 1,800, its final checkpoint, which has not been served yet (see Verification status). Acceptance on held-out sessions stopped improving after step 1,000 (3.65 at 1,000, 3.61 at 1,400, 3.70 at 1,600), and the run was stopped at step 1,955.

Earlier weights remain in this repo's history:

• General chat draft, stage-2 step 11,000 (commit 5bc7f2b). It accepts slightly more on chat prompts.
• Stage-2 step 5,000 (commit cb6e624).
• Stage-1 step 56,000 (commit f653cfb).

Pass --revision 5bc7f2b to hf download to serve the general draft.

• 5 decoder layers, 664.6M parameters, BF16
• Reads target layers 1, 12, 23, 34, 45; shares the target's embedding and LM head (not included here)
• Trained block size 7, vanilla Markov head (rank 256), confidence head

Serving with SGLang 0.5.21

Stock SGLang 0.5.21 does not hand the target's intermediate hidden states to a DSpark draft for this model. The small qwen38_dspark_sglang package in this repo fixes that at load time, so no SGLang patching is needed:

pip install "sglang==0.5.21"
hf download shadowpa0327/Qwen3.8-Flash-Next-DSpark --local-dir qwen38-dspark

PYTHONPATH=$PWD/qwen38-dspark SGLANG_EXTERNAL_MODEL_PACKAGE=qwen38_dspark_sglang \
python -m sglang.launch_server \
  --model-path nvidia/Qwen3.8-Flash-Next-NVFP4 --tp-size 2 \
  --speculative-algorithm DSPARK \
  --speculative-draft-model-path $PWD/qwen38-dspark \
  --speculative-dspark-block-size 3

The server log must contain qwen38_dspark_sglang: qwen4_exp aux hidden-state capture installed. If it shows Ignore import error when loading qwen38_dspark_sglang... instead (e.g. a different SGLang version), the add-on did not load and DSpark will not work.

Block size. Qwen3.8-Flash-Next's sparse attention (QSA) caps a verify step at 4 tokens in stock SGLang, so use --speculative-dspark-block-size 3 there. All results below use block size 7 (verify width 8). That needs a patched QSA build, not part of this repo, that sizes the pending index-key ring for wider verify steps. Acceptance at block size 3 has not been measured for this revision.

Our evaluation servers (H100, TP2) used that build with --speculative-dspark-block-size 7 --max-running-requests 32 --mem-fraction-static 0.88 --dtype bfloat16 --attention-backend fa3 --enforce-disable-flashinfer-allreduce-fusion --disable-radix-cache, and --context-length 131584 for the agent sessions.

What the add-on does

qwen38_dspark_sglang/qwen4_exp_aux_capture.py replaces three forward methods of SGLang's qwen4_exp model in place (Qwen4ExpModel, Qwen4ExpVLModel, Qwen4ExpForConditionalGeneration). Before each captured layer it takes the mean of the layer input's 4 hyper-connection branches, which is how the draft was trained, and returns those captures to the draft. The code is SpecForge's SGLang 0.5.21 capture patch for qwen4_exp, adapted from SGLang's own qwen4_exp.py (Apache-2.0). It only has an effect when a DFlash or DSpark draft requests aux capture, and it refuses to load on any SGLang other than 0.5.21.

Training

SpecForge online DSpark training (hidden states streamed from the target, never stored): one TP2 NVFP4 target server feeding a DP2 trainer on 4x H100, 512 anchors per sample.

1. Original answers. UltraChat-200k + Open-PerfectBlend (95% train split), ~1.56M conversations with their dataset responses. Global batch 64, max length 4,096, cosine schedule, stopped at step 56,000 (commit f653cfb).
2. Target-written answers. Warm-started from stage 1. 387,390 Open-PerfectBlend conversations whose assistant turns were regenerated by Qwen3.8-Flash-Next in non-thinking mode at its default sampling (temperature 1.0, top-p 0.95, top-k 20). Global batch 64, max length 4,096, learning rate 6e-4. Step 5,000 and step 11,000 were uploaded (commits cb6e624, 5bc7f2b). Step 12,000 starts stage 3.
3. Coding-agent sessions (this revision). 95,596 sessions from nvidia/Open-SWE-Traces v1.2 that Qwen3.8-Flash-Next wrote itself in thinking mode. The harnesses are OpenHands, SWE-agent and mini-SWE-agent, running real repository tasks from SWE-rebench-V2 and Scale-SWE.
   • Each session is rendered once with the target's chat template, with tools and every turn's reasoning kept, so one row trains every turn the target wrote.
   • Rows are cut at the last turn end within 32,768 tokens. Most sessions are longer than that: 81% of rows end within 10% of the cap. The rows total 2.9B tokens.
   • Repos whose name hash falls in a 2% bucket are held out for evaluation.
   • Global batch 16, max length 32,768.
   • Two short runs on subsets came first: 300 steps on an 11,999-session subset (learning rate 2e-4), then 375 steps on 6,000 other sessions (1.4e-4).
   • Then one pass over all sessions: learning rate 1.5e-4, 1% warmup, cosine over 5,975 planned steps. It was stopped at step 1,955 once held-out acceptance stopped improving.
   • This checkpoint is step 1,800 of that pass, about 28,800 sessions.

Results

SGLang 0.5.21 with the same capture code and the patched QSA build, H100 TP2, greedy decoding, block size 7. Acceptance = committed tokens per target verify step, bonus token included.

Acceptance on held-out coding-agent sessions. The test set has 120 cut points: 20 per context-depth bucket, drawn from 300 sessions of held-out repos, with harnesses interleaved. Each prompt is a session prefix that ends where an assistant turn starts, in thinking mode, and the target writes that turn (up to 512 new tokens, 6 concurrent requests):

Context	0-8K	8-16K	16-32K	32-64K	64-96K	96-128K	≤32K	>32K	all
DSpark, agent step 1,600	5.64	4.72	3.48	3.41	3.95	3.16	4.10	3.46	3.70
DSpark, stage 2 step 12,000 (before agent training)	3.47	3.15	2.67	2.70	2.67	2.33	2.94	2.55	2.69
MTP, 7 steps	5.07	5.02	3.95	3.76	4.69	4.07	4.41	4.13	4.24

Acceptance on chat, math and code prompts. This set has 240 prompts, 40 each from six sets, using the first user turn in non-thinking mode, 32 concurrent requests and 512 new tokens:

	mt-bench	gsm8k	humaneval	math500	alpaca	PerfectBlend held-out	all
DSpark, agent step 1,600	3.48	6.37	6.76	5.63	3.21	4.44	4.55
DSpark, stage 2 step 12,000	3.68	6.60	6.81	5.91	3.37	4.52	4.75
MTP, 7 steps	3.69	5.86	6.96	5.67	3.51	4.44	4.67

The previous upload (stage 2 step 11,000) scored 4.70 on the full 480-prompt version of this set. Its README has the chat speed measurements.

Median time per output token (ms) on held-out agent sessions, measured on agent step 1,200 and not re-measured for this revision. Requests in a batch start together, and TPOT is taken from the token stream, so prefill and queueing are excluded. Generation is greedy and runs to the natural end of the turn (512 tokens max). The first six columns are batch 1, with 4 sessions per bucket. The last two (×4) are batch 4, with 4 rounds per context range:

Context	0-8K	8-16K	16-32K	32-64K	64-96K	96-128K	0-32K ×4	32-96K ×4
DSpark, agent step 1,200	1.75	2.61	2.29	3.65	3.09	3.87	8.09	14.45
MTP, 7 steps	2.03	2.59	2.77	3.25	3.03	3.52	9.63	19.69
MTP, 5 steps	2.14	2.77	2.72	3.23	3.33	3.44	10.48	16.19
No speculative decoding	7.21	7.35	7.24	7.29	7.30	7.37	12.06	22.19

A repeat DSpark run the same day was 4-9% faster at batch 1. At batch 4 it measured 8.78 ms for 0-32K and 16.95 ms for 32-96K. With this few requests, differences under about 10% at batch 1, and under about 20% at batch 4, are within run-to-run variation.

• Below 32K context, DSpark is fastest or tied at batch 1 and fastest at batch 4.
• Beyond 32K, DSpark takes up to 13% longer per token than MTP at batch 1, because MTP accepts more there. At batch 4 it is level with MTP at 5 steps.

DSpark was trained only on the first 32K tokens of each session; training at longer lengths is the next step.

Speculative decoding keeps greedy output unchanged. On earlier checkpoints, output mismatches against a clean greedy prefill were the same as without speculation (1.46-1.51% vs 1.47-1.50% of tokens on >2K-token prompts).

Verification status

• Step 1,800 itself has not been served yet. Every number above comes from step 1,600 of the same run (speed from step 1,200). Step 1,600 was served end to end on SGLang 0.5.21 with SpecForge's capture patch and the patched QSA build.
• Step 1,800's weights differ from step 1,600's by 1.3% (relative L2 norm). Its training-side acceptance estimate is the same within noise (walk accepted length 4.09 vs 4.07).
• The config and file layout are identical to the previous upload: same 62 tensors, same shapes and dtypes.
• An end-to-end serving run of this repo through the add-on on stock SGLang 0.5.21 (block size 3) is still pending.

License

The weights are a draft for, and were trained on hidden states of, nvidia/Qwen3.8-Flash-Next-NVFP4; use is subject to that model's license. Stage 3 trained on nvidia/Open-SWE-Traces (CC-BY-4.0, arXiv 2606.16038).