[Bug] NEXTN/MTP draft acceptance decays to ~0 over server uptime on qwen4_exp (Qwen3.8-Flash-Next), fully restored by a restart

Summary

On RadixArk/Qwen3.8-Flash-Next-NVFP4 (qwen4_exp, hybrid GDN linear-attention + QSA sparse
full-attention MoE) served with NEXTN speculative decoding, draft acceptance is healthy at boot and
decays to near-zero over roughly 24 hours of uptime. Throughput falls ~2.9x. There is no crash,
no error, and no warning in the log - the server just gets progressively slower while reporting
accept rate: 0.00. A full restart restores it completely and reproducibly.

This is distinct from #21138 (constant 0.33 from boot, NemotronH weight-loading, fixed by #30456)
and from #19796 (radix-cache prefix hit → NaN crash, not gradual decay).

Measurements

Identical prompt, max_tokens: 400, temperature: 0, single stream.

Decode log at boot:

Decode batch, #running-req: 1, ..., accept len: 3.27, accept rate: 0.76, ..., gen throughput (token/s): 31.40

Decode log after ~4 days:

Decode batch, #running-req: 1, ..., accept len: 1.00, accept rate: 0.00, ..., gen throughput (token/s): 13.78

Onset: booted 08-27, first accept rate: 0.00 batches appear 08-28 (~24 h). Zeros then interleave
with occasional low non-zeros (0.05–0.27) rather than being uniformly zero, i.e. it is per-request
and degrades progressively, not a single hard break.

Ruled out

• Draft weight loading / quantization. speculative_draft_model_quantization resolves to
  modelopt_fp4 while the checkpoint's quantization_config.ignore lists mtp.*. This is
  identical in the healthy-at-boot run, so it is not the cause. Draft loads as
  Qwen4ExpForCausalLMMTP, draft decode/extend CUDA graphs capture normally.
• Thinking mode. enable_thinking: true and false degrade equally.
• Load/contention. Degraded numbers reproduce at #running-req: 1.

Suspected cause

We believe this is the ghost-node path fixed by #35821 (commit 23e51dd, merged 2026-08-21),
which describes exactly the accumulate-over-time mechanism we observe: an empty node inserted with
key_len=0 creating stale copy-on-write sources that poison subsequent match_prefix calls.
Poisoned prefix matches would degrade draft quality progressively and be wiped by a restart.

Neither half of #35821 appears to be present in the lmsysorg/sglang:qwen38flashnext image
(digest sha256:12d3392b…, built 2026-08-26, sglang 0.0.0.dev1+gd91c3682b):

• srt/speculative/spec_utils.py computes
  to_track_ith = torch.clamp(tracking_point - seq_lens_pre_verify - 1, min=0) — the lower bound
  only. The torch.minimum(..., accept_lens - 1) upper bound from #35821 is absent; there is no
  torch.minimum in the file.
• srt/mem_cache/mamba_radix_cache.py has if len(key) == 0: return 0, True, which returns early
  but does not free unprotected KV/mamba slots as #35821 describes.

So the day-0 qwen38flashnext image branch appears to have been cut without these fixes even
though they merged five days before the image was built.

Question for maintainers: is this the same root cause as #35821, or a separate uptime-decay
path specific to qwen4_exp? If the former, can the qwen38flashnext image be rebuilt to include
it?

Environment

• Model: RadixArk/Qwen3.8-Flash-Next-NVFP4 (176B total / 6B active, 512 experts top-10,
  48 layers, GDN:QSA 3:1, MTP 1 layer, 262144 ctx)
• Hardware: 2× NVIDIA DGX Spark (GB10, SM121), TP=2 across nodes over ConnectX-7 RoCE
• Image: lmsysorg/sglang:qwen38flashnext @ sha256:12d3392b…, sglang 0.0.0.dev1+gd91c3682b

Disclosure: we run a local derivative of this image that swaps the QSA packed-varlen attention
for a Triton FlashDecoding-style fallback, because the stock image's flash-attn-4 CuTe-DLS kernels
fail MLIR compilation on SM121. That patch touches the attention path, not the mamba radix cache or
spec_utils. We can retest on the stock image if the QSA path is suspected, though it does not boot
here unpatched.

Launch args (abridged)

--tp-size 2 --nnodes 2 --node-rank 0 --quantization modelopt_fp4
--attention-backend flashinfer --page-size 64
--speculative-algorithm NEXTN --speculative-num-steps 3
--speculative-eagle-topk 1 --speculative-num-draft-tokens 4
--mamba-ssm-dtype bfloat16 --mamba-radix-cache-strategy extra_buffer
--mamba-track-interval 64 --mamba-full-memory-ratio 0.3
--mem-fraction-static 0.82 --chunked-prefill-size 1024
--max-running-requests 16 --context-length 262144
--disable-prefill-cuda-graph --enable-metrics

Reproducing / monitoring

# acceptance distribution across the server's whole uptime
docker logs <container> | grep -oE 'accept rate: [0-9.]+' | sort | uniq -c

# current state
curl -s localhost:8010/metrics -H "Authorization: Bearer $KEY" \
  | grep -E 'spec_accept_length|spec_accept_rate'

Note --decode-log-interval defaults to 40, so the decode log samples every 40 steps.