Add SGLang serving instructions
Browse filesAdds SGLang as a supported runtime and documents a validated text-only
serving command for nvidia/Muse-Glimmer-30B-NVFP4.
Validation:
- SGLang 0.0.0.dev1+g4358a1617 on one NVIDIA Blackwell B200 GPU
- Chat, streaming, reasoning parser, and tool-call parser smoke tests passed
- Full GSM8K: 1319/1319 examples, 96.44% accuracy, 0 request errors,
and 0 truncated responses
- Fixed random benchmark used 1024 input and 256 output tokens, with two
discarded warmups and three formal repetitions at concurrency 1, 16,
and 64
DFlash also passed accuracy validation at 96.21%, but was intentionally
omitted from the README because total throughput at concurrency 64 was
28.00% below Standard.
GSM8K used high reasoning, temperature 1.0, top-p 0.95, and seed 42.
sgl-eval 0.1.1 did not expose a top-k option.
|
@@ -64,6 +64,7 @@ Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated sys
|
|
| 64 |
## Software Integration:
|
| 65 |
**Supported Runtime Engine(s):** <br>
|
| 66 |
* **vLLM** <br>
|
|
|
|
| 67 |
|
| 68 |
**Supported Hardware Microarchitecture Compatibility:** <br>
|
| 69 |
* NVIDIA Blackwell B200 <br>
|
|
@@ -103,7 +104,7 @@ The model version is NVFP4 1.0 version and is quantized with nvidia-modelopt **v
|
|
| 103 |
**Properties:** We evaluated the model on text-based reasoning, coding, agentic tool-use, and multimodal benchmarks: Terminal-Bench 2.1 is an open-source evaluation framework designed to test AI agents on 89 complex, real-world tasks inside sandboxed command-line and container environments. GDPval is an AI evaluation benchmark designed to measure how well frontier models perform real, economically valuable knowledge work across major industries. GPQA Diamond is a specialized benchmark of 198 graduate-level science questions designed to test advanced reasoning and expert knowledge in artificial intelligence. MMMU-Pro is an enhanced, more rigorous version of the MMMU benchmark designed to evaluate genuine multimodal understanding and reasoning in AI models. SciCode is a scientist-curated coding benchmark designed to evaluate the capability of Large Language Models (LLMs) to write code for complex, realistic scientific research problems. IFBench is evaluation benchmark measure how well language models generalize to unseen, out-of-domain (OOD) verifiable output constraints. Artificial Analysis Long Context Reasoning (AA-LCR) measures how well AI models extract, connect, and reason over information across multiple long documents.<br>
|
| 104 |
|
| 105 |
## Inference:
|
| 106 |
-
**Acceleration Engine:** **vLLM** <br>
|
| 107 |
**Test Hardware:** **NVIDIA Blackwell B200** <br>
|
| 108 |
|
| 109 |
## Post Training Quantization
|
|
@@ -143,6 +144,8 @@ The search used 512 calibration sequences of length 2,048, batch size 1, from th
|
|
| 143 |
|
| 144 |
## Usage
|
| 145 |
|
|
|
|
|
|
|
| 146 |
We tested serving the Muse-Glimmer-30B-NVFP4 with vLLM 0.28.0. The tested command inside that runtime is:
|
| 147 |
|
| 148 |
```bash
|
|
@@ -160,6 +163,23 @@ vllm serve /path/to/muse-glimmer-nvfp4 \
|
|
| 160 |
--gpu-memory-utilization 0.92 \
|
| 161 |
--mamba-cache-mode align
|
| 162 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 163 |
For matched quality, send chat-completion requests with `temperature=1.0`, `top_p=0.95`, `top_k=64`, thinking enabled, and high reasoning strength. Tool-use applications should use the packaged `chat_template.jinja` and Muse Glimmer reasoning/tool parsers.
|
| 164 |
|
| 165 |
## Evaluation
|
|
|
|
| 64 |
## Software Integration:
|
| 65 |
**Supported Runtime Engine(s):** <br>
|
| 66 |
* **vLLM** <br>
|
| 67 |
+
* **SGLang** <br>
|
| 68 |
|
| 69 |
**Supported Hardware Microarchitecture Compatibility:** <br>
|
| 70 |
* NVIDIA Blackwell B200 <br>
|
|
|
|
| 104 |
**Properties:** We evaluated the model on text-based reasoning, coding, agentic tool-use, and multimodal benchmarks: Terminal-Bench 2.1 is an open-source evaluation framework designed to test AI agents on 89 complex, real-world tasks inside sandboxed command-line and container environments. GDPval is an AI evaluation benchmark designed to measure how well frontier models perform real, economically valuable knowledge work across major industries. GPQA Diamond is a specialized benchmark of 198 graduate-level science questions designed to test advanced reasoning and expert knowledge in artificial intelligence. MMMU-Pro is an enhanced, more rigorous version of the MMMU benchmark designed to evaluate genuine multimodal understanding and reasoning in AI models. SciCode is a scientist-curated coding benchmark designed to evaluate the capability of Large Language Models (LLMs) to write code for complex, realistic scientific research problems. IFBench is evaluation benchmark measure how well language models generalize to unseen, out-of-domain (OOD) verifiable output constraints. Artificial Analysis Long Context Reasoning (AA-LCR) measures how well AI models extract, connect, and reason over information across multiple long documents.<br>
|
| 105 |
|
| 106 |
## Inference:
|
| 107 |
+
**Acceleration Engine:** **vLLM, SGLang** <br>
|
| 108 |
**Test Hardware:** **NVIDIA Blackwell B200** <br>
|
| 109 |
|
| 110 |
## Post Training Quantization
|
|
|
|
| 144 |
|
| 145 |
## Usage
|
| 146 |
|
| 147 |
+
### vLLM
|
| 148 |
+
|
| 149 |
We tested serving the Muse-Glimmer-30B-NVFP4 with vLLM 0.28.0. The tested command inside that runtime is:
|
| 150 |
|
| 151 |
```bash
|
|
|
|
| 163 |
--gpu-memory-utilization 0.92 \
|
| 164 |
--mamba-cache-mode align
|
| 165 |
```
|
| 166 |
+
|
| 167 |
+
### SGLang
|
| 168 |
+
|
| 169 |
+
We tested text-only serving with SGLang on one NVIDIA Blackwell B200 GPU. The tested command is:
|
| 170 |
+
|
| 171 |
+
```bash
|
| 172 |
+
sglang serve \
|
| 173 |
+
--model-path nvidia/Muse-Glimmer-30B-NVFP4 \
|
| 174 |
+
--language-model-only \
|
| 175 |
+
--reasoning-parser muse \
|
| 176 |
+
--tool-call-parser muse \
|
| 177 |
+
--kv-cache-dtype fp8_e4m3 \
|
| 178 |
+
--mem-fraction-static 0.85 \
|
| 179 |
+
--host 0.0.0.0 \
|
| 180 |
+
--port 30000
|
| 181 |
+
```
|
| 182 |
+
|
| 183 |
For matched quality, send chat-completion requests with `temperature=1.0`, `top_p=0.95`, `top_k=64`, thinking enabled, and high reasoning strength. Tool-use applications should use the packaged `chat_template.jinja` and Muse Glimmer reasoning/tool parsers.
|
| 184 |
|
| 185 |
## Evaluation
|