Update README.md
Browse filesinclude sglang support
README.md
CHANGED
|
@@ -62,6 +62,7 @@ Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated sys
|
|
| 62 |
## Software Integration:
|
| 63 |
**Supported Runtime Engine(s):** <br>
|
| 64 |
* **vLLM** <br>
|
|
|
|
| 65 |
|
| 66 |
**Supported Hardware Microarchitecture Compatibility:** <br>
|
| 67 |
* NVIDIA Blackwell <br>
|
|
@@ -128,6 +129,26 @@ vllm serve nvidia/Qwen3.8-27B-NVFP4 \
|
|
| 128 |
--enable-chunked-prefill
|
| 129 |
```
|
| 130 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 131 |
## Evaluation
|
| 132 |
The accuracy benchmark results are presented in the table below:
|
| 133 |
<table>
|
|
|
|
| 62 |
## Software Integration:
|
| 63 |
**Supported Runtime Engine(s):** <br>
|
| 64 |
* **vLLM** <br>
|
| 65 |
+
* **SGLang** <br>
|
| 66 |
|
| 67 |
**Supported Hardware Microarchitecture Compatibility:** <br>
|
| 68 |
* NVIDIA Blackwell <br>
|
|
|
|
| 129 |
--enable-chunked-prefill
|
| 130 |
```
|
| 131 |
|
| 132 |
+
To serve this checkpoint with [SGLang](https://github.com/sgl-project/sglang), you can start the docker `lmsysorg/sglang:dev` and run the sample command below:
|
| 133 |
+
|
| 134 |
+
```sh
|
| 135 |
+
sglang serve \
|
| 136 |
+
--trust-remote-code \
|
| 137 |
+
--model-path nvidia/Qwen3.8-27B-NVFP4 \
|
| 138 |
+
--kv-cache-dtype fp8_e4m3 \
|
| 139 |
+
--mem-fraction-static 0.85 \
|
| 140 |
+
--chunked-prefill-size 2048 \
|
| 141 |
+
--reasoning-parser qwen3 \
|
| 142 |
+
--tool-call-parser qwen3_coder \
|
| 143 |
+
--mamba-full-memory-ratio 4.59 \
|
| 144 |
+
--host 0.0.0.0 \
|
| 145 |
+
--port 30000 \
|
| 146 |
+
--mamba-radix-cache-strategy extra_buffer \
|
| 147 |
+
--mamba-ssm-dtype float32
|
| 148 |
+
```
|
| 149 |
+
|
| 150 |
+
For more details please refer to [SGLang cookbook](https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8-27B).
|
| 151 |
+
|
| 152 |
## Evaluation
|
| 153 |
The accuracy benchmark results are presented in the table below:
|
| 154 |
<table>
|