zjxia commited on
Commit
47717b8
·
verified ·
1 Parent(s): fed99d8

Update README.md

Browse files

include sglang support

Files changed (1) hide show
  1. README.md +21 -0
README.md CHANGED
@@ -62,6 +62,7 @@ Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated sys
62
  ## Software Integration:
63
  **Supported Runtime Engine(s):** <br>
64
  * **vLLM** <br>
 
65
 
66
  **Supported Hardware Microarchitecture Compatibility:** <br>
67
  * NVIDIA Blackwell <br>
@@ -128,6 +129,26 @@ vllm serve nvidia/Qwen3.8-27B-NVFP4 \
128
  --enable-chunked-prefill
129
  ```
130
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
131
  ## Evaluation
132
  The accuracy benchmark results are presented in the table below:
133
  <table>
 
62
  ## Software Integration:
63
  **Supported Runtime Engine(s):** <br>
64
  * **vLLM** <br>
65
+ * **SGLang** <br>
66
 
67
  **Supported Hardware Microarchitecture Compatibility:** <br>
68
  * NVIDIA Blackwell <br>
 
129
  --enable-chunked-prefill
130
  ```
131
 
132
+ To serve this checkpoint with [SGLang](https://github.com/sgl-project/sglang), you can start the docker `lmsysorg/sglang:dev` and run the sample command below:
133
+
134
+ ```sh
135
+ sglang serve \
136
+ --trust-remote-code \
137
+ --model-path nvidia/Qwen3.8-27B-NVFP4 \
138
+ --kv-cache-dtype fp8_e4m3 \
139
+ --mem-fraction-static 0.85 \
140
+ --chunked-prefill-size 2048 \
141
+ --reasoning-parser qwen3 \
142
+ --tool-call-parser qwen3_coder \
143
+ --mamba-full-memory-ratio 4.59 \
144
+ --host 0.0.0.0 \
145
+ --port 30000 \
146
+ --mamba-radix-cache-strategy extra_buffer \
147
+ --mamba-ssm-dtype float32
148
+ ```
149
+
150
+ For more details please refer to [SGLang cookbook](https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8-27B).
151
+
152
  ## Evaluation
153
  The accuracy benchmark results are presented in the table below:
154
  <table>