Instructions to use pytorch/Qwen3-8B-AWQ-INT4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use pytorch/Qwen3-8B-AWQ-INT4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="pytorch/Qwen3-8B-AWQ-INT4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("pytorch/Qwen3-8B-AWQ-INT4")
model = AutoModelForCausalLM.from_pretrained("pytorch/Qwen3-8B-AWQ-INT4", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use pytorch/Qwen3-8B-AWQ-INT4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "pytorch/Qwen3-8B-AWQ-INT4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pytorch/Qwen3-8B-AWQ-INT4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/pytorch/Qwen3-8B-AWQ-INT4

SGLang

How to use pytorch/Qwen3-8B-AWQ-INT4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "pytorch/Qwen3-8B-AWQ-INT4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pytorch/Qwen3-8B-AWQ-INT4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "pytorch/Qwen3-8B-AWQ-INT4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pytorch/Qwen3-8B-AWQ-INT4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use pytorch/Qwen3-8B-AWQ-INT4 with Docker Model Runner:
```
docker model run hf.co/pytorch/Qwen3-8B-AWQ-INT4
```

Int4ChooseQParamsAlgorithm error with vllm server

by witwicki - opened Sep 21, 2025

Discussion

witwicki

Sep 21, 2025

•

edited Sep 21, 2025

In a venv with:

torch 2.8.0
torchao 0.13.0
vllm 0.10.2rc3.dev324

I see the following error when running vllm serve pytorch/Qwen3-8B-AWQ-INT4:

(APIServer pid=109129)   Value error, Failed to find class Int4ChooseQParamsAlgorithm in any of the allowed modules: 
torchao.quantization, torchao.dtypes, torchao.prototype.awq, torchao.prototype.mx_formats, 
torchao.quantization.quantize_.common, torchao.sparsity.sparse_api, torchao.prototype.quantization [type=value_error, 
input_value=ArgsKwargs((), {'model_co...additional_config': {}}), input_type=ArgsKwargs]

Update: From the code, it looks like this model now depends on torchao>0.13.0, which as far as I can tell depends on torch >= 2.9.0, breaking the dependencies of vllm's latest nightly build (2.8.0). Is that right? Any suggestions on how to make it servable with vllm?

jerryzh168

pytorch org Sep 22, 2025

•

edited Oct 1, 2025

Thanks for trying this out, yeah this model requires more recent torchao, it has to use torchao nightly currently. and the compatibility is currently a bit complicated: https://github.com/pytorch/ao/issues/2919

One thing we can try is to install torchao, torch and vllm nightly now I think, through:

pip install --pre torchao torch vllm --extra-index-url https://download.pytorch.org/whl/nightly/cu128

vllm is added very recently, it is built against the torch nightly I think.

I found the following issues currently, but will fix soon

serving with vllm: seems this checkpoint produces wrong results, I also tests Phi4-mini-AWQ-INT4, that one is fine, so this is something specific to this checkpoint
Update[10/01/2025]: this seems to specific to Qwen3-4B-AWQ-INT4 and to vllm...don't have time to debug now though
serving with transformers, seems to work now, but I'll put up a fix if it failed

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment