Instructions to use AxionML/Mistral-Medium-3.5-128B-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AxionML/Mistral-Medium-3.5-128B-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="AxionML/Mistral-Medium-3.5-128B-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("AxionML/Mistral-Medium-3.5-128B-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("AxionML/Mistral-Medium-3.5-128B-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AxionML/Mistral-Medium-3.5-128B-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AxionML/Mistral-Medium-3.5-128B-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AxionML/Mistral-Medium-3.5-128B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/AxionML/Mistral-Medium-3.5-128B-NVFP4

SGLang

How to use AxionML/Mistral-Medium-3.5-128B-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AxionML/Mistral-Medium-3.5-128B-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AxionML/Mistral-Medium-3.5-128B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AxionML/Mistral-Medium-3.5-128B-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AxionML/Mistral-Medium-3.5-128B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use AxionML/Mistral-Medium-3.5-128B-NVFP4 with Docker Model Runner:
```
docker model run hf.co/AxionML/Mistral-Medium-3.5-128B-NVFP4
```

AxionML Mistral-Medium-3.5-128B-NVFP4

Mirrored by AxionML for open-source serving and deployment use cases. Part of AxionML's effort to provide ready-to-serve quantized models for the community.

Quantized by NVIDIA. The weights in this repository are an unmodified copy of nvidia/Mistral-Medium-3.5-128B-NVFP4 (revision b8c66d2098edd8c9c26bde2b2ff41b5967e111ae). All credit for the quantization belongs to NVIDIA.

This is an NVFP4-quantized version of mistralai/Mistral-Medium-3.5-128B (128B parameters, dense), quantized with NVIDIA Model Optimizer.

About NVFP4 quantization: NVFP4 on Blackwell couples a compact E2M1 FP4 codebook with blockwise FP8 (E4M3) scaling over 16-element micro-blocks, so that 4-bit stored values remain numerically useful for neural-network computation. The E2M1 codebook provides a small, nonuniform set of representable magnitudes up to ±6 and relies on saturating behavior rather than IEEE NaN/Inf encodings to maximize usable range per bit. Using an FP8 block scale (rather than power-of-two-only E8M0) enables fractional scales and error-minimizing scale selection. On Blackwell Tensor Cores, native FP4 multipliers exploit E2M1 simplicity while higher-precision FP32 accumulation protects dot-product accuracy.

Use is governed by Mistral AI's Modified MIT License (included as LICENSE): companies with more than US$20M global monthly revenue may not use this model without a commercial license from Mistral AI. NVIDIA's Open Model License also applies to the quantization (included as NVIDIA_OPEN_MODEL_LICENSE.md, with NOTICE).

Model Summary


Architecture	Dense transformer with vision encoder
Parameters	128B
Input	Text (11 languages), image
Context Length	262,144 tokens
Reasoning	Configurable per request (`reasoning_effort`)
Checkpoint Size	~95 GB

Evaluation Results

Benchmark	FP8	NVFP4
MMLU Pro	82.31	82.20
GPQA Diamond	76.88	76.80
AA-LCR	62.06	65.10
SciCode	42.50	42.60
AIME 2025	88.85	88.75
IFBench	70.25	69.17
MMMU Pro	63.35	62.79

Scores reported by NVIDIA for this checkpoint. reasoning_effort="high", temperature=0.7, top_p=0.95.

Quantization Details

Quantization format: mixed — MLP linear layers in decoder layers 4–86 in NVFP4; MLP in layers 0–3 and 87, all attention linear layers in FP8
KV cache: FP8
Calibration dataset: blend from Nemotron-Post-Training-v3
Tool: NVIDIA Model Optimizer v0.44.0

Usage

Deploy with SGLang

python3 -m sglang.launch_server \
    --model-path AxionML/Mistral-Medium-3.5-128B-NVFP4 \
    --tp 4 \
    --quantization modelopt_mixed \
    --trust-remote-code

Deploy with vLLM

vllm serve AxionML/Mistral-Medium-3.5-128B-NVFP4 \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.9 \
    --max-model-len 196608 \
    --config-format hf \
    --dtype auto \
    --trust-remote-code

Upstream validation was done with vllm/vllm-openai:v0.21.0 on B200; the SGLang command has not been validated upstream. Recommended sampling: temperature=0.7, top_p=0.95.

Limitations

The base model was trained on data that may contain toxic language and societal biases. The quantized model inherits these limitations. It may generate inaccurate, biased, or offensive content. Please refer to the original model card and the upstream quantized model card for full details.

Credits

Base model: mistralai/Mistral-Medium-3.5-128B
Quantization: nvidia/Mistral-Medium-3.5-128B-NVFP4 by NVIDIA
Mirror: AxionML

Downloads last month: 684

Safetensors

Model size

84B params

Tensor type

BF16

F8_E4M3

Model tree for AxionML/Mistral-Medium-3.5-128B-NVFP4

Base model

mistralai/Mistral-Medium-3.5-128B

Quantized

(41)

this model