tcclaviger

/

Step-3.7-Flash-240REAP-MXFP416

like 2

If you run into problem let me know, happy there are others out there rocking 9700s that get use out of the changes

2,5M KVCache that is crazy! will also try with original fp8 launch, because on main branch mtp broken

should i create own moe config, or your Docker build already have it?

Avg generation throughput: 39.9 tokens/s and Accepted throughput: 29.00 tokens/s

MTP acceptance rate looks solid, so that's good.
Base throughput seems to be on the lower than I expected side, but I'm pretty certain its the configs missing, mine are done for TP4 so the split will be different dimensions.
Below the config spam, do you see a line slightly below about the Triton unified attention config, did it load it or use the default values?

Either way, I'll post up the config sweep files used to quickly tune the kernels (by quick it's still a couple hours but not a couple days like it would be otherwise). Will probably be tomorrow before I can package it into a portable file that you can use.

The most important one being the triton attention tuning for endurance of decode speed as context length grows.
To get max speed also be sure to capture tunableop unseen shapes, tune them offline, and load them as well.

The top speed will definitely be --kv-cache-dtype fp8, but FP16 is not very far behind, just a few percent. The kernel GEMM configs are the big winners on speed.

You might be better off running it TP4 same as my docker compose in the repo so you inherit the configs i baked into the image. I have attempted to find a pattern, an formula that could calculate the ideal config data rather than needing to test it, if there is one AMD will need to write it from their internal docs. The tuning scripts do intelligently reduce total search range by ~93% compared to brute force though.

Now that I think about it, your results are not surprising the 8 card setup will a have larger GEMM shape mismatch than my 4 card does, meaning custom configs will benefit you more than they did me.

The container should at least, give throughput uplift for any model you load, specifically on prefill.

djdeniro,
I'll make it simpler but take a bit longer, I'll just do the TP8 shapes and bake them.

oh, amazing, but if i can run benchmark on my 8x gpu please share it , will tuning best gemm / moe parameters faster together

Adding teh quantizer and tuner into the image, then going to strip the image down over the weekend.

This way you should be able to quantize any defined model in the image yourself, tune for it, and go.

As new models are desired I'll need to add them to the model registry to handle edge cases on naming etc, but its an offline quantization that doesn't require activations so it's pretty quick for what it is, mostly constrained by CPU speed and Drive speed, the gpu compute is fast and parallelized.

In the interim, I'm going to validate the new items by running it to generate the TP2 and TP8 configs for MXFP416 on Step 3.7, Qwen3.6 27B, Qwen3.6 35B shapes as a baseline.

I'll move this all to gitlab or something at some point in the near future to make it easier for users.

Image updated with TP2 and TP8 configs baked in.

Robert, just pulled image, trying new stuff now

Thanks djdeniro, cool having someone with TP8 setup testing.

I am still working on building out a RDNA4 config tuning utility into the container so everyone can launch and get their own systems ideal configs, it makes a huge difference.

No MTP 36-37 t/s per req.

(APIServer pid=1) INFO 06-05 18:31:59 [utils.py:344] 
(APIServer pid=1) INFO 06-05 18:31:59 [utils.py:344]        █     █     █▄   ▄█
(APIServer pid=1) INFO 06-05 18:31:59 [utils.py:344]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.22.69
(APIServer pid=1) INFO 06-05 18:31:59 [utils.py:344]   █▄█▀ █     █     █     █  model   /app/models/models/vllm/Step-3.7-Flash-240REAP-MXFP416
(APIServer pid=1) INFO 06-05 18:31:59 [utils.py:344]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1) INFO 06-05 18:31:59 [utils.py:344] 
(APIServer pid=1) INFO 06-05 18:31:59 [utils.py:278] non-default args: {'model_tag': '/app/models/models/vllm/Step-3.7-Flash-240REAP-MXFP416', 'enable_auto_tool_choice': True, 'tool_call_parser': 'step3p5', 'host': '0.0.0.0', 'model': '/app/models/models/vllm/Step-3.7-Flash-240REAP-MXFP416', 'trust_remote_code': True, 'max_model_len': 262144, 'served_model_name': ['model-large'], 'override_generation_config': {'max_tokens': 16384}, 'reasoning_parser': 'step3p5', 'tensor_parallel_size': 8, 'enable_expert_parallel': True, 'gpu_memory_utilization': 0.9, 'enable_prefix_caching': True, 'max_num_batched_tokens': 8192, 'max_num_seqs': 128}
(APIServer pid=1) `torch_dtype` is deprecated! Use `dtype` instead!
(APIServer pid=1) INFO 06-05 18:32:13 [model.py:617] Resolved architecture: Step3p7ForConditionalGeneration
(APIServer pid=1) WARNING 06-05 18:32:13 [model.py:2037] Your device 'AMD_Radeon_R9700' (with compute capability 12.0) doesn't support bfloat16. Falling back to torch.bfloat16 for compatibility.
(APIServer pid=1) WARNING 06-05 18:32:13 [model.py:2090] Casting bfloat16 to torch.bfloat16.
(APIServer pid=1) INFO 06-05 18:32:13 [model.py:1752] Using max model len 262144
(APIServer pid=1) WARNING 06-05 18:32:13 [quark_ocp_mx.py:142] AITER is not found or QuarkOCP_MX is not supported on the current platform. QuarkOCP_MX quantization will not be available.
(APIServer pid=1) INFO 06-05 18:32:14 [scheduler.py:239] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=1) INFO 06-05 18:32:14 [vllm.py:977] Asynchronous scheduling is enabled.
(APIServer pid=1) INFO 06-05 18:32:14 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=1) The tokenizer you are loading from '/app/models/models/vllm/Step-3.7-Flash-240REAP-MXFP416' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(EngineCore pid=670) INFO 06-05 18:32:26 [core.py:124] Initializing a V1 LLM engine (v0.22.69) with config: model='/app/models/models/vllm/Step-3.7-Flash-240REAP-MXFP416', speculative_config=None, tokenizer='/app/models/models/vllm/Step-3.7-Flash-240REAP-MXFP416', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=mxfp4_16, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='step3p5', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=model-large, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=670) WARNING 06-05 18:32:26 [multiproc_executor.py:1033] Reducing Torch parallelism from 128 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=670) INFO 06-05 18:32:26 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.17.0.4 (local), world_size=8, local_world_size=8
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!
The tokenizer you are loading from '/app/models/models/vllm/Step-3.7-Flash-240REAP-MXFP416' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
The tokenizer you are loading from '/app/models/models/vllm/Step-3.7-Flash-240REAP-MXFP416' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
The tokenizer you are loading from '/app/models/models/vllm/Step-3.7-Flash-240REAP-MXFP416' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
The tokenizer you are loading from '/app/models/models/vllm/Step-3.7-Flash-240REAP-MXFP416' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
The tokenizer you are loading from '/app/models/models/vllm/Step-3.7-Flash-240REAP-MXFP416' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
The tokenizer you are loading from '/app/models/models/vllm/Step-3.7-Flash-240REAP-MXFP416' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
The tokenizer you are loading from '/app/models/models/vllm/Step-3.7-Flash-240REAP-MXFP416' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
The tokenizer you are loading from '/app/models/models/vllm/Step-3.7-Flash-240REAP-MXFP416' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(Worker pid=940) INFO 06-05 18:32:37 [parallel_state.py:1422] world_size=8 rank=7 local_rank=7 distributed_init_method=tcp://127.0.0.1:52517 backend=nccl
(Worker pid=934) INFO 06-05 18:32:37 [parallel_state.py:1422] world_size=8 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:52517 backend=nccl
(Worker pid=938) INFO 06-05 18:32:37 [parallel_state.py:1422] world_size=8 rank=5 local_rank=5 distributed_init_method=tcp://127.0.0.1:52517 backend=nccl
(Worker pid=935) INFO 06-05 18:32:37 [parallel_state.py:1422] world_size=8 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:52517 backend=nccl
(Worker pid=939) INFO 06-05 18:32:37 [parallel_state.py:1422] world_size=8 rank=6 local_rank=6 distributed_init_method=tcp://127.0.0.1:52517 backend=nccl
(Worker pid=936) INFO 06-05 18:32:37 [parallel_state.py:1422] world_size=8 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:52517 backend=nccl
(Worker pid=933) INFO 06-05 18:32:37 [parallel_state.py:1422] world_size=8 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:52517 backend=nccl
(Worker pid=937) INFO 06-05 18:32:37 [parallel_state.py:1422] world_size=8 rank=4 local_rank=4 distributed_init_method=tcp://127.0.0.1:52517 backend=nccl
(Worker pid=933) INFO 06-05 18:32:38 [pynccl.py:113] vLLM is using nccl==2.27.7
(Worker pid=933) INFO 06-05 18:33:18 [cuda_communicator.py:238] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=933) INFO 06-05 18:33:20 [cuda_communicator.py:238] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=933) INFO 06-05 18:33:20 [parallel_state.py:1735] rank 0 in world size 8 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker_TP0_EP0 pid=933) INFO 06-05 18:33:20 [gpu_model_runner.py:5070] Starting to load model /app/models/models/vllm/Step-3.7-Flash-240REAP-MXFP416...
(Worker_TP0_EP0 pid=933) INFO 06-05 18:33:21 [rocm.py:629] Using Torch SDPA backend for ViT model.
(Worker_TP0_EP0 pid=933) INFO 06-05 18:33:21 [mm_encoder_attention.py:372] Using AttentionBackendEnum.TORCH_SDPA for MMEncoderAttention.
(Worker_TP4_EP4 pid=937) INFO 06-05 18:33:21 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(Worker_TP6_EP6 pid=939) INFO 06-05 18:33:21 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(Worker_TP3_EP3 pid=936) INFO 06-05 18:33:21 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(Worker_TP5_EP5 pid=938) INFO 06-05 18:33:21 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(Worker_TP1_EP1 pid=934) INFO 06-05 18:33:21 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(Worker_TP0_EP0 pid=933) INFO 06-05 18:33:21 [vllm.py:977] Asynchronous scheduling is enabled.
(Worker_TP0_EP0 pid=933) INFO 06-05 18:33:21 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(Worker_TP7_EP7 pid=940) INFO 06-05 18:33:21 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(Worker_TP0_EP0 pid=933) INFO 06-05 18:33:21 [__init__.py:527] Selected TritonFp8BlockScaledMMKernel for Fp8BlockLinearMethod
(Worker_TP2_EP2 pid=935) INFO 06-05 18:33:21 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(Worker_TP6_EP6 pid=939) INFO 06-05 18:33:21 [rocm.py:561] Found incompatible backend(s) [TURBOQUANT] with AttentionType.DECODER. Overriding with TRITON_ATTN out of potential backends: ['TRITON_ATTN', 'ROCM_ATTN'].
(Worker_TP4_EP4 pid=937) INFO 06-05 18:33:21 [rocm.py:561] Found incompatible backend(s) [TURBOQUANT] with AttentionType.DECODER. Overriding with TRITON_ATTN out of potential backends: ['TRITON_ATTN', 'ROCM_ATTN'].
(Worker_TP3_EP3 pid=936) INFO 06-05 18:33:21 [rocm.py:561] Found incompatible backend(s) [TURBOQUANT] with AttentionType.DECODER. Overriding with TRITON_ATTN out of potential backends: ['TRITON_ATTN', 'ROCM_ATTN'].
(Worker_TP5_EP5 pid=938) INFO 06-05 18:33:21 [rocm.py:561] Found incompatible backend(s) [TURBOQUANT] with AttentionType.DECODER. Overriding with TRITON_ATTN out of potential backends: ['TRITON_ATTN', 'ROCM_ATTN'].
(Worker_TP7_EP7 pid=940) INFO 06-05 18:33:21 [rocm.py:561] Found incompatible backend(s) [TURBOQUANT] with AttentionType.DECODER. Overriding with TRITON_ATTN out of potential backends: ['TRITON_ATTN', 'ROCM_ATTN'].
(Worker_TP0_EP0 pid=933) INFO 06-05 18:33:21 [rocm.py:561] Found incompatible backend(s) [TURBOQUANT] with AttentionType.DECODER. Overriding with TRITON_ATTN out of potential backends: ['TRITON_ATTN', 'ROCM_ATTN'].
(Worker_TP2_EP2 pid=935) INFO 06-05 18:33:21 [rocm.py:561] Found incompatible backend(s) [TURBOQUANT] with AttentionType.DECODER. Overriding with TRITON_ATTN out of potential backends: ['TRITON_ATTN', 'ROCM_ATTN'].
(Worker_TP1_EP1 pid=934) INFO 06-05 18:33:21 [rocm.py:561] Found incompatible backend(s) [TURBOQUANT] with AttentionType.DECODER. Overriding with TRITON_ATTN out of potential backends: ['TRITON_ATTN', 'ROCM_ATTN'].
(Worker_TP0_EP0 pid=933) WARNING 06-05 18:33:21 [step3p5.py:486] Enable custom fused all reduce...
(Worker_TP0_EP0 pid=933) INFO 06-05 18:33:22 [expert_map_manager.py:245] [EP Rank 0/8] Expert parallelism is enabled. Expert placement strategy: linear. Local/global number of experts: 30/240. Experts local to global index map: 0->0, 1->1, 2->2, 3->3, 4->4, 5->5, 6->6, 7->7, 8->8, 9->9, 10->10, 11->11, 12->12, 13->13, 14->14, 15->15, 16->16, 17->17, 18->18, 19->19, 20->20, 21->21, 22->22, 23->23, 24->24, 25->25, 26->26, 27->27, 28->28, 29->29.
(Worker_TP0_EP0 pid=933) WARNING 06-05 18:33:23 [compilation.py:1312] Op 'sparse_attn_indexer' not present in model, enabling with '+sparse_attn_indexer' has no effect
(Worker_TP0_EP0 pid=933) INFO 06-05 18:33:23 [weight_utils.py:922] Filesystem type for checkpoints: EXT4. Checkpoint size: 107.40 GiB. Available RAM: 168.89 GiB.
(Worker_TP0_EP0 pid=933) INFO 06-05 18:33:23 [weight_utils.py:945] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards:  95% 18/19 [00:46<00:03,  3.12s/it](Worker_TP4_EP4 pid=937) INFO 06-05 18:34:10 [mxfp4_16_kernels.py:58] MXFP4-16 using custom codebook: [-127.0, -104.0, -83.0, -65.0, -49.0, -35.0, -22.0, -10.0, 1.0, 13.0, 25.0, 38.0, 53.0, 69.0, 89.0, 113.0]
Loading safetensors checkpoint shards: 100% 19/19 [00:46<00:00,  2.47s/it]
(Worker_TP0_EP0 pid=933) INFO 06-05 18:34:10 [default_loader.py:397] Loading weights took 46.98 seconds
(Worker_TP0_EP0 pid=933) INFO 06-05 18:34:10 [mxfp4_16_kernels.py:58] MXFP4-16 using custom codebook: [-127.0, -104.0, -83.0, -65.0, -49.0, -35.0, -22.0, -10.0, 1.0, 13.0, 25.0, 38.0, 53.0, 69.0, 89.0, 113.0]
(Worker_TP5_EP5 pid=938) INFO 06-05 18:34:10 [mxfp4_16_kernels.py:58] MXFP4-16 using custom codebook: [-127.0, -104.0, -83.0, -65.0, -49.0, -35.0, -22.0, -10.0, 1.0, 13.0, 25.0, 38.0, 53.0, 69.0, 89.0, 113.0]
(Worker_TP3_EP3 pid=936) INFO 06-05 18:34:10 [mxfp4_16_kernels.py:58] MXFP4-16 using custom codebook: [-127.0, -104.0, -83.0, -65.0, -49.0, -35.0, -22.0, -10.0, 1.0, 13.0, 25.0, 38.0, 53.0, 69.0, 89.0, 113.0]
(Worker_TP6_EP6 pid=939) INFO 06-05 18:34:10 [mxfp4_16_kernels.py:58] MXFP4-16 using custom codebook: [-127.0, -104.0, -83.0, -65.0, -49.0, -35.0, -22.0, -10.0, 1.0, 13.0, 25.0, 38.0, 53.0, 69.0, 89.0, 113.0]
(Worker_TP7_EP7 pid=940) INFO 06-05 18:34:10 [mxfp4_16_kernels.py:58] MXFP4-16 using custom codebook: [-127.0, -104.0, -83.0, -65.0, -49.0, -35.0, -22.0, -10.0, 1.0, 13.0, 25.0, 38.0, 53.0, 69.0, 89.0, 113.0]
(Worker_TP1_EP1 pid=934) INFO 06-05 18:34:10 [mxfp4_16_kernels.py:58] MXFP4-16 using custom codebook: [-127.0, -104.0, -83.0, -65.0, -49.0, -35.0, -22.0, -10.0, 1.0, 13.0, 25.0, 38.0, 53.0, 69.0, 89.0, 113.0]
(Worker_TP2_EP2 pid=935) INFO 06-05 18:34:10 [mxfp4_16_kernels.py:58] MXFP4-16 using custom codebook: [-127.0, -104.0, -83.0, -65.0, -49.0, -35.0, -22.0, -10.0, 1.0, 13.0, 25.0, 38.0, 53.0, 69.0, 89.0, 113.0]
(Worker_TP0_EP0 pid=933) INFO 06-05 18:34:11 [gpu_model_runner.py:5165] Model loading took 13.64 GiB memory and 49.763474 seconds
(Worker_TP0_EP0 pid=933) INFO 06-05 18:34:12 [gpu_model_runner.py:6174] Encoder cache will be initialized with a budget of 8192 tokens, and profiled with 2 image items of the maximum feature size.
(EngineCore pid=670) INFO 06-05 18:35:12 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore pid=670) INFO 06-05 18:36:12 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore pid=670) INFO 06-05 18:37:12 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(EngineCore pid=670) INFO 06-05 18:38:12 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP0_EP0 pid=933) INFO 06-05 18:38:18 [backends.py:1089] Using cache directory: /root/.cache/vllm/torch_compile_cache/813c003774/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0_EP0 pid=933) INFO 06-05 18:38:18 [backends.py:1148] Dynamo bytecode transform time: 13.10 s
(Worker_TP0_EP0 pid=933) INFO 06-05 18:38:51 [backends.py:378] Cache the graph of compile range (1, 8192) for later use
(EngineCore pid=670) INFO 06-05 18:39:12 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP0_EP0 pid=933) INFO 06-05 18:39:41 [backends.py:393] Compiling a graph for compile range (1, 8192) takes 82.56 s
(Worker_TP3_EP3 pid=936) INFO 06-05 18:39:46 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=1280,K=4096,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP0_EP0 pid=933) INFO 06-05 18:39:46 [decorators.py:708] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/4b63da58c5b6d2c13efd8b083df17421049ee825b45c77458a80777814139a4d/rank_0_0/model
(Worker_TP0_EP0 pid=933) INFO 06-05 18:39:46 [monitor.py:53] torch.compile took 100.83 s in total
(Worker_TP4_EP4 pid=937) INFO 06-05 18:39:46 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=1280,K=4096,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP0_EP0 pid=933) INFO 06-05 18:39:46 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=1280,K=4096,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP1_EP1 pid=934) INFO 06-05 18:39:46 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=1280,K=4096,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP7_EP7 pid=940) INFO 06-05 18:39:46 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=1280,K=4096,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP5_EP5 pid=938) INFO 06-05 18:39:46 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=1280,K=4096,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP6_EP6 pid=939) INFO 06-05 18:39:46 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=1280,K=4096,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP3_EP3 pid=936) INFO 06-05 18:39:47 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1024,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP4_EP4 pid=937) INFO 06-05 18:39:47 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1024,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP0_EP0 pid=933) INFO 06-05 18:39:47 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1024,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP1_EP1 pid=934) INFO 06-05 18:39:47 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1024,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP7_EP7 pid=940) INFO 06-05 18:39:47 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1024,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP2_EP2 pid=935) INFO 06-05 18:39:47 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=1280,K=4096,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP5_EP5 pid=938) INFO 06-05 18:39:47 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1024,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP6_EP6 pid=939) INFO 06-05 18:39:47 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1024,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP2_EP2 pid=935) INFO 06-05 18:39:48 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1024,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP0_EP0 pid=933) INFO 06-05 18:39:49 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=2816,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP4_EP4 pid=937) INFO 06-05 18:39:49 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=2816,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP1_EP1 pid=934) INFO 06-05 18:39:49 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=2816,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP2_EP2 pid=935) INFO 06-05 18:39:49 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=2816,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP3_EP3 pid=936) INFO 06-05 18:39:49 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=2816,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP7_EP7 pid=940) INFO 06-05 18:39:49 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=2816,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP6_EP6 pid=939) INFO 06-05 18:39:49 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=2816,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP5_EP5 pid=938) INFO 06-05 18:39:49 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=2816,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP4_EP4 pid=937) INFO 06-05 18:39:50 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1408,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP1_EP1 pid=934) INFO 06-05 18:39:50 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1408,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP3_EP3 pid=936) INFO 06-05 18:39:50 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1408,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP7_EP7 pid=940) INFO 06-05 18:39:50 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1408,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP5_EP5 pid=938) INFO 06-05 18:39:50 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1408,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP2_EP2 pid=935) INFO 06-05 18:39:50 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1408,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP0_EP0 pid=933) INFO 06-05 18:39:50 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1408,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP6_EP6 pid=939) INFO 06-05 18:39:50 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1408,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP0_EP0 pid=933) INFO 06-05 18:39:51 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=1792,K=4096,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP1_EP1 pid=934) INFO 06-05 18:39:51 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=1792,K=4096,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP7_EP7 pid=940) INFO 06-05 18:39:51 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=1792,K=4096,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP4_EP4 pid=937) INFO 06-05 18:39:51 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=1792,K=4096,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP6_EP6 pid=939) INFO 06-05 18:39:51 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=1792,K=4096,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP5_EP5 pid=938) INFO 06-05 18:39:51 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=1792,K=4096,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP3_EP3 pid=936) INFO 06-05 18:39:51 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=1792,K=4096,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP2_EP2 pid=935) INFO 06-05 18:39:51 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=1792,K=4096,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP3_EP3 pid=936) INFO 06-05 18:39:52 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1536,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP0_EP0 pid=933) INFO 06-05 18:39:52 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1536,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP4_EP4 pid=937) INFO 06-05 18:39:52 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1536,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP2_EP2 pid=935) INFO 06-05 18:39:52 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1536,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP1_EP1 pid=934) INFO 06-05 18:39:52 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1536,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP6_EP6 pid=939) INFO 06-05 18:39:52 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1536,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP5_EP5 pid=938) INFO 06-05 18:39:52 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1536,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP7_EP7 pid=940) INFO 06-05 18:39:52 [fp8_utils.py:824] Using configuration from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1536,device_name=AMD_Radeon_R9700,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
(Worker_TP2_EP2 pid=935) INFO 06-05 18:39:53 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=320,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP3_EP3 pid=936) INFO 06-05 18:39:53 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=320,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP7_EP7 pid=940) INFO 06-05 18:39:53 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=320,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP0_EP0 pid=933) INFO 06-05 18:39:53 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=320,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP4_EP4 pid=937) INFO 06-05 18:39:53 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=320,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP1_EP1 pid=934) INFO 06-05 18:39:53 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=320,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP6_EP6 pid=939) INFO 06-05 18:39:53 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=320,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP5_EP5 pid=938) INFO 06-05 18:39:53 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=320,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP2_EP2 pid=935) INFO 06-05 18:39:57 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=160,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP7_EP7 pid=940) INFO 06-05 18:39:57 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=160,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP0_EP0 pid=933) INFO 06-05 18:39:57 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=160,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP3_EP3 pid=936) INFO 06-05 18:39:57 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=160,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP1_EP1 pid=934) INFO 06-05 18:39:57 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=160,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP4_EP4 pid=937) INFO 06-05 18:39:57 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=160,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP6_EP6 pid=939) INFO 06-05 18:39:57 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=160,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP5_EP5 pid=938) INFO 06-05 18:39:57 [mxfp4_16_kernels.py:83] Using tuned config from /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=160,device_name=AMD_Radeon_R9700,dtype=mxfp4_16,group_size=16.json for MXFP4-16 kernel.
(Worker_TP2_EP2 pid=935) WARNING 06-05 18:39:58 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=2560,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP0_EP0 pid=933) WARNING 06-05 18:39:58 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=2560,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP0_EP0 pid=933) WARNING 06-05 18:39:58 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1280,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP2_EP2 pid=935) WARNING 06-05 18:39:58 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1280,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP7_EP7 pid=940) WARNING 06-05 18:39:58 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=2560,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP7_EP7 pid=940) WARNING 06-05 18:39:58 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1280,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP1_EP1 pid=934) WARNING 06-05 18:39:58 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=2560,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP1_EP1 pid=934) WARNING 06-05 18:39:58 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1280,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP3_EP3 pid=936) WARNING 06-05 18:39:58 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=2560,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP3_EP3 pid=936) WARNING 06-05 18:39:58 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1280,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP6_EP6 pid=939) WARNING 06-05 18:39:58 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=2560,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP6_EP6 pid=939) WARNING 06-05 18:39:58 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1280,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP5_EP5 pid=938) WARNING 06-05 18:39:58 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=2560,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP5_EP5 pid=938) WARNING 06-05 18:39:58 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1280,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP4_EP4 pid=937) WARNING 06-05 18:39:58 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=2560,K=4096,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP4_EP4 pid=937) WARNING 06-05 18:39:58 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=4096,K=1280,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP0_EP0 pid=933) INFO 06-05 18:40:05 [monitor.py:81] Initial profiling/warmup run took 19.09 s
(EngineCore pid=670) INFO 06-05 18:40:12 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
(Worker_TP0_EP0 pid=933) INFO 06-05 18:40:13 [gpu_worker.py:469] Available KV cache memory: 12.33 GiB
(EngineCore pid=670) WARNING 06-05 18:40:13 [kv_cache_utils.py:1157] Add 3 padding layers, may waste at most 9.09% KV cache memory
(EngineCore pid=670) INFO 06-05 18:40:13 [kv_cache_utils.py:1733] GPU KV cache size: 1,959,264 tokens
(EngineCore pid=670) INFO 06-05 18:40:13 [kv_cache_utils.py:1734] Maximum concurrency for 262,144 tokens per request: 7.47x
(Worker_TP0_EP0 pid=933) INFO 06-05 18:40:13 [triton_attn.py:104] Using tuned unified-attention config from /moe_configs/device_name=AMD_Radeon_R9700,num_kv_heads=1,head_size=128,dtype=bf16.json
(Worker_TP0_EP0 pid=933) INFO 06-05 18:40:13 [triton_attn.py:162] Unified-attention tuning: max_model_len=262144 -> ctx point 200000 (segments=128 tile_decode=32 warps=4 stages=1 waves=3)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  94% 33/35 [00:57<00:06,  3.27s/it](EngineCore pid=670) INFO 06-05 18:41:14 [shm_broadcast.py:698] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100% 35/35 [01:08<00:00,  1.96s/it]
Capturing CUDA graphs (decode, FULL): 100% 19/19 [00:18<00:00,  1.05it/s]
(Worker_TP0_EP0 pid=933) INFO 06-05 18:41:42 [gpu_model_runner.py:6494] Graph capturing finished in 88 secs, took 3.43 GiB
(Worker_TP7_EP7 pid=940) INFO 06-05 18:41:42 [jit_monitor.py:54] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP4_EP4 pid=937) INFO 06-05 18:41:42 [jit_monitor.py:54] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP6_EP6 pid=939) INFO 06-05 18:41:42 [jit_monitor.py:54] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP5_EP5 pid=938) INFO 06-05 18:41:42 [jit_monitor.py:54] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP0_EP0 pid=933) INFO 06-05 18:41:42 [jit_monitor.py:54] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP1_EP1 pid=934) INFO 06-05 18:41:42 [jit_monitor.py:54] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP3_EP3 pid=936) INFO 06-05 18:41:42 [jit_monitor.py:54] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_TP2_EP2 pid=935) INFO 06-05 18:41:42 [jit_monitor.py:54] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=670) INFO 06-05 18:41:42 [core.py:320] init engine (profile, create kv cache, warmup model) took 450.98 s (compilation: 102.08 s)
(EngineCore pid=670) `torch_dtype` is deprecated! Use `dtype` instead!
(EngineCore pid=670) The tokenizer you are loading from '/app/models/models/vllm/Step-3.7-Flash-240REAP-MXFP416' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(EngineCore pid=670) INFO 06-05 18:41:44 [vllm.py:977] Asynchronous scheduling is enabled.
(EngineCore pid=670) INFO 06-05 18:41:44 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=1) INFO 06-05 18:41:44 [api_server.py:600] Supported tasks: ['generate']
(APIServer pid=1) INFO 06-05 18:41:44 [parser_manager.py:202] "auto" tool choice has been enabled.
(APIServer pid=1) INFO 06-05 18:41:44 [step3p5_tool_parser.py:1520] vLLM registered tool parser Step3p5ToolParser
(APIServer pid=1) The tokenizer you are loading from '/app/models/models/vllm/Step-3.7-Flash-240REAP-MXFP416' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(APIServer pid=1) INFO 06-05 18:41:45 [hf.py:488] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this.
(APIServer pid=1) INFO 06-05 18:41:46 [base.py:224] Multi-modal warmup completed in 0.859s
(APIServer pid=1) INFO 06-05 18:41:47 [base.py:224] Readonly multi-modal warmup completed in 0.807s
(APIServer pid=1) INFO 06-05 18:41:47 [api_server.py:604] Starting vLLM server on http://0.0.0.0:8000
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:37] Available routes are:
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /v1/pin, Methods: POST
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=1) INFO 06-05 18:41:47 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=1) INFO:     Started server process [1]
(APIServer pid=1) INFO:     Waiting for application startup.
(APIServer pid=1) INFO:     Application startup complete.
(APIServer pid=1) INFO:     172.17.0.1:36158 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     172.17.0.1:36162 - "GET / HTTP/1.1" 404 Not Found
(Worker_TP0_EP0 pid=933) WARNING 06-05 18:41:49 [jit_monitor.py:103] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(Worker_TP0_EP0 pid=933) WARNING 06-05 18:41:51 [jit_monitor.py:103] Triton kernel JIT compilation during inference: _w8a8_triton_block_scaled_mm. This causes a latency spike; consider extending warmup to cover this shape/config.
(Worker_TP0_EP0 pid=933) WARNING 06-05 18:41:52 [jit_monitor.py:103] Triton kernel JIT compilation during inference: kernel_unified_attention. This causes a latency spike; consider extending warmup to cover this shape/config.
(Worker_TP0_EP0 pid=933) WARNING 06-05 18:41:53 [jit_monitor.py:103] Triton kernel JIT compilation during inference: mxfp4_16_linear_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=1) INFO 06-05 18:41:57 [loggers.py:271] Engine 000: Avg prompt throughput: 51.0 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 06-05 18:42:07 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 06-05 18:42:17 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 06-05 18:42:27 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 06-05 18:42:37 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%

(APIServer pid=1) INFO 06-05 18:44:07 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 68.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 33.9%
(APIServer pid=1) INFO 06-05 18:44:17 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 68.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 33.9%
(APIServer pid=1) INFO 06-05 18:44:27 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 67.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 33.9%
(APIServer pid=1) INFO 06-05 18:44:37 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 68.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 33.9%
(APIServer pid=1) INFO:     172.17.0.1:33376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 06-05 18:44:47 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 46.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 33.9%
(APIServer pid=1) INFO 06-05 18:44:57 [loggers.py:271] Engine 000: Avg prompt throughput: 25.5 tokens/s, Avg generation throughput: 60.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 33.8%
(APIServer pid=1) INFO:     172.17.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 06-05 18:45:07 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 58.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 33.8%
(APIServer pid=1) INFO 06-05 18:45:17 [loggers.py:271] Engine 000: Avg prompt throughput: 22.8 tokens/s, Avg generation throughput: 58.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 39.4%
(APIServer pid=1) INFO 06-05 18:45:27 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 67.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 39.4%
(APIServer pid=1) INFO 06-05 18:45:37 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 68.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 39.4%
(APIServer pid=1) INFO 06-05 18:45:47 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 67.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 39.4%
(APIServer pid=1) INFO:     172.17.0.1:53454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 06-05 18:45:57 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 44.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 39.4%
(APIServer pid=1) INFO 06-05 18:46:07 [loggers.py:271] Engine 000: Avg prompt throughput: 47.9 tokens/s, Avg generation throughput: 38.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 36.6%
(APIServer pid=1) INFO 06-05 18:46:17 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 67.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 36.6%
(APIServer pid=1) INFO:     172.17.0.1:44668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 06-05 18:46:27 [loggers.py:271] Engine 000: Avg prompt throughput: 4.7 tokens/s, Avg generation throughput: 40.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 39.3%
(APIServer pid=1) INFO 06-05 18:46:37 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 39.3%
(APIServer pid=1) INFO 06-05 18:46:47 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 39.3%
(APIServer pid=1) INFO 06-05 18:46:57 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 39.3%
(APIServer pid=1) INFO 06-05 18:47:07 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 39.3%
(APIServer pid=1) INFO 06-05 18:47:17 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 39.3%
(APIServer pid=1) INFO 06-05 18:47:27 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 39.3%
(APIServer pid=1) INFO 06-05 18:47:37 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 39.3%
(APIServer pid=1) INFO 06-05 18:47:47 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 39.3%
(APIServer pid=1) INFO 06-05 18:47:57 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 39.3%
(APIServer pid=1) INFO 06-05 18:48:07 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 39.3%

Seem right, if you do tunableop tuning it should come up a bit and MTP should more than double throughput at MTP3. Your base throughput is roughly == what I'm getting without MTP, but I'm sure your prefill will murder mine :P Step is genuinely just a lot slower than Qwen3.6 series on decode.

The great this is how long it will hold out at good decode speeds now because of:
(Worker_TP0_EP0 pid=933) INFO 06-05 18:40:13 [triton_attn.py:104] Using tuned unified-attention config from /moe_configs/device_name=AMD_Radeon_R9700,num_kv_heads=1,head_size=128,dtype=bf16.json

I think you're paying a bit more tax on the multicard communication operations tax than I am unfortunately. Good news is... you could run this model in FP8 with full context and get both more speed (FP8 hardware acceleration and its a lighter kernel than mine) and better accuracy out of the 8 way setup.

i launch via llama swap -> ./run-large.sh model_name port

docker rm -f "$1-recached" 2>/dev/null || true

docker run --name "$1-recashed" \
  --rm --tty --ipc=host --shm-size=128g \
  --device /dev/kfd:/dev/kfd \
  --device /dev/dri/renderD128:/dev/dri/renderD128 \
  --device /dev/dri/renderD129:/dev/dri/renderD129 \
  --device /dev/dri/renderD130:/dev/dri/renderD130 \
  --device /dev/dri/renderD132:/dev/dri/renderD132 \
  --device /dev/dri/renderD137:/dev/dri/renderD137 \
  --device /dev/dri/renderD138:/dev/dri/renderD138 \
  --device /dev/dri/renderD139:/dev/dri/renderD139 \
  --device /dev/dri/renderD140:/dev/dri/renderD140 \
  --device /dev/mem:/dev/mem \
  -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e ROCR_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e VLLM_ROCM_USE_AITER=0 \
  -e TRUST_REMOTE_CODE=1 \
  -e VLLM_TUNED_CONFIG_FOLDER=/moe_configs \
  -v /mnt/tb_disk/llm:/app/models:ro \
  -e PYTORCH_TUNABLEOP_ENABLED=1 \
  -e PYTORCH_TUNABLEOP_TUNING=0 \
  -e PYTORCH_TUNABLEOP_RECORD_UNTUNED=0 \
  -e PYTORCH_ALLOC_CONF=expandable_segments:True \
  -e PYTORCH_HIP_ALLOC_CONF=expandable_segments:True \
  -e PYTORCH_TUNABLEOP_FILENAME=/tunableop/tunableop_merged.csv \
  -e PYTORCH_TUNABLEOP_UNTUNED_FILENAME=/tunableop/tunableop_untuned%%d.csv \
  -e GPU_MAX_HW_QUEUES=1 \
  -p "$2":8000 \
  tcclaviger/vllm22:latest \
  /app/models/models/vllm/Step-3.7-Flash-240REAP-MXFP416  \
  --served-model-name "$1" --host 0.0.0.0 --port 8000 --trust-remote-code \
  --tensor-parallel-size 8 \
  --disable-cascade-attn \
  --reasoning-parser step3p5 \
  --enable-auto-tool-choice --tool-call-parser step3p5 \
  --enable-prefix-caching --gpu-memory-utilization 0.9 --max-num-batched-tokens 8192 \
  --enable-expert-parallel --max-model-len 262144 --max-num-seqs 128  \
  --override-generation-config '{"max_tokens": 16384}' --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'

Seem right, if you do tunableop tuning it should come up a bit and MTP should more than double throughput at MTP3. Your base throughput is roughly == what I'm getting without MTP, but I'm sure your prefill will murder mine :P Step is genuinely just a lot slower than Qwen3.6 series on decode.

Today i launch MiniMax M2.7-FP8 (Original with 190k tokens), got 36-38 t/s output, but super small kv-cache space, obly for 1 request per time. I tried to create own MXFP4 quant for this model, but have not luck yet :)

The great this is how long it will hold out at good decode speeds now because of:
(Worker_TP0_EP0 pid=933) INFO 06-05 18:40:13 [triton_attn.py:104] Using tuned unified-attention config from /moe_configs/device_name=AMD_Radeon_R9700,num_kv_heads=1,head_size=128,dtype=bf16.json

Yes, thanks!

I think you're paying a bit more tax on the multicard communication operations tax than I am unfortunately. Good news is... you could run this model in FP8 with full context and get both more speed (FP8 hardware acceleration and its a lighter kernel than mine) and better accuracy out of the 8 way setup.

yes FP8 speed gives me same output speed, but, mtp does not work on them.

The better think we got with your first image is Qwen3.5-397B-A17B. i run it with 80k Context size and super fast speed of decoding.

2 days ago i rewrite kernels via AI to launch DeepSeek V4 Flash, but got only 12-14 t/s per request output speed, input was normal as waited.

Today situation for 8x gpu is more different, i can put here 200-220B model with FP8 and normal context and up to 400B Model with 64k context window.

Good context window for us is around 200k, it's minimal we need for stable usage agentic tasks, i think i understood why you looking for qwen3.6, not for qwen3.5-122b.

MTP gives me 99.1 t/s generation on 1 req. that is crazy!

What do you thing, will it work on our kernels or it is impossible to launch?

DeepSeek V4 Flash MXFP4: https://huggingface.co/haanjack/DeepSeek-V4-Flash-MXFP4/blob/main/config.json

bench mtp 3

The MXFP416 quantizer i have yet to publish, but I will, and will likely rename the quant to avoid confusion with standard mxfp4, they're quite different (thinking RNL4).

~90-100 tps is exactly where it should be, so the configs did work to speed your system up a good bit, great to have verification on that, regarding your FP8 throughput, I also have a fix for that I will bake in eventually. FP8 throughput should be, hands down, the fastest possible format on R9700 setups, the problem is specifically, the FP8 kernel included in the mainline vllm image. I was working on porting in mxfp4 and fp8 custom kernels that are much much faster but want to get the self-tuning baked in so people can tune fheir configs for all quants on their own for R9700.

MTP not working is almost certainly the quantizer script simply not copying them over on the FP8 generated safetensors, there's no reason the BF16 MTP heads can't work with FP8 Block 128 quantization.

In order my roadmap for my vllm22 branch is:

• Get autotuner released for RDNA4 baked in as a callable script. Done but Needs validation
• Get mxfp416 quantizer built in as a callable script so anyone can quantize their own models. Done but Needs validation
• Get the attention kernel tuenr added to the linear/moe tuner. Done, deployed.
• Add definitions for Gemma 4, Qwen 3, Deepseek 4, Minimax M2.7 to various locations, attention, quantization, and config tuning scripts (models not mentioned are not interesting enough for me to work on).
• Get the mxfp4/fp8/nvfp4 kernels added in. Done but Needs validation
• Expand the mxfp4/fp8/nvfp4 kernels to accept config files and add them to the autotuner.
• Expand the quantizer to do quick NVFP4 and FP8 Block 128 quantization.
• Start work on DS4 so I have a solid understanding of what integrating the new architectural pieces it adds looks like. For my use, I need to prune it a bit, and adjust the quantization scheme. Will be a somewhat large effort due to new architecture.
• Finally, at some point, rewrite the ROCM ATTN c++ kernel to not be constrained to log 2 (this is why the ultra slow fallback kernel is used on most models on RDNA4) and use 3D tensors, similar to the Triton unified attention.
• Maybe someday, add activation aware quantization for mxfp4/nvfp4/FP8B128/MXFP416.

With these changes in my image, RDNA4 is finally memory bandwidth bound instead of blocking software bound, so we're fairly close to the full potential of the cards based on the runtime profiling I've done.

Performance gap that could see meaningful uplift is allgather and allreduce, its ~35% of my time when I profile during prefill, and its using the slowest possible path in vllm for a variety of reasons. I took an initial shot at it and ran into a hardware issue on RDNA4 that is a blocking issue. I can get around it, but it increases the effort to do so like 10x, basically forcing me to alter the driver and rccl which...I am reluctant to do right now (too many other things to work on to take a week or two dive into that).

I am undecided if I want to release my other custom quant scheme, it's technically better in every way (speed, size, MSE, perpl) but I want to publish the paper on it first so there's no attribution questions about where it came from.

bench mtp 3

It may be that I need to reprofile the attention kernel to improve the concurrency, another thing on the list of things I need to work on :P

I remember now, it overflows out of my tuned config and falls back on the default TRITON ATTN kernel, if I were to vary be batch size it would break cuda graphing, not good. I am working on a bypass to fix concurrency. Interesting, I expect this will provide very good uplift on single request speed as well...standby 😔

i made quantization of MXFP4_16 for Minimax M2.7 it's amazing! Much better than normal mxfp4, can i ask you create one more pack of kernal configs?

(Worker_TP1_EP1 pid=938) WARNING 06-06 02:57:46 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=3072,K=3072,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP6_EP6 pid=943) WARNING 06-06 02:57:46 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=3072,K=3072,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP3_EP3 pid=940) WARNING 06-06 02:57:46 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=3072,K=3072,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP7_EP7 pid=944) WARNING 06-06 02:57:46 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=3072,K=3072,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP5_EP5 pid=942) WARNING 06-06 02:57:46 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=3072,K=3072,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP2_EP2 pid=939) WARNING 06-06 02:57:46 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=3072,K=3072,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP6_EP6 pid=943) WARNING 06-06 02:57:46 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=3072,K=1536,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP5_EP5 pid=942) WARNING 06-06 02:57:46 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=3072,K=1536,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP2_EP2 pid=939) WARNING 06-06 02:57:46 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=3072,K=1536,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP3_EP3 pid=940) WARNING 06-06 02:57:46 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=3072,K=1536,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP7_EP7 pid=944) WARNING 06-06 02:57:46 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=3072,K=1536,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP1_EP1 pid=938) WARNING 06-06 02:57:46 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=3072,K=1536,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP0_EP0 pid=937) WARNING 06-06 02:57:46 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=3072,K=3072,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP0_EP0 pid=937) WARNING 06-06 02:57:46 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=3072,K=1536,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP4_EP4 pid=941) WARNING 06-06 02:57:46 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=3072,K=3072,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json
(Worker_TP4_EP4 pid=941) WARNING 06-06 02:57:46 [mxfp4_16_kernels.py:86] Using default MXFP4-16 kernel config. Performance might be sub-optimal! Config file not found at /app/vllm/vllm/model_executor/layers/quantization/utils/configs/N=3072,K=1536,device_name=AMD_Radeon_R9700,dtype=mxfp4_16_moe,group_size=16.json

also got his obe:
(Worker_TP0_EP0 pid=937) WARNING 06-06 02:57:58 [triton_attn.py:129] Using default unified-attention launch config (no config for device_name=AMD_Radeon_R9700,num_kv_heads=1,head_size=128,dtype=bf16). Performance may be sub-optimal! Run the offline tuner (patches/tools/tune_triton_attn.py) and drop the resulting JSON in the attention configs/ dir to tune for this device/shape.

With these changes in my image, RDNA4 is finally memory bandwidth bound instead of blocking software bound, so we're fairly close to the full potential of the cards based on the runtime profiling I've done.

Performance gap that could see meaningful uplift is allgather and allreduce, its ~35% of my time when I profile during prefill, and its using the slowest possible path in vllm for a variety of reasons. I took an initial shot at it and ran into a hardware issue on RDNA4 that is a blocking issue. I can get around it, but it increases the effort to do so like 10x, basically forcing me to alter the driver and rccl which...I am reluctant to do right now (too many other things to work on to take a week or two dive into that).

I am undecided if I want to release my other custom quant scheme, it's technically better in every way (speed, size, MSE, perpl) but I want to publish the paper on it first so there's no attribution questions about where it came from.

You have an incredible amount of free time! What a brilliant plan!

Isn't there a warning that everything you're doing will become obsolete in a short time? For example, the models will become obsolete, or something like that.

I was able to get deepseek to write a quantizer for your new build, and it named it IQ4_NL. I can't even explain why, but it eventually worked. After four hours of trying to quantize, one attempt resulted in deepseek creating a quantizer for your old mxfp4 build :).

But the second attempt—40 minutes of quantizing, and several more hours of debugging—resulted in the minimax-m2 launcher having to be rewritten for the patch to apply.

Yeah I'll get it up soon.

I have a big win on the attention system, I did a lot of profiling last nigth and was able to identify some concrete patterns in results, that allowed me to isolate the narrow set of values that need testing to get ideal attention kernel tuning for a given setup. I built an auto-profiler as part of the startup for both decode and attention values. When you start the models, you'll see new progress bars like this:

(EngineCore pid=137) INFO 06-06 15:32:22 [kv_cache_utils.py:1734] Maximum concurrency for 200,000 tokens per request: 1.21x
(Worker_TP0 pid=163) INFO 06-06 15:32:23 [attn_autotune.py:395] attn autotune: M=4 grid_base=24 pred_seg=128 -> seg=128 waves=1 (tile=32 warps=1) 0.6097 ms
(Worker_TP0 pid=163) INFO 06-06 15:32:23 [attn_autotune.py:395] attn autotune: M=8 grid_base=48 pred_seg=64 -> seg=128 waves=1 (tile=32 warps=1) 1.0321 ms
Autotuning unified-attention (decode, 3D): 100%|██████████| 2/2 [00:01<00:00, 1.45bucket/s]
Autotuning unified-attention (prefill, 2D): 100%|██████████| 27/27 [00:07<00:00, 3.74cfg/s]
(Worker_TP0 pid=163) INFO 06-06 15:32:30 [attn_autotune.py:273] prefill autotune (qlen=2048, kv=8192): tile_2d=32 warps=1 (pinned) stages=1 waves=1 -> 2.3914 ms
(Worker_TP0 pid=163) INFO 06-06 15:32:30 [attn_autotune.py:317] attn autotune results dumped to /attn_autotune_dump/autotune_AMD_Radeon_R9700_kv2_q16_h128_bfloat16_kvfp8_ql4.json
(Worker_TP0 pid=163) INFO 06-06 15:32:30 [triton_attn.py:361] Unified-attention autotune skipped for windowed group (sliding_window=512, attention_chunk_size=None); using defaults.

The result is, no need to bake in "best" tuned configs, it simply tunes them based on actual runtime values at server startup, the time penalty is not as big as it seems as the triton compile is done earlier, so the cuda graphing reuses it and we only pay that price once.

A new container going up shortly that has the auto profiling built in, hope it helps. I validated autotuner results against my offline ideal results and there is no throughput difference.

I am working on the concurrency issue on Step 3.7 and in general, hopefully this helps your deployment with it.

Also there's a missing step in MXFP4_16 that the LLMs wont know to do that significantly lifts the quality, standby, Ill get the quantizer in, its a bounded search that significantly lowers MSE at quantization time.

The config profiler will be built into the tcclaviger/vllm22:latest image on my next push, I am currently validating the functionality/correctness to guard against unsupported quant types.

Using the new docker image.

Amazing, thank you!!!

just download new version of image, will share some results soon

It should improve perf for other models as well, the only real issue might come from tensor naming conventions but it should be fairly resistant to that as it reads the vllm loading data. To start it will just be FP8 W8A8 Block 128 quants and MXFP416 tuning, as I get nvfp4, mxfp4, and gptq/awq built I'll incorporate them (they need correctness verification against my attention tuning assumptions about sequences, waves, steps, etc first).

Validating the config tuner and quantizer now for correctness, once done I'll push the update.
All visible using the man command to get guides, and if you want to add new models, they simply need to be added to the model_registry.py to define the shape for quant, any LLM will be able to read it and compare to the safetensors/config.json to show how to update it.

Quantizer will have various modes to suite various system setups, cpu, all gpus, per gpu, and a memory tight gpu mode that keeps system ram use to a minimum for typical gaming pc or small system use.

i got problem with one of GPU, it got 105-110 C, i will fix it and come back, now via LACT i limit power to 210W for each GPU

update image gain, still testing from my side.

Do you think we can do DFlash with MXFP4_16?

like this: https://mimo.xiaomi.com/blog/mimo-tilert-1000tps

Possibly but there are a few things I want to add in before dflash is on the table.

https://huggingface.co/djdeniro/MiniMax-M2.7-MXFP416/blob/main/docs/vllm_deploy_guide.md

i upload M2.7 MXFP416, if you will got 8 GPU's or anyone want to use some it will help

https://huggingface.co/djdeniro/MiniMax-M2.7-MXFP416/blob/main/docs/vllm_deploy_guide.md

i upload M2.7 MXFP416, if you will got 8 GPU's or anyone want to use some it will help

Did you use the quantizer in the new image? Curious how it worked for you. I'm waiting on M3 to drop, super excited for it, I love the MiniMax models I just wish they have VL capability.

I am expanding MXFP4_16 to have an optional parameter, a rotation of the weights before building the scales/binning the values. It is a massive leap in accuracy vs current implementation but it pays a -8% speed tax, so before I publish I need to sort that out more.

Rotation vs no-rotation quantized versions will maintain backwards compatibility via an optional flag, but, with rotation Qwen3.6 27B in MXFP4_16Rot scores PPL scores ~equal the to full FP16 model!

Did you use the quantizer in the new image? Curious how it worked for you. I'm waiting on M3 to drop, super excited for it, I love the MiniMax models I just wish they have VL capability.

No, i ask DeepSeek V4 Pro to make quantization via opencode. and after 5-10 hours i did it. Quantization tooks around 10 minutes, but trys to quantize is was not completed from old image.

I am expanding MXFP4_16 to have an optional parameter, a rotation of the weights before building the scales/binning the values. It is a massive leap in accuracy vs current implementation but it pays a -8% speed tax, so before I publish I need to sort that out more.

i think we need dynamic quants, based on amount of GPU, for example, i don't need to quantize all of layers into mxfp4 for MiniMax-M2.7 - so i need to quantize half of safetensors 🤗

Rotation vs no-rotation quantized versions will maintain backwards compatibility via an optional flag, but, with rotation Qwen3.6 27B in MXFP4_16Rot scores PPL scores ~equal the to full FP16 model!

Amazing!

This i need to test my MiniMax M2.7 also, how you did test with WikiText?

I solved the speed issue, with rotation is now ~free on speed.

I'll pack the PPL test into the :dev image in the future so it can be duplicated and we're all on even footing, I don't like the lm-evlauation perp test as it is not directly comparable with llama.cpp and mine is.

If you look into or ask a model to have a look in the :latest image and find the /tools/mxfp4_16_quant/model_registry.py, and from there it should be able to figure out how to register new models for quantization, it just needs the weigh definitions/names etc.

If you launch the container with --tune, you can tune the GEMMS for an TP target size using any number of GPUs, and the --quantize flag will launch it as a quantizer. Both have help/man available to explain how to use them. I recommend gpu-streaming flag so you get gpu quantizing but without blowing up memory use on VRAM or system RAM.

Happy tuning! Once rotation variant is validated fully I'll publish a new :latest so everyone can enjoy improved accuracy for ~~free (will also republish all my mxfp416 repos with rotation included, it will become the default for the quant).

tuning not creating correct json.

docker run --name "testtuning" \
  --rm --tty --ipc=host --shm-size=128g \
  --device /dev/kfd:/dev/kfd \
  --device /dev/dri/renderD128:/dev/dri/renderD128 \
  --device /dev/dri/renderD129:/dev/dri/renderD129 \
  --device /dev/dri/renderD130:/dev/dri/renderD130 \
  --device /dev/dri/renderD132:/dev/dri/renderD132 \
  --device /dev/dri/renderD137:/dev/dri/renderD137 \
  --device /dev/dri/renderD138:/dev/dri/renderD138 \
  --device /dev/dri/renderD139:/dev/dri/renderD139 \
  --device /dev/dri/renderD140:/dev/dri/renderD140 \
  --device /dev/mem:/dev/mem \
  -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e ROCR_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e VLLM_ROCM_USE_AITER=0 \
  -e TRUST_REMOTE_CODE=1 \
  -v /mnt/tb_disk/llm:/app/models:ro \
  -v /home/user/scripts/moe_configs_best_one:/app/best_tune \
  --group-add video \
  tcclaviger/vllm22:latest --tune --modelpath /app/models/models/vllm/MiniMax-M2.7-MXFP416  --tp 8  --outputpath /app/best_tune

i run like this, where is my mistake?

i did MiniMax M3 Quantization with REAP 15% experts https://huggingface.co/djdeniro/MiniMax-M3-MXFP4_16 (uploading in background)

Robert, is it possible to make IQ4_XS Quant for largest models?

Out of town at the moment.

My suggestion is to switch to the other format now in the latest, RFP458.
10% smaller.
~15% faster throughput.
Better accuracy.
Quant process for rfp458 give indications of supers found in logs.
Mixed precision super expert protection using fp16 per-expert pairs.

As for why it's not using your configs, volume mount an output path, then user a docker file to make a simple copy overlay image that bakes them into the folder they go in alongside the other configs. Ill have a look when I get home but no PC here.

Thanks! Don't worry, I'll figure out a way to adjust the tuning. Thanks for the quantization tip, I'll give it a try!

I also want to try using the DeepSeek V4 Flash.

I also tried the GLM-5.1-REAP-50, but it still hasn't worked. I'll let you know if it works :)

How are you? i see new version, what's new?

Sorry I've been consumed with work and this on the side.

For your setup a solution to Minimax M3 serving is nearly ready, soon... then a RFA_S (see below for what RFA_S is), should be able to squeeze into your box (might need like 4-8 experts pruned out).

For tuning, be sure to output them into a folder in the model directory, then volume mount them into the location vllm looks for them when loading the image (you can just stomp on the contents there, nothing there needs to exist that overwriting with the volume mount will break).

I've integrated Minimax M3 serving code into :dev (still untested, quantizing a model as a I type to validate it), trimmed the image of all the cruft (from 35gb down to 10 before triton cache insertions), recompiled to get the newer C primitives indie.

Found some serious hidden, latent GDN attention bugs that i fixed, and a landmine bug that caused the loaded models to hang under certain conditions.

Kvarn is now inside, but only works in some scenarios (GDN and full attention, SWA models are kind of broken with it). k4v4 and k4v2 work quite well, but they're touch about the quantitation of the model they're applied too, and in the end, for my use scenario kv in fp8 is == kv in k4v4 space usage because of the kvarn buffers. For setups with more overhead, its an easy way to get huge kv cache space.

Implemented vMLX JANG2_L quantizer, tuner, and kernels in :dev, untested (waiting on M3 quant into JANG2_L to finish so I can validate both).

Built out RXF, Robs Xtra Fast quant which...isn't ,lmao, abandonware I need to extract from the image as it simply didn't work out as I had envisioned.

RFP458 is the definitive 4bit quant when accuracy is the priority, particularly for MoE's, it allows preservation of super experts in FP16. On paper, by MSE or other activation blind measurements, MXFP416 rotated or not, is ~= to RFP458, however, RFP458 protects outliers better and is ~identical on serving speed. It also does well when using it to quantize attention weights, Qwen3.6 27B can be quantized down to 20.6 GB and still be indiscernible on MMLU, logprobs, perplexity, code needle and the lm-evaluation agentic tool testing, paired with kv in FP8 you can fit 27B on 1 R9700 with 150k context.

PLUS: I've ported all of it to Strix Halo 😔 Allas, 395+ simply doesn't have the juice to serve via vllm effectively imho.

Short list is:
Validate RX2, the 2 bit affine scalar quantization format.
Validate M3 inference.
Requantize all the models I care about into RFP458 + RFP458 attention and FP8 attention variants and publish, plus fully rename RFP458 across all code RFA, so I can establish my naming convention.
RFA_S 4.512bpw mlp weights + attention weights
RFA_M 4.512bpw mlp weights + FP8 attention
RFA_L 4.512bpw mlp weights + FP16 attention

Later list is:
Finish RF2, the vector based quantization format I'm 1/2 way done with that can scale from true 2bpw to ~4bpw and break the scalar quantization accuracy barrier.

You have nothing to apologize for! I'm just writing here because I have no other way to contact you! And I haven't yet met anyone else online who develops with the same patience as you! I admire you!

Confirmed.

Initial testing:
M3 works.
JANG2_L adapted to vllm for ROCM on triton attention kernel works.

Model is coherent and not a dunce, more testing, but :dev image going up that incorporates all of it, will publish model after further investigation, but initial test of:
https://huggingface.co/JANGQ-AI/MiniMax-M3-REAP22-Coder is quite good.

i will try to launch it! I think i can run it without pruning from Cerebras, because i have 256GB of VRAM?

Hey Rob - just made an account so I could thank you personally. I recently put a 4x R9700 rig together and have spent countless hours rolling through different serving solutions, bug fixing, bug fixing more.... finally landing on your stable VLLM image. Your posts on reddit and pervasive efforts in squeezing every last drop out of these cards is so incredibly appreciated, thank you. I've been using this MXFP416 model for a few weeks and it's been running super well.

Hitting the 128gb breakpoint was super important for me because I know how much better models can get at this parameter number, but of course good quantizations of 2-400B models would be incredible...

But I was also following this awesome discussion between you two trying to learn everything I can, and your dive into this RFP458 quant and even including the quantizer so we can do it locally - man seriously, thank you for your amazing work - I just got it up and running:

• Downloaded your 240REAP bf16
• Ran the quantizer in your latest image (well, pulled it 2 days ago) to get the RFP458
• ~102GB so ~6GB leaner than the MXFP416 on disk
• Maxing out at ~90tg/s which is incredible

Can't wait to try out other big models through this quantizer - definitely looking at M3 so will keep an eye out on the image you have coming through, and the progress here.

If there's any way I can give back and help out by donating some computing or tp4 testing, do let me know.

Hey Robert! Can yoy share flags how you launch m3 version? i try many times, no luck with dev image, deepseek v4 pro, helps me, but after 8 times try's they give up 😀

I pulled MiniMax-M3-REAP22-Coder from HF and trying to serve it
with tcclaviger/vllm22:dev using:

--enable-expert-parallel --block-size 128 --quantization rfp458

The RFP458 quant config is picked up correctly, MoE backend inits fine,
but weight loading fails with:

ValueError: There is no module or parameter named 'lm_head.biases'
in MiniMaxM3SparseForCausalLM. The available parameters belonging
to lm_head (ParallelLMHead) are: {'lm_head.weight'}

The safetensors index has ~1100 bias params (.bias and .biases)
that vLLM's minimax_m3 model code doesn't seem to support.
Looks like the quantized checkpoint packs per-channel biases
but the vLLM implementation doesn't expect them.

Am I missing some launch flags, or is this still WIP on the :dev image?

Hey Robert, long time no see.

Just launched: Ornith-1.0-397B-RFP458-REAP22, 37 t/s with 225W power limited 8xGPU's. using with latest version of your build! it's amazing!

Thanks all. Just glad I can help sort through the madness of getting these cards to do what they should.

I have been developing a new dynamic quantization that allows variable bit widths, and executes on the matrix course with native 8-bit instead of dequant kernels.

RFP458 has been renamed RFA, it essentially pushes the limits of what a scaler 4-bit quantization can achieve so that survives.

The design inspired a new variable bit width quantization scheme named RFI. RFI supports two-bit 6-bit and 8-bit. RFI two bit is more accurate than affine 2 bit mlx used by 50%.

RFI 6bit is more accurate than FP8, preserving tails and outliers about 40% better whilst avoiding crushing the typical values, the majority by using a 8-bit mantesa group 32.

RFI 8-bit is almost BF16 accuracy, it is significantly ahead of FP8 and the built-in INT8 and VLM. Both RFI 8-bit and RFI 6-bit can now be used for kv-cache-dtype, though only 8-bit is completely stable right now.

I will be publishing a new step 3.7 model probably replacing this one in the repo because I've been able to achieve smaller size with higher accuracy using the RFI quantization with no speed regression.

The :latest image is well behind dev, but dev has too many unfinished things to build latest right now.

RFI 2,6,8 and RFA, will be what I quantize into going forward, it's hands down better.

RFI aligns well to WMMA gemm dimensions and all use Group32, Hadmard rotation, int8 per-group mantissa, and channel wide int8 exponent.

It is "not" gated for only RDNA4, only int8 codepath is behind a guard so it should work on most cards.

I crushed Qwen3.6 35B down into 2 bit experts and RFI8 attention/vision/MTP, whilst preserving norms, head, etc as FP16, crushed down to 15.6GB, still ~9k prefill and 160-240 decode, and still completely functional. Code needle, agentic tool testing, chat, coding it all works still. I' dont advise 2 bit if you don't need it, but it's viable, shockingly.

Now that I've nailed down the data schemas I'll start fixing some of the semi-working stuff and see if I can get a ~90% performance solution without people needing to tune for a given model/dimension.

Oh and dev also has an improved FP8 native kernel optimized for RDNA4's additional tuning options the basic one doesn't offer.

NVFP4/MXFP4 are completely obsolete at this point so they're very low on the list to port over.

NOTE: Minimax M3 in my image uses a reimplemented tool call parser, the original is in rust, simply ported to python. Few other tweaks to accommodate M3, consider it...experimental still but it does work.

Hey Robert! Can yoy share flags how you launch m3 version? i try many times, no luck with dev image, deepseek v4 pro, helps me, but after 8 times try's they give up 😀

I pulled MiniMax-M3-REAP22-Coder from HF and trying to serve it
with tcclaviger/vllm22:dev using:

--enable-expert-parallel --block-size 128 --quantization rfp458

The RFP458 quant config is picked up correctly, MoE backend inits fine,
but weight loading fails with:

ValueError: There is no module or parameter named 'lm_head.biases'
in MiniMaxM3SparseForCausalLM. The available parameters belonging
to lm_head (ParallelLMHead) are: {'lm_head.weight'}

The safetensors index has ~1100 bias params (.bias and .biases)
that vLLM's minimax_m3 model code doesn't seem to support.
Looks like the quantized checkpoint packs per-channel biases
but the vLLM implementation doesn't expect them.

Am I missing some launch flags, or is this still WIP on the :dev image?

It's a recompile required issue, M3 introduced a new silu function in the c code. Try today's :dev, if it won't launch delete the dev image and clear docker cache then download it again so it picks up the recompiled base version.