nunchaku-lite-kernels

Prebuilt kernels for Nunchaku Lite, which runs SVDQuant (Nunchaku) INT4 / NVFP4 diffusion checkpoints (FLUX.1, FLUX.2 Klein, Qwen-Image, Z-Image, SDXL) in Diffusers pipelines. Nunchaku Lite loads this repository automatically through the kernels library; you only need it directly to call the ops yourself.

  • NVIDIA GPUs: W4A4 GEMMs (INT4 and NVFP4) with the fused SVD low-rank branch, AWQ W4A16 GEMM/GEMV, and fused modulation, q/k norm + RoPE and attention kernels.
  • Apple silicon (PyTorch MPS): W8A8 / W4A8 GEMMs on the M5 Neural Accelerators with the low-rank branch in the epilogue, plus fused q/k norm + RoPE and Neural Accelerator flash attention.

Builds

Backend PyTorch Variants
CUDA (Linux x86_64) 2.11-2.15 CUDA 12.6-13.4; compute capability 7.5, 8.0, 8.6, 8.9, 12.0a, 12.1a
Metal (macOS, arm64) 2.10-2.15 macOS 26+ (Metal 4); the MPS GEMMs and attention need an M5-class GPU

How to use

With Nunchaku Lite nothing is needed: load_nunchaku_pipeline(...) fetches the matching build. To call the ops directly:

# pip install -U kernels
from kernels import get_kernel

kernels_module = get_kernel("rootonchair/nunchaku-lite-kernels", version=2, trust_remote_code=True)
ops = kernels_module.ops

# Apple silicon (M5): Neural Accelerator flash attention on (B, L, H, D) tensors
import torch

q, k, v = (torch.randn(1, 4096, 24, 128, device="mps", dtype=torch.bfloat16) for _ in range(3))
out = ops.attention_mps(q, k, v, None, 128**-0.5)  # (1, 4096, 24, 128)

Available functions

All ops are registered with torch.library (with fake implementations for torch.compile).

CUDA:

Op Purpose
gemm_w4a4 SVDQuant W4A4 GEMM (INT4 or NVFP4) with the low-rank branch and fused epilogues (q/k norm, RoPE, GELU)
quantize_w4a4_act_fuse_lora Activation quantization fused with the low-rank down projection
awq_gemm_w4a16_g128_int16, awq_gemm_w4a16_g64_int32, gemv_awq AWQ W4A16 GEMM / GEMV
fused_rms_norm_modulate, fused_affine_modulate AdaLN modulation
fused_cross_head_qk_norm_rope q/k RMSNorm + RoPE
attention_fp16 FP16 attention

Python helpers: svdq_gemm_w4a4_cuda, svdq_quantize_w4a4_act_fuse_lora_cuda, awq_gemv_w4a16_cuda, attention_fp16_cuda.

Metal (MPS):

Op Purpose
svdq_quantize_act_int8 Per-token int8 activation quantization with smoothing and shift
svdq_gemm_w8a8 int8 x int8 GEMM on the Neural Accelerators; scales, bias and low-rank branch in the epilogue
svdq_gemm_w4a8 Same with 4-bit weights expanded in the kernel (about half the weight memory)
qk_norm_rope_mps_ q/k RMSNorm + RoPE + layout change in one pass; reads column slices of fused projections, works in place
attention_mps Flash attention on the Neural Accelerators (head dim 64/96/128, grouped-query, bool or additive masks)
svdq_qk_norm_rope_ Earlier in-place q/k norm + RoPE, kept for older Nunchaku Lite releases

Shapes and dtypes are documented in svdq_metal.hpp in the source.

Versions

Releases stay on the v2 branch while changes are backward compatible (new ops, faster kernels). Nunchaku Lite pins the version it was tested with.

Benchmarks

Run kernels benchmark rootonchair/nunchaku-lite-kernels --version 2.

License and credits

Apache-2.0. The CUDA kernels derive from Nunchaku (Apache-2.0). attention_mps is MLX's Neural Accelerator flash attention from MLX (MIT, Copyright © 2023 Apple Inc.; the notice ships inside the compiled kernel source).

Downloads last month
439
kernels
diffusion
quantization
svdquant
metal
cuda
apache-2.0
Supported hardwares new
CUDA
7.58.08.68.912.0a12.1a
GPU
RTX Spark N1X Laptop
24GB
GPU
RTX Spark N1X Desktop
128GB
DGX Spark
GB10
64GB
GPU
RTX PRO 6000 WS
96GB
GPU
RTX PRO 6000 Max-Q
96GB
GPU
RTX PRO 5000
48GB
GPU
RTX PRO 4500 WS
32GB
RTX
RTX 5090
32GB
RTX
RTX 5090 D
32GB
GPU
RTX PRO 4000
24GB
GPU
RTX PRO 4000 SFF
24GB
RTX
RTX 5090 Mobile
24GB
GPU
RTX PRO 2000
16GB
RTX
RTX 5080
16GB
RTX
RTX 5080 Mobile
16GB
RTX
RTX 5070 Ti
16GB
RTX
RTX 5060 Ti
16GB
RTX
RTX 5070
12GB
RTX
RTX 5070 Mobile
8GB
RTX
RTX 5070 Ti Mobile
12GB
RTX
RTX 5060
8GB
RTX
RTX 5060 Mobile
8GB
RTX
RTX 5050
8GB
RTX
RTX 5050 Mobile
8GB
GPU
RTX 5880 Ada
48GB
GPU
RTX 2000 Ada
16GB
RTX
RTX 4080 SUPER
16GB
RTX
RTX 4070 Ti Super
16GB
RTX
RTX 4070 Super
12GB
GPU
L40s
48GB
GPU
L20
48GB
RTX
RTX 4090D
24GB
RTX
RTX 5000 Ada
32GB
GPU
L4
24GB
GPU
RTX 4500 Ada
24GB
RTX
RTX 4000 Ada
20GB
RTX
RTX 4000 SFF Ada
20GB
RTX
RTX 4090 Mobile
16GB
RTX
RTX 4060 Ti
8GB
RTX
RTX 4090 Laptop
16GB
Jetson
Jetson Orin NX 16GB
16GB
GPU
RTX 3500 Ada Mobile
12GB
RTX
RTX 4080 Mobile
12GB
RTX
RTX 4070
12GB
RTX
RTX 4070 Ti
12GB
RTX
RTX 4080 Laptop
12GB
RTX
RTX 4070 Mobile
8GB
RTX
RTX 4060
8GB
RTX
RTX 4070 Laptop
8GB
RTX
RTX 4060 Laptop
8GB
Jetson
Jetson Orin NX 8GB
8GB
Jetson
Jetson Orin Nano 8GB
8GB
RTX
RTX 4050 Laptop
6GB
Jetson
Jetson Orin Nano 4GB
4GB
GPU
A800
40GB
Jetson
Jetson AGX Orin 64GB
64GB
GPU
L40
48GB
GPU
RTX 6000 Ada
48GB
Jetson
Jetson AGX Orin 32GB
32GB
RTX
RTX 4090
24GB
RTX
RTX 3090 Ti
24GB
RTX
RTX 4080
16GB
RTX
RTX 3070 Ti Mobile
8GB
RTX
RTX 3050
8GB
RTX
RTX 3050 Mobile
4GB
GPU
RTX A2000 Embedded
4GB
GPU
GTX 1630
4GB
GPU
RTX A5000
8GB
GPU
A30
24GB
GPU
A10
24GB
GPU
RTX A5000 Max-Q
16GB
GPU
RTX A5000 Mobile
16GB
GPU
RTX A4000
16GB
GPU
A2
16GB
RTX
RTX 3080 Mobile
16GB
GPU
RTX A3000 Mobile
6GB
GPU
RTX A2000
6GB
RTX
RTX 3080 Ti
12GB
RTX
RTX 3060
12GB
GPU
RTX 2060 12GB
12GB
GPU
RTX A4000 Max-Q
8GB
GPU
RTX A4000 Mobile
8GB
GPU
RTX A2000 Max-Q
4GB
GPU
RTX A2000 Mobile
4GB
RTX
RTX 3070 Ti
8GB
RTX
RTX 3060 Mobile
6GB
GPU
RTX 2050 Mobile
4GB
GPU
A100
80GB
GPU
RTX A6000
48GB
GPU
A40
48GB
RTX
RTX 3090
24GB
GPU
T10
16GB
RTX
RTX 3080
12GB
RTX
RTX 3070
8GB
RTX
RTX 3060 Ti
8GB
GPU
RTX 2070 SUPER Mobile
8GB
GPU
GTX 1650 Ti Mobile
4GB
GPU
T40
24GB
GPU
RTX 2080 SUPER
8GB
GPU
RTX 2070 SUPER
8GB
GPU
RTX 2060 SUPER
8GB
GPU
RTX 2060
6GB
GPU
RTX 2060 Mobile
6GB
GPU
GTX 1660 Ti
6GB
GPU
GTX 1660 Ti Mobile
6GB
GPU
GTX 1660 SUPER
6GB
GPU
GTX 1660
6GB
GPU
GTX 1650 SUPER
4GB
GPU
GTX 1650
4GB
GPU
GTX 1650 Mobile
4GB
GPU
Quadro RTX 8000
48GB
GPU
RTX Titan
24GB
GPU
Quadro RTX 6000
24GB
GPU
RTX 2080 Ti
11GB
NVIDIA T4
T4
16GB
GPU
Quadro RTX 5000
16GB
GPU
RTX 2080
8GB
GPU
RTX 2070
8GB
Metal
Apple Silicon Ultra
Apple M5 Ultra
96GB
Apple Silicon Max
Apple M5 Max
36GB
Apple Silicon Pro
Apple M5 Pro
24GB
Apple Silicon
Apple M6
16GB
Apple Silicon
Apple MacBook Neo
8GB
Apple Silicon Ultra
Apple M3 Ultra
96GB
Apple Silicon
Apple M5
16GB
Apple Silicon Max
Apple M4 Max
36GB
Apple Silicon Pro
Apple M4 Pro
24GB
Apple Silicon
Apple M4
16GB
Apple Silicon Ultra
Apple M2 Ultra
64GB
Apple Silicon Max
Apple M3 Max
36GB
Apple Silicon Max
Apple M2 Max
32GB
Apple Silicon Pro
Apple M3 Pro
18GB
Apple Silicon Pro
Apple M2 Pro
16GB
Apple Silicon
Apple M3
8GB
Apple Silicon Ultra
Apple M1 Ultra
16GB
Apple Silicon
Apple M2
8GB
Apple Silicon Max
Apple M1 Max
16GB
Apple Silicon Pro
Apple M1 Pro
16GB
Apple Silicon
Apple M1
8GB
Torch
2.14
OS
macoslinux
Arch
x86_64aarch64
Kernel Builder
d33d13d