Instructions to use moonshotai/Kimi-K3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use moonshotai/Kimi-K3 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="moonshotai/Kimi-K3", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("moonshotai/Kimi-K3", trust_remote_code=True, device_map="auto")

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use moonshotai/Kimi-K3 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "moonshotai/Kimi-K3"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "moonshotai/Kimi-K3",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/moonshotai/Kimi-K3

SGLang

How to use moonshotai/Kimi-K3 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "moonshotai/Kimi-K3" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "moonshotai/Kimi-K3",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "moonshotai/Kimi-K3" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "moonshotai/Kimi-K3",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use moonshotai/Kimi-K3 with Docker Model Runner:
```
docker model run hf.co/moonshotai/Kimi-K3
```

This might be the cheapest GPU type for Kimi K3 with 100% Data-on-GPU — Gift for AMD MI300x8

#168

by ghostplant - opened about 1 month ago

Discussion

ghostplant

about 1 month ago

•

edited about 1 month ago

Even B200 x 8 has trouble to launch Kimi K3, however, older MI300 is still quite possible:

hf download --local-dir moonshotai/Kimi-K3 moonshotai/Kimi-K3

[Azure ND_MI300_192G_v5 (Single Node)]
docker run -e WORKER=1 -e LOCAL_SIZE=8 -p 8000:8000 -it --rm --ipc=host --shm-size=8g \
      --ulimit memlock=-1 --ulimit stack=67108864 -v /:/host -w /host$(pwd) \
      --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --device=/dev/kfd --device=/dev/dri --group-add=video \
      tutelgroup/deepseek-671b:mi300x8-chat-20260808 --serve=core \
        --try_path moonshotai/Kimi-K3 --max_seq_len 120000

x-polyglot-x

30 days ago

Fantastic news - I found an Ebay listing (8 x 192gb cards) for $500,000. Incredible value!

ghostplant

29 days ago

•

edited 23 days ago

Did you pay more for MI300 than the B200? You just need to pay a quarter..

cheeseman182

24 days ago

$500,000 dollars... only if i could buy it with my $4.72 in my account...

ghostplant

24 days ago

•

edited 23 days ago

@cheeseman182 I believe x-polyglot-x was joking. The price is here:

Machine Price:

MI300: $150,000
B300: $550,000

Power Expense:

MI300: 750W
B300: 1400W

Rent Price (monthly):

MI300: $6,200
B300: $25,000

athrael-soju

6 days ago

You can offload weights onto RAM and get it to work. Or...You can wait for a very, very special surprize that will allow running it ON B200 & MI300X, with enough to spare for 1m context size KV cache.

ghostplant

5 days ago

You can offload weights onto RAM and get it to work. Or...You can wait for a very, very special surprize that will allow running it ON B200 & MI300X, with enough to spare for 1m context size KV cache.

700K context has been possible for MI300, but B200 is less lucky because it is just 180GB instead of 192GB per device.

athrael-soju

5 days ago

•

edited 5 days ago

You can offload weights onto RAM and get it to work. Or...You can wait for a very, very special surprize that will allow running it ON B200 & MI300X, with enough to spare for 1m context size KV cache.

700K context has been possible for MI300, but B200 is less lucky because it is just 180GB instead of 192GB per device.

Trust me, you really don't want anything over 512k context, because post-compaction forgetfulness will wreck your agent's performance. It's generally more productive to have more sessions at lower context, or use swarm/sub-agents. Kimi code is really good with swarm mode.

Just my 2 cents.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment