Instructions to use beyoru/Kiwen1.1-27B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use beyoru/Kiwen1.1-27B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="beyoru/Kiwen1.1-27B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("beyoru/Kiwen1.1-27B")
model = AutoModelForMultimodalLM.from_pretrained("beyoru/Kiwen1.1-27B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use beyoru/Kiwen1.1-27B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "beyoru/Kiwen1.1-27B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "beyoru/Kiwen1.1-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/beyoru/Kiwen1.1-27B

SGLang

How to use beyoru/Kiwen1.1-27B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "beyoru/Kiwen1.1-27B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "beyoru/Kiwen1.1-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "beyoru/Kiwen1.1-27B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "beyoru/Kiwen1.1-27B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use beyoru/Kiwen1.1-27B with Docker Model Runner:
```
docker model run hf.co/beyoru/Kiwen1.1-27B
```

Longer agentic work

by Throghar - opened Sep 14

Discussion

Throghar

Sep 14

•

edited Sep 14

Today i tested it against harder coding problem, it was longer context agentic work and to my surprise model halucinated several times about doing work which he didnt. One time model confidently just stated the work is done but it was only in reasoning part, nothing was written. I am using largest Q6K quant from https://huggingface.co/NikiKrutan/Kiwen1.1-27B-MTP-GGUF

beyoru

Owner Sep 14

•

edited Sep 14

Thanks for testing. Kiwen1.1 wasn't trained on long-horizon agentic trajectories (poor GPU cluster XD ), so extended tool loops are out of distribution losing track of what it executed is expected there.

But "claims done, nothing written" can also be the template emitting the tool call inside the reasoning block, so the parser never fires it. Could you share the raw stream around the "done" claim plus your llama-server flags? Q6_K isn't the issue.

Throghar

Sep 14

•

edited Sep 14

llama-server.exe --port 1234 --threads 12 --n-gpu-layers 999 --model "Kiwen1.1-27B-23000-NIKI-Q6_K.gguf" --ctx-size 140000 --parallel 1 --temp 1 --min-p 0.0 --top-k 20 --top-p 0.95 --batch-size 8192 --flash-attn on --split-mode layer --offline --main-gpu 0 -ts 0.5,0.5 -ctk q8_0 -ctv q8_0 -ub 256 --jinja --ctx-checkpoints 10 --cache-ram 8192 --chat-template-kwargs "{"preserve_thinking":true,"reasoning_effort":"xhigh"}" --spec-type draft-mtp --spec-draft-n-max 2 -lv 4 -ot "nomatch=CPU" -lm none --spec-draft-device vulkan1 -ctkd q8_0 -ctvd q8_0 --spec-draft-p-min 0.3

Harness: Ante terminal cli

Been testing same prompt with other models, so far no hallucination like with kiwen1.1

I dont think i saw tool call inside reasoning block there, cant rule it out though, whole session was about 100k context tried it several times and everytime problem with hallucination occured in some form.

NikiKrutan

Sep 14

From my experience after ~80% of full 262K context there may be some noticeable degradation including looping. I sometimes switch to Qwen if I can't compact before 80% and task is really hard (but Qwen wastes all the precious 20% very fast though so better compact early).
Also template sometimes is the issue (in my quants I used v20 of this template ). It is better for my workflow than froggeric's but it is not always so. Most templates give too many additional instructions and logic (including Moore's one). Try different templates.

Throghar

Sep 14

I also experienced several occasions of looping

Throghar

Sep 14

i may try later with this template: https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

Throghar

Sep 14

Tested, experiencing looping, didnt get to halucination because of that

beyoru

Owner Sep 14

Before chasing this further both templates are community-patched. Could you run the original one ?

Throghar

Sep 14

•

edited Sep 14

Well currently with https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates i got one loop, second run is fine for now, im gonna use it more before considering whether hallucination is or is not related to template.

I'd love to see more poeple test it in some harder agentic work with longer context so it can show whether this is related to any specifics in my environment.

NikiKrutan

Sep 21

I've tried original beyoru's template and looping went away. I haven't seen any other artifacts so far. But I haven't tested much. So I suspect that froggeric's and moore's chat templates do not work well with this model.
Same thing with Kiwen1.2. It awfully loops with froggeric's but ok with original template.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment