Instructions to use DevQuasar/prithivMLmods.Qwen2-VL-OCR2-2B-Instruct-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use DevQuasar/prithivMLmods.Qwen2-VL-OCR2-2B-Instruct-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf DevQuasar/prithivMLmods.Qwen2-VL-OCR2-2B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf DevQuasar/prithivMLmods.Qwen2-VL-OCR2-2B-Instruct-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf DevQuasar/prithivMLmods.Qwen2-VL-OCR2-2B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf DevQuasar/prithivMLmods.Qwen2-VL-OCR2-2B-Instruct-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf DevQuasar/prithivMLmods.Qwen2-VL-OCR2-2B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama cli -hf DevQuasar/prithivMLmods.Qwen2-VL-OCR2-2B-Instruct-GGUF:Q4_K_M

Use Docker

docker model run hf.co/DevQuasar/prithivMLmods.Qwen2-VL-OCR2-2B-Instruct-GGUF:Q4_K_M

Llama new
LM Studio
Jan

vLLM

How to use DevQuasar/prithivMLmods.Qwen2-VL-OCR2-2B-Instruct-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "DevQuasar/prithivMLmods.Qwen2-VL-OCR2-2B-Instruct-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DevQuasar/prithivMLmods.Qwen2-VL-OCR2-2B-Instruct-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/DevQuasar/prithivMLmods.Qwen2-VL-OCR2-2B-Instruct-GGUF:Q4_K_M

Ollama
How to use DevQuasar/prithivMLmods.Qwen2-VL-OCR2-2B-Instruct-GGUF with Ollama:
```
ollama run hf.co/DevQuasar/prithivMLmods.Qwen2-VL-OCR2-2B-Instruct-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use DevQuasar/prithivMLmods.Qwen2-VL-OCR2-2B-Instruct-GGUF with Docker Model Runner:
```
docker model run hf.co/DevQuasar/prithivMLmods.Qwen2-VL-OCR2-2B-Instruct-GGUF:Q4_K_M
```

Lemonade

How to use DevQuasar/prithivMLmods.Qwen2-VL-OCR2-2B-Instruct-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull DevQuasar/prithivMLmods.Qwen2-VL-OCR2-2B-Instruct-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.prithivMLmods.Qwen2-VL-OCR2-2B-Instruct-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

How to load the model?

pinned

by Blithe8002 - opened Sep 11, 2025

Discussion

Blithe8002

Sep 11, 2025

•

edited Sep 11, 2025

I want to use the 4bit quantized model to run an OCR related task on an image. Can you please tell me how to do that? I tried llama.cpp but I was not able to make it work.

csabakecskemeti pinned discussion Sep 15, 2025

csabakecskemeti

DevQuasar org Sep 17, 2025

•

edited Sep 17, 2025

Thanks for raising the issue
The vision adapter has missed the 'mmproj' prefix so that could cause issue, that like the llama.cpp server were not automatically recognizing the vision adapter as they rely on filename patterns.
Filname fixed
Easy way to use is with LMStudio, but it should work similarly with llama.cpp

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment