Instructions to use neph1/Mistral-Nemo-Instruct-bellman-12b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use neph1/Mistral-Nemo-Instruct-bellman-12b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="neph1/Mistral-Nemo-Instruct-bellman-12b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("neph1/Mistral-Nemo-Instruct-bellman-12b")
model = AutoModelForCausalLM.from_pretrained("neph1/Mistral-Nemo-Instruct-bellman-12b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use neph1/Mistral-Nemo-Instruct-bellman-12b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "neph1/Mistral-Nemo-Instruct-bellman-12b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "neph1/Mistral-Nemo-Instruct-bellman-12b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/neph1/Mistral-Nemo-Instruct-bellman-12b

SGLang

How to use neph1/Mistral-Nemo-Instruct-bellman-12b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "neph1/Mistral-Nemo-Instruct-bellman-12b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "neph1/Mistral-Nemo-Instruct-bellman-12b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "neph1/Mistral-Nemo-Instruct-bellman-12b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "neph1/Mistral-Nemo-Instruct-bellman-12b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Unsloth Studio new

How to use neph1/Mistral-Nemo-Instruct-bellman-12b with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for neph1/Mistral-Nemo-Instruct-bellman-12b to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for neph1/Mistral-Nemo-Instruct-bellman-12b to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for neph1/Mistral-Nemo-Instruct-bellman-12b to start chatting

Load model with FastModel

pip install unsloth
from unsloth import FastModel
model, tokenizer = FastModel.from_pretrained(
    model_name="neph1/Mistral-Nemo-Instruct-bellman-12b",
    max_seq_length=2048,
)

Docker Model Runner
How to use neph1/Mistral-Nemo-Instruct-bellman-12b with Docker Model Runner:
```
docker model run hf.co/neph1/Mistral-Nemo-Instruct-bellman-12b
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Model Card for Bellman

This version of bellman is finetuned from Mistral-Nemo-Instruct-2407. It's a rank 128 qlora trained for about 1 epoch. It's finetuned for prompt question answering, based on a dataset created from Swedish wikipedia, with a lot of Sweden-centric questions. New from previous versions is questions from a translated code-feedback dataset, as well as a number of stories.

Consider this a work in progress as I adjust the training for this new model size. Will provide a few updates to the model.

Quality seems to be dependent on "initial seed". Sometimes it's just amazing, and sometimes quite bad. If you're having trouble getting good results, try lowering temperature.

For GGUFs, please look to:
https://huggingface.co/mradermacher/Mistral-Nemo-Instruct-bellman-12b-GGUF and
https://huggingface.co/mradermacher/Mistral-Nemo-Instruct-bellman-12b-i1-GGUF

Model Details

Rank: 128

Trained with Unsloth on a 3090

Differences from base model:

The base model is pretty good at Swedish already, but my 'vibe check' says this finetune is slightly improved. There are less weird wordings.

Bellman is trained on fairly short answers and tends to be less verbose.

Output example (Mistral-Nemo-Instruct-bellman-12b.i1-Q4_K_M.gguf):

User: "Hej!"

Bellman: "Hej! Vad kan jag hjälpa dig med?"

User: "Berätta lite om Sverige"

Bellman: "Sverige är ett land i norra Europa med en rik historia. Det består av 21 län och har en yta på drygt 450,000 kvadratkilometer. Stockholm är Sveriges huvudstad."

User: "När grundades det?"

Bellman: "Det är inte exakt känt när Sverige grundades. Det var först under 1300-talet att Sverige började ge sig uttryck i internationella affärer."

User: "Vad hände 1523?"

Bellman: "År 1523 blev Gustav Vasa kung över Sverige och inledde reformationen i landet. Han var också den förste monarken att bära titeln "Sveriges Konung"."

User: "Hur länge levde han?"

Bellman: "Gustav Vasa levde från år 1496 till 1560."

Training Parameters

per_device_train_batch_size = 2,
gradient_accumulation_steps = 64,
num_train_epochs=3,
warmup_steps = 5,
learning_rate = 1e-4,
logging_steps = 15,
optim = "adamw_8bit",
weight_decay = 0.01,
lr_scheduler_type = "linear",
seed = 3407,
per_device_eval_batch_size = 2,
evaluation_strategy="steps",
eval_accumulation_steps = 64,
eval_steps = 15,
eval_delay = 0,
save_strategy="steps",
save_steps=50,