Instructions to use AtomicChat/gemma-4-12B-it-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use AtomicChat/gemma-4-12B-it-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf AtomicChat/gemma-4-12B-it-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf AtomicChat/gemma-4-12B-it-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf AtomicChat/gemma-4-12B-it-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf AtomicChat/gemma-4-12B-it-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf AtomicChat/gemma-4-12B-it-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf AtomicChat/gemma-4-12B-it-GGUF:Q4_K_M
Use Docker
docker model run hf.co/AtomicChat/gemma-4-12B-it-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use AtomicChat/gemma-4-12B-it-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "AtomicChat/gemma-4-12B-it-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AtomicChat/gemma-4-12B-it-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/AtomicChat/gemma-4-12B-it-GGUF:Q4_K_M
- Ollama
How to use AtomicChat/gemma-4-12B-it-GGUF with Ollama:
ollama run hf.co/AtomicChat/gemma-4-12B-it-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use AtomicChat/gemma-4-12B-it-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AtomicChat/gemma-4-12B-it-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "AtomicChat/gemma-4-12B-it-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use AtomicChat/gemma-4-12B-it-GGUF with Docker Model Runner:
docker model run hf.co/AtomicChat/gemma-4-12B-it-GGUF:Q4_K_M
- Lemonade
How to use AtomicChat/gemma-4-12B-it-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull AtomicChat/gemma-4-12B-it-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.gemma-4-12B-it-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use AtomicChat/gemma-4-12B-it-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AtomicChat/gemma-4-12B-it-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default AtomicChat/gemma-4-12B-it-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use AtomicChat/gemma-4-12B-it-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AtomicChat/gemma-4-12B-it-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "AtomicChat/gemma-4-12B-it-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
JinjaThink limiter limiting parameters Atomic for llama cpp
I would like to request that you add a thought limiter to the Atomic Llama CPP, "--jinjathink: 'none - minimal-low-medium-high-maximum', '512'" to limit Jinja's thoughts.
--jinjathink: "minimal": 512 ----tokens
@VaLtEc-BoY yo, thanks π this is genuinely a good one. dropped it in the backlog.
we'll tag you right here once it's implemented so you can take it for a spin π€
Thank you, I even used the parameter --reasoning-budget 500" but it doesn't work I don't know if it's jinja from the model that forces but model to think non-stop I even put rules in the prompt but still the model thinks a lot...
Bug Report & Fix: Support for Gemma 4 Assistant Models (E4B and 12B MTP)
This patch provides a unified fix for loading errors encountered when running the Gemma 4 Assistant MTP models (both the E4B and the 12B versions).
What was causing the issues:
Metadata Key Mismatch (Mainly E4B): Some of the newer Gemma 4 GGUFs (like the E4B) use the embedding_length_out key instead of the expected n_embd_backbone in their metadata. When the engine failed to find n_embd_backbone, it crashed.
Tensor Naming & Mapping (E4B & 12B): Both the E4B and 12B MTP assistant models use the nextn.* prefix for internal projection tensors instead of the expected mtp.* prefix. Even though the engine could parse the string "nextn.pre_projection", it failed to map these new NEXTN tensors in the LLM_TENSOR_INFOS array. This caused llm_tensor_info_for to throw a std::out_of_range exception during tensor assignment (e.g., missing tensor info mapping for nextn.pre_projection.weight), aborting the load.
The Solution:
Added fallback support to read embedding_length_out if n_embd_backbone == 0.
Added fallback logic for tensor creation in llama-model.cpp (e.g., attempting to create LLM_TENSOR_NEXTN_PRE_PROJECTION if LLM_TENSOR_MTP_PRE_PROJECTION is not required/found).
Crucially, we mapped the NEXTN tensor family inside LLM_TENSOR_INFOS (in llama-arch.cpp), explicitly defining the mathematical operations (GGML_OP_MUL_MAT and GGML_OP_GET_ROWS) that the engine must apply to them.
Applying the diffs below fixes the support for the entire Gemma 4 Assistant family in a single pass:
diff
diff --git a/src/llama-arch.h b/src/llama-arch.h
index cefe024fb..47594458b 100644
--- a/src/llama-arch.h
+++ b/src/llama-arch.h
@@ -350,6 +350,7 @@ enum llm_kv {
LLM_KV_GEMMA4_ASSISTANT_N_CENTROIDS,
LLM_KV_GEMMA4_ASSISTANT_CENTROID_TOP_K,
LLM_KV_GEMMA4_ASSISTANT_N_EMBD_BACKBONE,
- LLM_KV_GEMMA4_ASSISTANT_EMBEDDING_LENGTH_OUT,
LLM_KV_GEMMA4_ASSISTANT_ATTENTION_K_EQ_V,
LLM_KV_GEMMA4_ASSISTANT_USE_ORDERED_EMBEDDINGS,
LLM_KV_GEMMA4_ASSISTANT_REQUIRES_TARGET_ARCH,
@@ -570,6 +571,10 @@ enum llm_tensor {
LLM_TENSOR_MTP_POST_PROJECTION,
LLM_TENSOR_MTP_CENTROIDS,
LLM_TENSOR_MTP_TOKEN_ORDERING, - LLM_TENSOR_NEXTN_PRE_PROJECTION,
- LLM_TENSOR_NEXTN_POST_PROJECTION,
- LLM_TENSOR_NEXTN_CENTROIDS,
- LLM_TENSOR_NEXTN_TOKEN_ORDERING,
};
diff
diff --git a/src/llama-arch.cpp b/src/llama-arch.cpp
index 3a7c7619c..dce4ed1a9 100644
--- a/src/llama-arch.cpp
+++ b/src/llama-arch.cpp
@@ -292,10 +292,11 @@ static const std::map<llm_kv, const char *> LLM_KV_NAMES =
{ LLM_KV_DENSE_3_FEAT_IN, "%s.dense_3_feat_in" },
{ LLM_KV_DENSE_3_FEAT_OUT, "%s.dense_3_feat_out" },
- { LLM_KV_GEMMA4_ASSISTANT_N_CENTROIDS, "%s.n_centroids" },
- { LLM_KV_GEMMA4_ASSISTANT_CENTROID_TOP_K, "%s.centroid_top_k" },
- { LLM_KV_GEMMA4_ASSISTANT_N_EMBD_BACKBONE, "%s.n_embd_backbone" },
- { LLM_KV_GEMMA4_ASSISTANT_ATTENTION_K_EQ_V, "%s.attention.k_eq_v" },
- { LLM_KV_GEMMA4_ASSISTANT_N_CENTROIDS, "%s.n_centroids" },
- { LLM_KV_GEMMA4_ASSISTANT_CENTROID_TOP_K, "%s.centroid_top_k" },
- { LLM_KV_GEMMA4_ASSISTANT_N_EMBD_BACKBONE, "%s.n_embd_backbone" },
- { LLM_KV_GEMMA4_ASSISTANT_EMBEDDING_LENGTH_OUT, "%s.embedding_length_out" },
- { LLM_KV_GEMMA4_ASSISTANT_ATTENTION_K_EQ_V, "%s.attention.k_eq_v" },
{ LLM_KV_GEMMA4_ASSISTANT_USE_ORDERED_EMBEDDINGS, "%s.use_ordered_embeddings" },
@@ -560,6 +561,10 @@ static const std::map<llm_tensor, const char *> LLM_TENSOR_NAMES = {
{ LLM_TENSOR_MTP_POST_PROJECTION, "mtp.post_projection" },
{ LLM_TENSOR_MTP_CENTROIDS, "mtp.centroids" },
{ LLM_TENSOR_MTP_TOKEN_ORDERING, "mtp.token_ordering" },
- { LLM_TENSOR_NEXTN_PRE_PROJECTION, "nextn.pre_projection" },
- { LLM_TENSOR_NEXTN_POST_PROJECTION, "nextn.post_projection" },
- { LLM_TENSOR_NEXTN_CENTROIDS, "nextn.centroids" },
- { LLM_TENSOR_NEXTN_TOKEN_ORDERING, "nextn.token_ordering" },
};
@@ -787,6 +792,10 @@ static const std::map<llm_tensor, llm_tensor_info> LLM_TENSOR_INFOS = {
{LLM_TENSOR_MTP_POST_PROJECTION, {LLM_TENSOR_LAYER_OUTPUT, GGML_OP_MUL_MAT}},
{LLM_TENSOR_MTP_CENTROIDS, {LLM_TENSOR_LAYER_OUTPUT, GGML_OP_MUL_MAT}},
{LLM_TENSOR_MTP_TOKEN_ORDERING, {LLM_TENSOR_LAYER_OUTPUT, GGML_OP_GET_ROWS}},
- {LLM_TENSOR_NEXTN_PRE_PROJECTION, {LLM_TENSOR_LAYER_INPUT, GGML_OP_MUL_MAT}},
- {LLM_TENSOR_NEXTN_POST_PROJECTION, {LLM_TENSOR_LAYER_OUTPUT, GGML_OP_MUL_MAT}},
- {LLM_TENSOR_NEXTN_CENTROIDS, {LLM_TENSOR_LAYER_OUTPUT, GGML_OP_MUL_MAT}},
- {LLM_TENSOR_NEXTN_TOKEN_ORDERING, {LLM_TENSOR_LAYER_OUTPUT, GGML_OP_GET_ROWS}},
// Nemotron 3 Super
diff
diff --git a/src/llama-model.cpp b/src/llama-model.cpp
index fd565d735..c3477ca8b 100644
--- a/src/llama-model.cpp
+++ b/src/llama-model.cpp
@@ -1659,6 +1659,9 @@ void llama_model::load_hparams(llama_model_loader & ml) {
ml.get_key(LLM_KV_GEMMA4_ASSISTANT_N_EMBD_BACKBONE, hparams.n_embd_backbone, false); if (hparams.n_embd_backbone == 0) {ml.get_key(LLM_KV_GEMMA4_ASSISTANT_EMBEDDING_LENGTH_OUT, hparams.n_embd_backbone, false);} ml.get_key(LLM_KV_GEMMA4_ASSISTANT_N_CENTROIDS, hparams.n_centroids, false); ml.get_key(LLM_KV_GEMMA4_ASSISTANT_CENTROID_TOP_K, hparams.centroid_top_k, false);
@@ -4789,8 +4792,14 @@ bool llama_model::load_tensors(llama_model_loader & ml) {
tok_embd = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, 0);
mtp_pre_projection = create_tensor(tn(LLM_TENSOR_MTP_PRE_PROJECTION, "weight"), {2 * (int64_t) n_bb, n_embd}, 0);mtp_post_projection = create_tensor(tn(LLM_TENSOR_MTP_POST_PROJECTION, "weight"), {n_embd, (int64_t) n_bb}, 0);
mtp_pre_projection = create_tensor(tn(LLM_TENSOR_MTP_PRE_PROJECTION, "weight"), {2 * (int64_t) n_bb, n_embd}, TENSOR_NOT_REQUIRED);if (!mtp_pre_projection) {mtp_pre_projection = create_tensor(tn(LLM_TENSOR_NEXTN_PRE_PROJECTION, "weight"), {2 * (int64_t) n_bb, n_embd}, 0);}mtp_post_projection = create_tensor(tn(LLM_TENSOR_MTP_POST_PROJECTION, "weight"), {n_embd, (int64_t) n_bb}, TENSOR_NOT_REQUIRED);if (!mtp_post_projection) {mtp_post_projection = create_tensor(tn(LLM_TENSOR_NEXTN_POST_PROJECTION, "weight"), {n_embd, (int64_t) n_bb}, 0);} if (hparams.use_ordered_embeddings) { const uint32_t n_c = hparams.n_centroids;
mtp_centroids = create_tensor(tn(LLM_TENSOR_MTP_CENTROIDS, "weight"), {n_embd, (int64_t) n_c}, 0);
mtp_centroids = create_tensor(tn(LLM_TENSOR_MTP_CENTROIDS, "weight"), {n_embd, (int64_t) n_c}, TENSOR_NOT_REQUIRED);if (!mtp_centroids) {mtp_centroids = create_tensor(tn(LLM_TENSOR_NEXTN_CENTROIDS, "weight"), {n_embd, (int64_t) n_c}, 0);} mtp_token_ordering = create_tensor(tn(LLM_TENSOR_MTP_TOKEN_ORDERING, "weight"), {(int64_t) n_vocab}, TENSOR_NOT_REQUIRED);if (!mtp_token_ordering) {mtp_token_ordering = create_tensor(tn(LLM_TENSOR_NEXTN_TOKEN_ORDERING, "weight"), {(int64_t) n_vocab}, 0);} }
#I'd like to leave a fix here. Thanks for the support, I believe in the power of ATOMIC!