JinjaThink limiter limiting parameters Atomic for llama cpp

#1
by VaLtEc-BoY - opened

I would like to request that you add a thought limiter to the Atomic Llama CPP, "--jinjathink: 'none - minimal-low-medium-high-maximum', '512'" to limit Jinja's thoughts.

--jinjathink: "minimal": 512 ----tokens

VaLtEc-BoY changed discussion title from JinjaThink limiter limiting parameters for llamma ccp to JinjaThink limiter limiting parameters Atomic for llama cpp
Atomic Chat org

@VaLtEc-BoY yo, thanks πŸ™ this is genuinely a good one. dropped it in the backlog.

we'll tag you right here once it's implemented so you can take it for a spin πŸ€™

Thank you, I even used the parameter --reasoning-budget 500" but it doesn't work I don't know if it's jinja from the model that forces but model to think non-stop I even put rules in the prompt but still the model thinks a lot...

Bug Report & Fix: Support for Gemma 4 Assistant Models (E4B and 12B MTP)
This patch provides a unified fix for loading errors encountered when running the Gemma 4 Assistant MTP models (both the E4B and the 12B versions).

What was causing the issues:

Metadata Key Mismatch (Mainly E4B): Some of the newer Gemma 4 GGUFs (like the E4B) use the embedding_length_out key instead of the expected n_embd_backbone in their metadata. When the engine failed to find n_embd_backbone, it crashed.
Tensor Naming & Mapping (E4B & 12B): Both the E4B and 12B MTP assistant models use the nextn.* prefix for internal projection tensors instead of the expected mtp.* prefix. Even though the engine could parse the string "nextn.pre_projection", it failed to map these new NEXTN tensors in the LLM_TENSOR_INFOS array. This caused llm_tensor_info_for to throw a std::out_of_range exception during tensor assignment (e.g., missing tensor info mapping for nextn.pre_projection.weight), aborting the load.
The Solution:

Added fallback support to read embedding_length_out if n_embd_backbone == 0.
Added fallback logic for tensor creation in llama-model.cpp (e.g., attempting to create LLM_TENSOR_NEXTN_PRE_PROJECTION if LLM_TENSOR_MTP_PRE_PROJECTION is not required/found).
Crucially, we mapped the NEXTN tensor family inside LLM_TENSOR_INFOS (in llama-arch.cpp), explicitly defining the mathematical operations (GGML_OP_MUL_MAT and GGML_OP_GET_ROWS) that the engine must apply to them.
Applying the diffs below fixes the support for the entire Gemma 4 Assistant family in a single pass:

diff
diff --git a/src/llama-arch.h b/src/llama-arch.h
index cefe024fb..47594458b 100644
--- a/src/llama-arch.h
+++ b/src/llama-arch.h
@@ -350,6 +350,7 @@ enum llm_kv {
LLM_KV_GEMMA4_ASSISTANT_N_CENTROIDS,
LLM_KV_GEMMA4_ASSISTANT_CENTROID_TOP_K,
LLM_KV_GEMMA4_ASSISTANT_N_EMBD_BACKBONE,

  • LLM_KV_GEMMA4_ASSISTANT_EMBEDDING_LENGTH_OUT,
    LLM_KV_GEMMA4_ASSISTANT_ATTENTION_K_EQ_V,
    LLM_KV_GEMMA4_ASSISTANT_USE_ORDERED_EMBEDDINGS,
    LLM_KV_GEMMA4_ASSISTANT_REQUIRES_TARGET_ARCH,
    @@ -570,6 +571,10 @@ enum llm_tensor {
    LLM_TENSOR_MTP_POST_PROJECTION,
    LLM_TENSOR_MTP_CENTROIDS,
    LLM_TENSOR_MTP_TOKEN_ORDERING,
  • LLM_TENSOR_NEXTN_PRE_PROJECTION,
  • LLM_TENSOR_NEXTN_POST_PROJECTION,
  • LLM_TENSOR_NEXTN_CENTROIDS,
  • LLM_TENSOR_NEXTN_TOKEN_ORDERING,

};
diff
diff --git a/src/llama-arch.cpp b/src/llama-arch.cpp
index 3a7c7619c..dce4ed1a9 100644
--- a/src/llama-arch.cpp
+++ b/src/llama-arch.cpp
@@ -292,10 +292,11 @@ static const std::map<llm_kv, const char *> LLM_KV_NAMES =
{ LLM_KV_DENSE_3_FEAT_IN, "%s.dense_3_feat_in" },
{ LLM_KV_DENSE_3_FEAT_OUT, "%s.dense_3_feat_out" },

  • { LLM_KV_GEMMA4_ASSISTANT_N_CENTROIDS, "%s.n_centroids" },
  • { LLM_KV_GEMMA4_ASSISTANT_CENTROID_TOP_K, "%s.centroid_top_k" },
  • { LLM_KV_GEMMA4_ASSISTANT_N_EMBD_BACKBONE, "%s.n_embd_backbone" },
  • { LLM_KV_GEMMA4_ASSISTANT_ATTENTION_K_EQ_V, "%s.attention.k_eq_v" },
  • { LLM_KV_GEMMA4_ASSISTANT_N_CENTROIDS, "%s.n_centroids" },
  • { LLM_KV_GEMMA4_ASSISTANT_CENTROID_TOP_K, "%s.centroid_top_k" },
  • { LLM_KV_GEMMA4_ASSISTANT_N_EMBD_BACKBONE, "%s.n_embd_backbone" },
  • { LLM_KV_GEMMA4_ASSISTANT_EMBEDDING_LENGTH_OUT, "%s.embedding_length_out" },
  • { LLM_KV_GEMMA4_ASSISTANT_ATTENTION_K_EQ_V, "%s.attention.k_eq_v" },
    { LLM_KV_GEMMA4_ASSISTANT_USE_ORDERED_EMBEDDINGS, "%s.use_ordered_embeddings" },

@@ -560,6 +561,10 @@ static const std::map<llm_tensor, const char *> LLM_TENSOR_NAMES = {
{ LLM_TENSOR_MTP_POST_PROJECTION, "mtp.post_projection" },
{ LLM_TENSOR_MTP_CENTROIDS, "mtp.centroids" },
{ LLM_TENSOR_MTP_TOKEN_ORDERING, "mtp.token_ordering" },

  • { LLM_TENSOR_NEXTN_PRE_PROJECTION, "nextn.pre_projection" },
  • { LLM_TENSOR_NEXTN_POST_PROJECTION, "nextn.post_projection" },
  • { LLM_TENSOR_NEXTN_CENTROIDS, "nextn.centroids" },
  • { LLM_TENSOR_NEXTN_TOKEN_ORDERING, "nextn.token_ordering" },

};

@@ -787,6 +792,10 @@ static const std::map<llm_tensor, llm_tensor_info> LLM_TENSOR_INFOS = {
{LLM_TENSOR_MTP_POST_PROJECTION, {LLM_TENSOR_LAYER_OUTPUT, GGML_OP_MUL_MAT}},
{LLM_TENSOR_MTP_CENTROIDS, {LLM_TENSOR_LAYER_OUTPUT, GGML_OP_MUL_MAT}},
{LLM_TENSOR_MTP_TOKEN_ORDERING, {LLM_TENSOR_LAYER_OUTPUT, GGML_OP_GET_ROWS}},

  • {LLM_TENSOR_NEXTN_PRE_PROJECTION, {LLM_TENSOR_LAYER_INPUT, GGML_OP_MUL_MAT}},
  • {LLM_TENSOR_NEXTN_POST_PROJECTION, {LLM_TENSOR_LAYER_OUTPUT, GGML_OP_MUL_MAT}},
  • {LLM_TENSOR_NEXTN_CENTROIDS, {LLM_TENSOR_LAYER_OUTPUT, GGML_OP_MUL_MAT}},
  • {LLM_TENSOR_NEXTN_TOKEN_ORDERING, {LLM_TENSOR_LAYER_OUTPUT, GGML_OP_GET_ROWS}},
    // Nemotron 3 Super
    diff
    diff --git a/src/llama-model.cpp b/src/llama-model.cpp
    index fd565d735..c3477ca8b 100644
    --- a/src/llama-model.cpp
    +++ b/src/llama-model.cpp
    @@ -1659,6 +1659,9 @@ void llama_model::load_hparams(llama_model_loader & ml) {
    ml.get_key(LLM_KV_GEMMA4_ASSISTANT_N_EMBD_BACKBONE, hparams.n_embd_backbone, false);
  •            if (hparams.n_embd_backbone == 0) {
    
  •                ml.get_key(LLM_KV_GEMMA4_ASSISTANT_EMBEDDING_LENGTH_OUT, hparams.n_embd_backbone, false);
    
  •            }
               ml.get_key(LLM_KV_GEMMA4_ASSISTANT_N_CENTROIDS,           hparams.n_centroids, false);
               ml.get_key(LLM_KV_GEMMA4_ASSISTANT_CENTROID_TOP_K,         hparams.centroid_top_k, false);
    

@@ -4789,8 +4792,14 @@ bool llama_model::load_tensors(llama_model_loader & ml) {
tok_embd = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, 0);

  •                mtp_pre_projection  = create_tensor(tn(LLM_TENSOR_MTP_PRE_PROJECTION,  "weight"), {2 * (int64_t) n_bb, n_embd}, 0);
    
  •                mtp_post_projection = create_tensor(tn(LLM_TENSOR_MTP_POST_PROJECTION, "weight"), {n_embd, (int64_t) n_bb}, 0);
    
  •                mtp_pre_projection  = create_tensor(tn(LLM_TENSOR_MTP_PRE_PROJECTION,  "weight"), {2 * (int64_t) n_bb, n_embd}, TENSOR_NOT_REQUIRED);
    
  •                if (!mtp_pre_projection) {
    
  •                    mtp_pre_projection = create_tensor(tn(LLM_TENSOR_NEXTN_PRE_PROJECTION, "weight"), {2 * (int64_t) n_bb, n_embd}, 0);
    
  •                }
    
  •                mtp_post_projection = create_tensor(tn(LLM_TENSOR_MTP_POST_PROJECTION, "weight"), {n_embd, (int64_t) n_bb}, TENSOR_NOT_REQUIRED);
    
  •                if (!mtp_post_projection) {
    
  •                    mtp_post_projection = create_tensor(tn(LLM_TENSOR_NEXTN_POST_PROJECTION, "weight"), {n_embd, (int64_t) n_bb}, 0);
    
  •                }
    
                   if (hparams.use_ordered_embeddings) {
                       const uint32_t n_c = hparams.n_centroids;
    
  •                    mtp_centroids      = create_tensor(tn(LLM_TENSOR_MTP_CENTROIDS,      "weight"), {n_embd, (int64_t) n_c}, 0);
    
  •                    mtp_centroids = create_tensor(tn(LLM_TENSOR_MTP_CENTROIDS, "weight"), {n_embd, (int64_t) n_c}, TENSOR_NOT_REQUIRED);
    
  •                    if (!mtp_centroids) {
    
  •                        mtp_centroids = create_tensor(tn(LLM_TENSOR_NEXTN_CENTROIDS, "weight"), {n_embd, (int64_t) n_c}, 0);
    
  •                    }
                       mtp_token_ordering = create_tensor(tn(LLM_TENSOR_MTP_TOKEN_ORDERING, "weight"), {(int64_t) n_vocab}, TENSOR_NOT_REQUIRED);
    
  •                    if (!mtp_token_ordering) {
    
  •                        mtp_token_ordering = create_tensor(tn(LLM_TENSOR_NEXTN_TOKEN_ORDERING, "weight"), {(int64_t) n_vocab}, 0);
    
  •                    }
                   }
    

#I'd like to leave a fix here. Thanks for the support, I believe in the power of ATOMIC!

Sign up or log in to comment