functiongemma-270m-relay-tools

Fine-tuned FunctionGemma 270M translator for Relay: single-turn natural-language instruction → exactly one tool call over the harness's 33-tool registry (filesystem, editing, execution, git, web, environment, utility, interaction). It does not reason or chat — the Relay runtime owns validation, confidence gates, and approvals; this model only selects the tool by its exact registered name plus small args. Bulk text (file contents, code, commands) travels in runtime payload blocks, never through this model.

Because selection is by exact tool name, misfires on concrete instructions are negligible in practice — failures appear almost only on abstract or ambiguous edge-case prompts (e.g. "Show develop for docs/." or "Compare X with release/1.2"), never on everyday phrasings like "Read src/auth.py" or "Show the last 5 commits".

Training

• Base: unsloth/functiongemma-270m-it, full fine-tune (no LoRA), Unsloth, WSL + CUDA.
• Hyperparameters: 3 epochs, batch 4 / grad-accum 8 (eff. 32), lr 2e-5, --save-steps 50, max_seq_length 4096. Main run outputs/full4, continued as outputs/full5 from checkpoint-100; published weights are the checkpoint-200 export (~200 updates).
• Dataset: one atomic tool call per row (instruction + tool + arguments), 30–100 rows per tool across all 33 tools, explicit and implicit phrasings; prompts rendered with Relay's own FunctionGemmaActionModel renderer (zero train/serve skew), 10% held out per tool.
• Export: merged 16-bit → GGUF, both quants in this repo: Q4_K_M (serving quant, 261 MB) and BF16 (reference precision). No retraining was needed at any point: the one export failure was a converter vocab assertion, fixed at export time.

Evaluation

Held-out set, 20 prompts × 33 tools (660 runs) through the real harness adapter (eval_finetune.py), each quant served via llama-server (avg 0.10 s/call). Three tiers — functional (right tool + behaviorally equivalent args: payload slots exempt, values normalized, missing/extra optional args pass, ask_user passes on any non-empty question), selection (right tool), exact (byte-identical args):

	Functional	Selection	Exact
Base model	81/660 = 12.3%	136/660 = 20.6%	54/660 = 8.2%
Fine-tuned Q4_K_M	583/660 = 88.3%	603/660 = 91.4%	446/660 = 67.6%
Fine-tuned BF16	601/660 = 91.1%	609/660 = 92.3%	461/660 = 69.8%

Quantization costs ~2.8pp functional — serve the Q4, keep the BF16 for reference. The exact→functional gap is dominated by contract-correct placeholder slots (__PAYLOAD_*__, filled by the runtime at inference) and equivalent phrasings ((2 ** 5 % 7) ≡ 2 ** 5 % 7), not wrong behavior. Genuine weak spots: payload-tool selection (run_process/run_python confusion), git-range prompts (git_diff vs git_show/read_file), and occasional garbled paths — keep these behind runtime approval. Fifteen tools score a perfect 20/20 functional on BF16 (copy_file, create_directory, delete_file, delete_text, file_info, find_executable, git_checkout, git_commit, insert_text, move_file, read_directory, read_file, replace_text, run_powershell, web_extract).

Per-tool functional scores (Q4_K_M / BF16, 20 prompts each)

apply_patch 17 / 17 · ask_user 17 / 19 · calculator 17 / 17 · copy_file 20 / 20 · create_directory 20 / 20 · delete_file 20 / 20 · delete_text 20 / 20 · file_info 18 / 20 · find_executable 19 / 20 · get_time 17 / 17 · get_working_directory 17 / 18 · git_branch_list 19 / 19 · git_checkout 20 / 20 · git_commit 20 / 20 · git_diff 10 / 10 · git_log 15 / 16 · git_show 14 / 13 · git_stage 18 / 18 · git_status 14 / 16 · insert_text 20 / 20 · move_file 20 / 20 · process_info 19 / 17 · read_directory 20 / 20 · read_file 19 / 20 · replace_text 18 / 20 · run_powershell 20 / 20 · run_process 15 / 15 · run_python 17 / 18 · search_files 13 / 15 · web_extract 20 / 20 · web_open 19 / 19 · web_search 18 / 19 · write_file 13 / 18

Usage

Built for Relay (FunctionGemmaActionModel, --fg-gguf):

# llama-server with the GGUF (CPU is fine — ~0.6 s per translation)
llama-server -m functiongemma-270m-relay-tools-q4_k_m.gguf -c 32768 --port 8081

# inside Relay (auto-detects models/*.gguf, or pass --fg-gguf)
relay live --workspace /path/to/project --fg-gguf models/fg-tools.gguf
relay live --workspace /path/to/project --fg-gguf models/fg-tools.gguf --ui

Runtime settings: fg_max_tokens = 256 (larger values make it babble into input-independent calls), fg_temperature = 0.0. Wire format follows Google's FunctionGemma docs ( developer turn, call:name{arg:v}); the model emits no confidence — Relay parses strictly single-call and treats anything else as a retryable error.

Limitations

• Knows only the 33 Relay tools it was trained on; unknown tools fail closed.
• No multi-tool plans, no chit-chat, no refusals — ambiguity handling lives in the runtime gates, not in this model.
• Keep payload-tool outputs (execution, edits) behind approval; eval shows these are its weakest selections.

License Derivative of FunctionGemma — use is subject to the Gemma Terms of Use (https://ai.google.dev/gemma/terms). If you redistribute these weights, propagate the same terms.