SWE-bench Pro: 10 agent harnesses x 2 models

Every rollout behind the harness-vs-model comparison: 10 coding agent harnesses run against both models on the same 250 held-out SWE-bench Pro tasks, one rollout per task, 5,000 rollouts in total. A third, partial arm runs Gemma 4 with thinking disabled; see Thinking-off ablation.

The harness rankings do not transfer between the two models: the rank correlation between them is -0.05. Vendor-shipped harnesses fall on the small model (codex 2nd of 10 to 9th, claude_code 3rd to 7th) while model-agnostic ones climb (crush 7th to 1st, opencode 8th to 2nd).

Thinking-off ablation

gemma-4-26B-A4B-it-thinkoff is the same sweep with the model's thinking disabled. It was never finished: 8 of the 10 harnesses, and between 86 and 191 of the 250 tasks each rather than all of them, so the denominators differ per harness and the missing tasks are the ones that had not run yet rather than a random sample. It is published because the direction is already clear, not because the numbers are final.

Turning thinking off costs almost every harness, and it costs most exactly the harnesses that did best with it on: crush, opencode and pi give up two thirds of their score, while claude_code does not move. So the harness ranking on this model is partly a ranking of how well each one exploits the thinking budget, not of its scaffolding alone.

Layout

swebench_pro/test/<model>/<harness>/<task_id>/t0/    one rollout
swebench_pro/test/<harness>/<model>.summary.json     pass@1, Wilson 95% CI, coverage

The thinking-off arm has rollout directories but no .summary.json: it was never aggregated by the sweep runner, so its numbers above come from the scores.json files directly.

Each rollout directory holds:

What is not here

Harness home directories (caches, sqlite journals, downloaded skills, shell snapshots) are excluded. They are an order of magnitude larger than the record itself -- one codex rollout is 68MB of which under 1MB describes the run -- and they are where a harness writes a credential if it writes one at all. Only the two directories above, which hold real transcripts, survive that cut.

Token-shaped strings matching hf_[A-Za-z0-9]{20,} are replaced with <REDACTED_HF_TOKEN>. Private-key headers and sk- prefixes that remain are content from the benchmark's own task repositories (gravitational/teleport, flipt-io/flipt) and from branch names like task-Fix-Timezone, not credentials.

Caveats

250 tasks and one rollout put a Wilson 95% interval of roughly +/-6pp around any single pass@1, so neighbouring harnesses are not separated. The reshuffling of the ranking between the two models is a joint pattern across ten harnesses and is the robust result. The thinking-off arm is wider still, both because it has fewer tasks and because it has a different set of them per harness.

Costs quoted elsewhere use list API rates as of 2026-08-06: GLM-5.2 at Z.ai's own card ($1.40/$4.40 per M tokens), Gemma 4 at DeepInfra, the cheapest of 13 providers ($0.07/$0.34).