Coding Agent Harness Benchmark: Same Model, 17.5x Cost Gap

TL;DR
FrontierHarness ran one model through 9 coding agent harnesses. Pass rates sat within 17 points, median cost per pass spread 17.5x. What to measure before you pick Claude Code, Codex, Pi or OpenCode.
Last updated: November 16, 2026
If you search for a coding agent harness benchmark, you mostly find model leaderboards with the harness held fixed. FrontierHarness Eval flips that: it holds the model fixed and varies the harness. The headline from its repository is blunt - similar pass rates, 17.5x cost differences - and it lands in the same place as an independent study from September. Your choice of Claude Code, Codex, Pi or OpenCode may matter more to your bill than to your merge rate.
What the primary source says#
The FrontierHarness Eval repository ran one model, Kimi K3, through nine harnesses (12 configurations in total) on the same 30 software-engineering tasks, with the runtime held constant. The published table:
| Harness | Pass rate | Median cost per pass | Cache hit (median cell) | Median time |
|---|---|---|---|---|
| Codex | 66.7% | $3.47 | 88.0% | 6m 43s |
| Claude Code | 63.3% | $18.34 | 67.8% | 9m 38s |
| Pi | 60.0% | $2.43 | 79.4% | 7m 33s |
| OpenCode | 50.0% | $3.24 | 78.4% | 6m 27s |
| Exo Harness | 53.3% | $1.05 | 70.3% | 6m 17s |
| Hermes | 50.0% | $2.90 | 85.9% | 6m 58s |
The repository's own summary names Codex for quality, Pi for balance and Exo Harness for cost. Three things in the table are worth more than the ranking:
- Cost spread is far wider than pass-rate spread. Pass rates run from 50.0% to 66.7%. Median cost per pass runs from $1.05 to $18.34, the 17.5x in the headline.
- Claude Code's 63.3% is within one task of Codex's 66.7% (30 tasks, so each task is 3.3 points), but costs about 5x as much per pass in this run.
- Cache behavior differs. Claude Code's median cache rate is 67.8% against Codex's 88.0%. Prompt caching is where a large share of agent cost goes, so a harness that busts the cache pays for it on every turn.
Caveats the repository itself states or implies: this is one model, 30 tasks, and the tested versions were Codex 0.148.0 and Claude Code 2.1.237, both far behind today's releases. Claude Code is also running a non-Anthropic model here, which is not the configuration Anthropic tunes it for.
The same finding from an independent study#
This is not a one-off. In September, Arena and Berkeley researchers published HarnessTax, testing 21 model-harness pairs across Claude Code, Codex CLI and Pi on SWE-bench Lite and Terminal-Bench 2.0. Their findings, from the post:
- Harness choice moved success rate by within +/-2% on SWE-bench Lite and about +/-5% on Terminal-Bench 2.0, but the same model could cost up to 5x more.
- Claude Fable 5 solved 97.8% of attempts in Claude Code and 96.7% in both Codex and Pi, yet Claude Code cost $1.33 per attempt against Pi's $0.67.
- Across shared models, Claude Code cost about 2.0x Pi and 1.6x Codex on SWE-bench Lite.
- A likely cause is the first call: Claude Code's mean initial context was over 10x Pi's, with longer instructions and larger tool schemas. Pi gets by with four tools (read, write, edit, bash).
- An alternative harness had the highest observed success rate in nine of twelve Anthropic and OpenAI model comparisons.
The authors flag that both benchmarks are open and models may have seen them in training, so the numbers describe those tasks, not your repo.
The angle: stop comparing pass rate alone#
Two independent runs agree that pass rate compresses and cost spreads. That changes what you should measure.
Who benefits. Teams paying API rates for high-volume agent work, such as CI-driven fixes or batch refactors. A lighter harness at similar success is a direct cut in spend. Vendors with lean harnesses and strong caching win the cost-per-pass column even when they do not win the pass-rate column.
Who is exposed. Harnesses whose value rests on richer tooling and bigger system prompts. That scaffolding can earn its keep on hard, long-horizon work. Both sources say as much: HarnessTax notes richer features may still help other models and workloads. But on short benchmark-sized tasks it is overhead you pay on every call.
What to do Monday. Run your own ten tasks through two harnesses with the same model and log three numbers per run: pass, total cost, and cache hit rate. The FrontierHarness repository ships a workflow for this. Its README documents installing the evaluation skill with npx skills add frontier-harness-eval/eval --skill frontierharness-eval and then asking your agent to evaluate a harness against one Terminal-Bench task and one DeepSWE task to start. It also warns that new runs get no leaderboard rank unless you also run a control harness under the same network policy, so compare against your own control, not their table. If you are already budgeting agent spend, pair this with harness engineering and the token budget and the context costs described in the agent harness map.
What people are actually saying#
I could not reach Hacker News from the environment this was drafted in, so there is no HN thread summarized here. What I did see:
- A r/LocalLLaMA thread points at a separate Hugging Face harness leaderboard, described as a tool for picking a model, stage and reasoning effort and comparing harnesses. I read only the thread's title and description, not its comments.
- The counter-case. On that Hugging Face Space, per its page excerpt, Claude Code (v2.1.300) posts a 49.0% macro score against Codex (v0.162.1) at 47.7%, the opposite order from the FrontierHarness table, with a 23-point best-to-worst harness gap (OpenCode 52.5% versus OpenClaw 29.7%). Different tasks, models and versions give a different leader. That is the point: rankings are workload-specific.
What this does and does not prove#
It does show that harness choice is a real cost lever, and that it is measurable. It does not show that any named harness is "better". Both studies used open benchmarks, small task counts, a fixed model and effort setting, and versions that have since moved on (the harnesses in this FrontierHarness run are many releases behind current; see our notes on recent Codex CLI changes). Vendors ship fast, so treat every table here as a dated snapshot and re-run before committing a team to one tool. For model-side context on the model used in the FrontierHarness run, see the Kimi K3 developer guide; for how these harnesses differ in features, see Claude Code vs Codex vs Cursor vs OpenCode.
Continue Reading#
- What Is an Agent Harness? A Developer's Map for 2026 - the pillar guide on what a harness is and how to choose one
- Claude Code vs Codex vs Cursor vs OpenCode - the feature-level comparison of the harnesses benchmarked here
- Harness Engineering and the Token Budget - where harness tokens go and how to cap them
- We Read DeepSeek Harness - a code read of one of the harnesses on the leaderboard
- Kimi K3 Developer Guide - the model every harness ran in the FrontierHarness test
Sources#
- FrontierHarness Eval repository and results - primary source for the table, methodology and harness versions
- FrontierHarness live leaderboard - interactive results referenced by the repository
- HarnessTax: How Much Does the Harness Matter for Coding Agents? - Arena, September 16, 2026
- Harness Leaderboard on Hugging Face - counter-case leaderboard, read from page excerpt
- r/LocalLLaMA thread on the harness leaderboard
Get the next deep dive like this in your inbox
One email a week on AI Coding and the rest of the AI dev stack. Free.
Read next on AI coding tools
What Is an Agent Harness? A Developer's Map for 2026
An agent harness is everything around the model: loop, tools, context, sandbox. How Pi, DeepSeek, Codex and Claude Code differ, and how to choose.
5 min readClaude Code vs Codex vs Cursor vs OpenCode: Which Agent Ships More Code?
Four agents, same tasks: honest trade-offs between Claude Code, Codex CLI, Cursor Agent and OpenCode from a developer shipping production apps with all of them.
10 min readWe Read DeepSeek Harness: What 453K Lines of Agent Runtime Actually Say
DeepSeek Harness (dsh) code read: a 453K-line plugin runtime on a vendored Cordis fork, three patterns worth stealing, and the gaps the launch does not mention.
9 min readNew here? Start with
Technical content at the intersection of AI and development. Building with AI agents, Claude Code, and modern dev tools - then showing you exactly how it works.







