Terminal-Bench Shows Harness Scaling Is the Coding-Agent Benchmark Now

Last updated: August 22, 2026

Long-Horizon-Terminal-Bench started as the benchmark that made coding agents look less magical and more measurable.

That is a good thing.

The August update is more interesting. A new paper called StateM argues that the benchmark is no longer only measuring model intelligence. It is measuring the execution system around the model: durable state, phase-local context, checked transitions, recoverable runbooks, and procedural practices that survive across attempts.

That should change how teams read every coding-agent leaderboard.

Most coding-agent demos still optimize for the first five minutes: clone the repo, find the bug, edit a file, run a test, show a diff. Real agent work is slower and messier. A serious task might need environment setup, failed installs, data inspection, repeated debugging, partial discoveries, and a long tail of "almost" work that binary pass/fail scoring throws away.

The new Long-Horizon-Terminal-Bench paper tries to measure that missing middle. It introduces 46 terminal tasks across nine categories, including software engineering, experiment reproduction, scientific computing, multimodal analysis, and interactive games. Each task runs in a containerized terminal environment and is decomposed into graded subtasks, so an agent can get credit for progress instead of only receiving a final solved/failed label.

That matters for anyone buying, building, or managing coding agents. The question is no longer simply "can an agent solve SWE-bench style issues?" The more practical question is whether it can keep making grounded progress after the first plan breaks.

If you have been following the agent-eval cluster here, this sits directly next to Dockerless verification, Microsoft's CLI coding-agent rollout study, and the older argument that agent evals need baseline receipts. LHTB adds the endurance layer.

StateM adds the next layer: harness scaling. The paper's claim is not "a better model solved the benchmark." It is "the same model behaves differently when the runtime makes state, lessons, phase boundaries, and recovery rules explicit." That is the same direction we covered in Harness Handbook, DataFlow-Harness, and SkillHone: coding-agent performance increasingly comes from the system that keeps the model oriented.

What LHTB Actually Tests#

The benchmark is built around terminal-use agents. The agent receives a task, operates in a terminal, and has up to 90 minutes per run under the paper's reported evaluation setup.

The paper's headline numbers are intentionally sobering:

Those numbers come from arXiv v2, revised July 13, 2026. The Hugging Face paper page also shows the paper as the #1 paper of the day with active community discussion and a linked GitHub/project page.

The interesting part is not only the low solve rate. We already know frontier agents still fail. The useful part is the measurement shape: dense reward, long execution windows, hidden verifiers, and task decompositions that let researchers see where the agent ran out of steam.

That is closer to the way developers actually evaluate agents in a repo. You care whether the agent found the right subsystem, wrote a reasonable failing test, reduced the bug, noticed its own bad assumption, and stopped before corrupting state. A final green check is important, but it is not the only evidence.

What StateM Changes#

StateM lands one month later and flips the conversation from "agents stall" to "harnesses can be scaled."

The abstract reports several numbers that are worth keeping separate:

Do not flatten that into "StateM is the new leaderboard winner." The public Terminal-Bench 2.1 leaderboard still lists verified submissions separately, with a note that submissions may not modify timeouts or resources. The leaderboard snapshot I checked on August 22 showed Claude Code with Fable 5 at 83.8% and Codex with GPT-5.5 xhigh at 83.1% among verified entries.

The more useful reading is architectural. StateM is testing whether agent runs improve when the harness preserves lessons as explicit state instead of asking the model to rediscover them in a long transcript. That is not a benchmark trick to ignore. It is exactly the production problem teams hit when long-running agents repeat failed setup steps, forget which invariant mattered, or lose a useful diagnostic after context compaction.

In other words: the model still matters, but the harness is now a first-class optimization surface.

Why Binary Agent Evals Are Too Thin#

Binary evals are easy to explain: pass or fail. They are also easy to overread.

If an agent fails a long task after 80 minutes, that failure can mean several different things:

• It never understood the task.
• It understood the task but chose the wrong plan.
• It made useful progress but hit an environment problem.
• It solved the core bug but failed a packaging or artifact step.
• It looped on a near miss because it lacked a better verifier.
• It found the right answer but could not prove it within the harness.

Those are not the same failure. They should not produce the same product decision.

This is where LHTB's partial-credit design is useful. It gives teams a vocabulary for "the agent is bad at long horizon recovery" instead of "the model scored low." That distinction matters because different fixes live at different layers.

If the agent fails at planning, you might need better task decomposition. If it fails at verification, Dockerless-style pre-CI checks or stronger local harnesses may help. If it fails after compaction, your context policy is suspect. If it burns 9.9M tokens on average and still cannot close, the model may not be the bottleneck your budget owner thinks it is.

StateM makes this sharper. A benchmark score can move because the model got better, because the harness got better, because the budget changed, because the verifier changed, or because the runbook learned the benchmark's shape. Those are different claims. A serious agent eval should name which layer changed before anyone quotes the final percentage.

The Practical Takeaway For Teams#

Do not read LHTB as "coding agents are bad." That is the lazy take.

Read it as evidence that coding agents need endurance metrics before they deserve bigger blast radius. The same agent can be useful for narrow issue work, risky for autonomous multi-hour refactors, and excellent as a research assistant that stops at a reproducible evidence bundle.

For teams rolling agents into daily engineering, I would turn the paper into four operating questions.

First: what is your partial-credit rubric? If your internal eval only records whether the PR merged, you are missing signal. Track whether the agent found the right files, created a relevant test, preserved public APIs, minimized diff size, and explained residual risk.

Second: where does the agent lose time? LHTB reports an average 85.3 minutes per run. In a real company, that is not just latency. It is CI queue time, review time, token spend, and developer attention. Instrument environment setup, edit loops, test retries, and repeated tool calls separately.

Third: what counts as a safe stop? A long-running agent should not keep mutating a repo forever because it has not reached 1.0 reward. The best production agents will learn when to stop with a clean handoff: the failing command, the smallest repro, the files touched, and the next human decision.

Fourth: does your benchmark match your work? LHTB includes scientific computing and experiment reproduction alongside software engineering. That breadth is useful, but your internal scorecard should still reflect your own repo shapes. A frontend team, infra team, and data platform team should not all optimize for the same task mix.

That is the bridge from research benchmark to engineering policy.

StateM adds three more questions.

Fifth: what state is durable? If the agent learns that the test harness needs a seed, that a dependency install is broken, or that one file is generated, does that fact become a checked precondition? Or does it live as fragile prose somewhere deep in the transcript?

Sixth: who can inspect the runbook? StateM's useful claim is not just persistence. It is inspectable persistence. Developers should be able to see the rules an agent is carrying forward, remove benchmark-specific residue, and distinguish general project practice from one-off recovery lore.

Seventh: what changed between baseline and candidate? If a vendor improves Terminal-Bench by adding a special runtime, that may be valuable. But you should compare model, harness, timeout, tool policy, cost, and verification separately. The buying decision is about the whole agent system, not a single model row.

What To Watch Next#

The community leaderboard is the part to monitor. A static paper result is a snapshot. A reproducible long-horizon benchmark becomes useful when model providers, agent builders, and independent teams can submit comparable runs under the same budget and verifier rules.

The second thing to watch is whether vendors start optimizing for dense progress, not only final solve rate. A coding agent that gets 30% of a hard task done in a clean, reviewable way may be more valuable than one that occasionally solves the whole task after a chaotic million-token drift.

The third thing to watch is cost attribution. Microsoft's field study measured rollout and PR output at organizational scale. LHTB shows what a single hard task can consume under controlled conditions. StateM claims a final-score run around $15 versus a much higher GPT reference. Put those together and the serious enterprise question becomes: which work should receive 90-minute agent attempts, and which work should stay in the human-review loop after a five-minute scout pass?

That is where the next useful tooling wave lives: scout agents, long-horizon agents, verifiers, rollback logs, stateful runbooks, and manager dashboards that distinguish useful partial work from expensive thrashing.

My Read#

Long-Horizon-Terminal-Bench is not a replacement for SWE-bench, Terminal-Bench, internal evals, or production telemetry. It is a needed pressure test for the part of agent work that demos usually skip.

StateM does not cancel that lesson. It updates it. The best use is not to crown a winner. It is to ask better questions before you hand agents longer tasks:

• Can it recover after the first plan fails?
• Can it preserve state across hundreds of tool calls?
• Can it turn postmortem lessons into explicit future preconditions?
• Can it make reviewable partial progress?
• Can it stop cleanly when the verifier says no?
• Can it justify the token and wall-clock cost?

That is a much better bar than "the demo made a pull request."

For a broad primer on the category, start with what an AI coding agent is in 2026. For the evidence stack around evaluating those agents, pair this with agent evals need baseline receipts, security agents need repro harnesses, Dockerless verification, and Harness Handbook's map of agent behavior.

FAQ#

What is Long-Horizon-Terminal-Bench?#

Long-Horizon-Terminal-Bench is a benchmark for terminal-use AI agents. It contains 46 long-horizon tasks across nine categories and uses dense partial-credit grading instead of only final pass/fail scoring.

Why does Long-Horizon-Terminal-Bench matter for coding agents?#

It tests the part of coding-agent work that short demos usually hide: long planning loops, repeated debugging, environment friction, verifier failures, and partial progress over many tool calls.

Did any model solve Long-Horizon-Terminal-Bench reliably?#

No. In the arXiv v2 results, the strongest tested model reached 15.2% pass@1 at the 0.95 reward threshold and 10.9% at the perfect-reward threshold. The reported mean pass rates were much lower.

Should teams use LHTB before adopting coding agents?#

Use it as a research signal, not as your only buying test. Teams should still build internal evals that match their own repo types, risk tolerance, CI setup, and review workflow.

How should engineering leaders apply this benchmark?#

Track partial progress, safe stopping, time spent by phase, and cost per useful artifact. Do not only track whether an agent eventually opened a pull request.

What is StateM?#

StateM is an agent runtime described in an August 2026 paper. It organizes long agent runs around durable states, phase-local context, checked transitions, recoverable runbooks, and versioned procedures so lessons from earlier execution can guide later steps.

Does StateM prove models are less important than harnesses?#

No. It shows that model choice is not the only lever. The same family of models can move significantly when the harness preserves useful state, recovery rules, and task procedures. For buyers and builders, the correct unit of comparison is the model plus the harness plus the verifier plus the budget.

Should teams optimize for Terminal-Bench scores?#

Use Terminal-Bench as one signal, not as the product goal. A strong internal eval should include your repo shapes, allowed tools, security policy, review standards, cost ceiling, and failure-handoff rules.

Sources#

• Long-Horizon-Terminal-Bench on arXiv, v2 checked July 14, 2026.
• StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling, checked August 22, 2026.
• Long-Horizon-Terminal-Bench project page, checked July 14, 2026.
• Long-Horizon-Terminal-Bench on Hugging Face Papers, checked July 14, 2026.
• StateM on Hugging Face Papers, checked August 22, 2026.
• Terminal-Bench 2.1 leaderboard, checked August 22, 2026.
• Google Trends checked August 22, 2026 for StateM Terminal-Bench, Terminal-Bench, SWE-bench, coding agent benchmark, AI coding agents, Claude Code, agent harness, coding agent harness, agent evaluation, and AI agent benchmark. Exact StateM demand was too sparse for numeric claims; broader lane numbers are listed in the research notes table.