Long-Horizon Terminal Bench Shows Why Coding Agents Still Stall

Last updated: July 14, 2026

Long-Horizon-Terminal-Bench is the kind of benchmark that makes coding agents look less magical and more measurable.

That is a good thing.

Most coding-agent demos still optimize for the first five minutes: clone the repo, find the bug, edit a file, run a test, show a diff. Real agent work is slower and messier. A serious task might need environment setup, failed installs, data inspection, repeated debugging, partial discoveries, and a long tail of "almost" work that binary pass/fail scoring throws away.

The new Long-Horizon-Terminal-Bench paper tries to measure that missing middle. It introduces 46 terminal tasks across nine categories, including software engineering, experiment reproduction, scientific computing, multimodal analysis, and interactive games. Each task runs in a containerized terminal environment and is decomposed into graded subtasks, so an agent can get credit for progress instead of only receiving a final solved/failed label.

That matters for anyone buying, building, or managing coding agents. The question is no longer simply "can an agent solve SWE-bench style issues?" The more practical question is whether it can keep making grounded progress after the first plan breaks.

If you have been following the agent-eval cluster here, this sits directly next to Dockerless verification, Microsoft's CLI coding-agent rollout study, and the older argument that agent evals need baseline receipts. LHTB adds the endurance layer.

What LHTB Actually Tests#

The benchmark is built around terminal-use agents. The agent receives a task, operates in a terminal, and has up to 90 minutes per run under the paper's reported evaluation setup.

The paper's headline numbers are intentionally sobering:

Those numbers come from arXiv v2, revised July 13, 2026. The Hugging Face paper page also shows the paper as the #1 paper of the day with active community discussion and a linked GitHub/project page.

The interesting part is not only the low solve rate. We already know frontier agents still fail. The useful part is the measurement shape: dense reward, long execution windows, hidden verifiers, and task decompositions that let researchers see where the agent ran out of steam.

That is closer to the way developers actually evaluate agents in a repo. You care whether the agent found the right subsystem, wrote a reasonable failing test, reduced the bug, noticed its own bad assumption, and stopped before corrupting state. A final green check is important, but it is not the only evidence.

Why Binary Agent Evals Are Too Thin#

Binary evals are easy to explain: pass or fail. They are also easy to overread.

If an agent fails a long task after 80 minutes, that failure can mean several different things:

• It never understood the task.
• It understood the task but chose the wrong plan.
• It made useful progress but hit an environment problem.
• It solved the core bug but failed a packaging or artifact step.
• It looped on a near miss because it lacked a better verifier.
• It found the right answer but could not prove it within the harness.

Those are not the same failure. They should not produce the same product decision.

This is where LHTB's partial-credit design is useful. It gives teams a vocabulary for "the agent is bad at long horizon recovery" instead of "the model scored low." That distinction matters because different fixes live at different layers.

If the agent fails at planning, you might need better task decomposition. If it fails at verification, Dockerless-style pre-CI checks or stronger local harnesses may help. If it fails after compaction, your context policy is suspect. If it burns 9.9M tokens on average and still cannot close, the model may not be the bottleneck your budget owner thinks it is.

The Practical Takeaway For Teams#

Do not read LHTB as "coding agents are bad." That is the lazy take.

Read it as evidence that coding agents need endurance metrics before they deserve bigger blast radius. The same agent can be useful for narrow issue work, risky for autonomous multi-hour refactors, and excellent as a research assistant that stops at a reproducible evidence bundle.

For teams rolling agents into daily engineering, I would turn the paper into four operating questions.

First: what is your partial-credit rubric? If your internal eval only records whether the PR merged, you are missing signal. Track whether the agent found the right files, created a relevant test, preserved public APIs, minimized diff size, and explained residual risk.

Second: where does the agent lose time? LHTB reports an average 85.3 minutes per run. In a real company, that is not just latency. It is CI queue time, review time, token spend, and developer attention. Instrument environment setup, edit loops, test retries, and repeated tool calls separately.

Third: what counts as a safe stop? A long-running agent should not keep mutating a repo forever because it has not reached 1.0 reward. The best production agents will learn when to stop with a clean handoff: the failing command, the smallest repro, the files touched, and the next human decision.

Fourth: does your benchmark match your work? LHTB includes scientific computing and experiment reproduction alongside software engineering. That breadth is useful, but your internal scorecard should still reflect your own repo shapes. A frontend team, infra team, and data platform team should not all optimize for the same task mix.

That is the bridge from research benchmark to engineering policy.

What To Watch Next#

The community leaderboard is the part to monitor. A static paper result is a snapshot. A reproducible long-horizon benchmark becomes useful when model providers, agent builders, and independent teams can submit comparable runs under the same budget and verifier rules.

The second thing to watch is whether vendors start optimizing for dense progress, not only final solve rate. A coding agent that gets 30% of a hard task done in a clean, reviewable way may be more valuable than one that occasionally solves the whole task after a chaotic million-token drift.

The third thing to watch is cost attribution. Microsoft's field study measured rollout and PR output at organizational scale. LHTB shows what a single hard task can consume under controlled conditions. Put those together and the serious enterprise question becomes: which work should receive 90-minute agent attempts, and which work should stay in the human-review loop after a five-minute scout pass?

That is where the next useful tooling wave lives: scout agents, long-horizon agents, verifiers, rollback logs, and manager dashboards that distinguish useful partial work from expensive thrashing.

My Read#

Long-Horizon-Terminal-Bench is not a replacement for SWE-bench, Terminal-Bench, internal evals, or production telemetry. It is a needed pressure test for the part of agent work that demos usually skip.

The best use is not to crown a winner. It is to ask better questions before you hand agents longer tasks:

• Can it recover after the first plan fails?
• Can it preserve state across hundreds of tool calls?
• Can it make reviewable partial progress?
• Can it stop cleanly when the verifier says no?
• Can it justify the token and wall-clock cost?

That is a much better bar than "the demo made a pull request."

For a broad primer on the category, start with what an AI coding agent is in 2026. For the evidence stack around evaluating those agents, pair this with agent evals need baseline receipts, security agents need repro harnesses, and Dockerless verification.

FAQ#

What is Long-Horizon-Terminal-Bench?#

Long-Horizon-Terminal-Bench is a benchmark for terminal-use AI agents. It contains 46 long-horizon tasks across nine categories and uses dense partial-credit grading instead of only final pass/fail scoring.

Why does Long-Horizon-Terminal-Bench matter for coding agents?#

It tests the part of coding-agent work that short demos usually hide: long planning loops, repeated debugging, environment friction, verifier failures, and partial progress over many tool calls.

Did any model solve Long-Horizon-Terminal-Bench reliably?#

No. In the arXiv v2 results, the strongest tested model reached 15.2% pass@1 at the 0.95 reward threshold and 10.9% at the perfect-reward threshold. The reported mean pass rates were much lower.

Should teams use LHTB before adopting coding agents?#

Use it as a research signal, not as your only buying test. Teams should still build internal evals that match their own repo types, risk tolerance, CI setup, and review workflow.

How should engineering leaders apply this benchmark?#

Track partial progress, safe stopping, time spent by phase, and cost per useful artifact. Do not only track whether an agent eventually opened a pull request.

Sources#

• Long-Horizon-Terminal-Bench on arXiv, v2 checked July 14, 2026.
• Long-Horizon-Terminal-Bench project page, checked July 14, 2026.
• Long-Horizon-Terminal-Bench on Hugging Face Papers, checked July 14, 2026.
• LHTB community leaderboard, checked July 14, 2026.
• Google Trends check attempted July 14, 2026 and returned HTTP 429 through pytrends, so this article does not use fresh Google Trends numbers.