HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

Authors: Varun Ursekar, Apaar Shanker, Yash Maurya, Shehab Yasser, Vijay S. Kalmath, Veronica Chatrath, Yuan Xue (Scale AI)

arXiv ID: 2608.06301

Problem: An LLM's deployed capability depends not only on its weights but on the harness - the prompts, tools, control flow, memory, and orchestration code surrounding it. Automated harness optimization (iterative, evaluation-guided improvement of a harness by an AI system) is both a route to better agents and a demanding capability in itself, yet there was no common protocol for measuring how well frontier LLMs do it.

Key Methodology:

An optimizer (LLM paired with a coding harness) receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget
The optimizer edits the harness and nominates a final candidate, scored by normalized gain over the seed on a held-out test partition inaccessible throughout search
A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit
5 frontier LLMs evaluated as optimizers, both under a shared coding harness and under their native harnesses, across 4 downstream tasks over 111 scored runs

Key Results:

Optimizer models separate more than the coding harnesses they act through - model choice dominates harness choice for optimization ability
Native harnesses are not consistently superior to a shared coding harness
Gains vary substantially across tasks and seed regimes, with large headroom remaining

Applied Context: Harness engineering becomes a measurable, benchmarkable capability with an audit trail. For teams running agent fleets, this provides a protocol to test "is our harness good, or is the model good" - and the headroom finding supports the thesis that harness work is where unspent agent performance lives.