Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets

Authors: Zongyou Yang, Yinghan Hou

arXiv ID: 2607.24268

Problem: A benchmark accuracy score collapses two distinct measurement questions into one number: whether the response reached an evaluable state (termination, answer exposure, parseability, completion length) and whether the answer was judged correct. When test-time methods are compared under matched token budgets, the budget itself changes how models fail - and the accuracy delta can be an artifact of the failure mixture, not the method.

Key Methodology:

A two-layer evaluation framework separating scorer-independent execution evidence from scorer-dependent correctness
2,550 outputs from five fixed Qwen and DeepSeek configurations on MATH and ARC-Challenge under matched 2,048-token limits
A coverage-audited targeted verification study on how candidate-selection and aggregation policies shift comparative accuracy estimates

Key Results:

49 of 450 Qwen MATH outputs terminate without a final answer, versus 5 of 300 DeepSeek MATH outputs and none of the 750 ARC outputs
The same 300 DeepSeek MATH question-model pairs show no missing-final-length termination at 8,192 tokens
Candidate-selection and aggregation policies can substantially alter comparative accuracy estimates

What it means for developers: "Same token budget" is not a fair comparison baseline: the budget determines which failure mode each model hits, and accuracy papers over the mixture. Any benchmark delta between models or test-time methods should be published with the pre-intervention execution states (termination rate, answer-exposure rate, parseability) and verification coverage - without them, the delta is not attributable. The same discipline applies to internal fleet evals: report how outputs fail (stalled, truncated, unparseable) alongside how often they pass.

Paper: arXiv:2607.24268