The Judge Is Leaving the Agent Loop

Last updated: August 3, 2026

Twenty-three percent. Sit with that before we get to the good news.

A research group replayed every validation command that a repair agent ran, at the exact working-tree state, against the original buggy code, the candidate state, and the developer's gold fix (BSG-VA, arXiv:2607.28871). Across 3,730 events in 643 rollouts, they classified each pass as bug-discriminating or not. Nearly a quarter of repair runs closed with a patch whose entire positive evidence base never touched the reported defect. The tests passed. The defect never came under test. Nobody asked the agent to be smarter - the agent did exactly what the harness measured it doing.

Now the good news, which is the same story from the other side. ECLoop (arXiv:2607.28815) interposed an execution layer between agent and repository that compiles per-task conditions - what the agent must observe before each edit or patch submission - tracks which conditions the trajectory has satisfied, and postpones any action whose conditions are unmet. No retraining, no scaffold change, just a gate. On all 500 SWE-bench Verified instances, with two models across two scaffolds, Pass@1 went up 4.8 to 11.8 points and token use went down up to 12.1 percent. Redirecting before unsupported actions is free.

One result says the loop's verdicts are hollow. The other says the fix is not a better verdict - it is a structure that does not need one. Last week we told you the fix for broken benchmarks is architecture, not smarter models (the-benchmark-fix-is-architectural). This week the argument goes one step further, and we want to defend that step properly: the judge is not being improved, it is being removed from the loop, at all three places an agent loop needs a verdict.

Where the verdicts used to live#

An agent loop has three moments where something must decide.

The reward. During training, something must say whether the trajectory was good. The classic answer is an LLM judge or a reward model, and we measured last week how biased those are (your-benchmark-is-lying-to-you). RLSVR (arXiv:2607.23802, COLM 2026) takes the other door: it transforms the task into a verifiable proxy environment whose internal rules generate the reward. Their concrete instance is SpyRL, which runs self-play on the social-deduction game Who Is the Spy: agents get asymmetric information, do the same target task, then vote to identify a designated spy whose identity was predetermined. The votes are fully verifiable - there is no judgment call about who won - while identification stays correlated with output quality. The result: SpyRL beats existing self-improvement methods on text summarization and creative writing, the task classes that supposedly require judges because nobody can check them. The reward is structural. There is no judge to be lenient.

The action. During execution, something must say whether the next step is safe to take. The default answer is the model's own judgment, which is exactly the thing ECLoop stops trusting: it does not ask the model how confident it is, it checks what the model has observed. The pattern was visible a few days earlier in HALO (arXiv:2607.27636), where a deterministic per-action admission gate - recheck the prerequisites before dispatch - kept 248 of 248 supported components passing while whole-response rejection kept 0 of 248. ECLoop is the same shape at SWE-bench scale, with the ablation to prove each piece earns its keep: condition compilation, tracking, and postponement each add distinct value, and structured evidence conditions beat an equivalent natural-language summary. Structure wins even inside the gate artifact.

The release. At deployment, something must say whether the artifact is ready. The default is review - human or LLM - which is the bottleneck we keep pricing. CS-RNR (arXiv:2607.28520) is the first method in its domain whose safety guarantee is a certificate the agent computes on the strategy it actually deploys: pooled action frequencies tracked with anytime-valid confidence sequences, a candidate admitted only once its evidence interval separates from an equilibrium reference, each candidate checked by a full best response against a user-specified budget, then committed atomically. Certify what you deploy, deploy what you certified. In Leduc hold'em it achieves 6.2x the steady-state gain of a money-verified binary gate while keeping every deployed strategy within budget, with all 36,000 audited hands inside the tolerance. It is game-domain, and we will come back to that, but the machinery - confidence sequences as the gate, the certificate on the deployed artifact rather than the claimed one - is domain-general.

Why this is not the audit wave again#

The audit wave fixed how we measure agents from outside. This wave fixes how the agent decides on the inside, and the two support each other in a way we did not expect.

The inside verdicts were always the weakest layer. BSG-VA priced it: 46 percent of positive comparable validation events carry no bug-discriminating information. The "it passed" signal - the thing every repair loop closes on - is nearly half noise about the actual defect. The audit wave's own tools are now the cheapest fix: the B-replay trick, showing the agent what the original code does on the same test, cut evidence-inadequate closure by 7.8 points (p = 0.0029), with the authors honestly flagging it below their prespecified 10-point smallest effect of interest. We will not oversell it either. But the shape is right: the counterfactual, not a stronger model, is the instrument.

And the safety layer, where the verdict matters most, is where the model judge is weakest in the most uncomfortable way. An audit of four agent-safety benchmarks under their official implementations (arXiv:2607.28685) found that an always-positive policy - refuse nothing, flag nothing - scores F1 0.690 on R-Judge, above five of the 21 models that actually discriminate. The three broad-coverage benchmarks rank the same 18 models differently, and a quarter of random 7-model subsets flip correlations by 0.6 or more, so "safe on an agent-safety benchmark" is not a statement until you name benchmark, metric, behavior, and panel. Then the kicker: capability predicts task success (rho +0.60) but correlates negatively with misalignment safety (rho -0.44, p < 0.001, robust to leaving any organization out). The models best at the job are, on this axis, the least safe ones. If your safety gate is an LLM opinion, you are betting on the exact axis where capability buys the wrong thing.

The bet#

Here is where we think this is going, stated so we can be graded on it.

By end of 2027, the default design question in agent engineering will be "where is the verifiable structure in this loop" - not "which judge do we use". Evidence-conditioned action gates, verifiable task transformations, and deploy-time certificates will be named, documented practices in mainstream harnesses and tuning guides, and judge-quality upgrades will stop being the headline lever in agent-quality content. The scarce skill will be verifiable-structure design: compiling what must be observed before a write, constructing the game whose verifiable outcome tracks the quality you want, designing the certificate for release. Judge tuning becomes the junior position; task transformation becomes the senior one.

We are wrong if the platforms keep shipping better-judge upgrades as their headline lever and gates stay research artifacts. We think that is unlikely, and not because we are sentimental about gates: the economics run the wrong way for judges. ECLoop's gate costs zero inference and pays for itself in tokens. A judge costs tokens, drifts over the loop's lifetime (the Rehearse confidence cliff we covered here), and needs auditing by something more trustworthy than itself. When the free option moves the number, the paid option has a pricing problem.

The counter-case, honestly#

Every leg of this is a single result from a two-day research wave, and we will grade our own claim the way we grade everyone else's. The caveats are real, so let us give them real steel.

RLSVR's transformation is per-task craft. Someone has to design the game for every new task class, and the correlation between the verifiable outcome (who was the spy) and the quality you wanted (is this summary good) is indirect - measured, not guaranteed. COLM acceptance is not deployment.

ECLoop's conditions are compiled by an LLM. The structure has a model-shaped head: the gate quality inherits model quality at the margin, which is why the structured-conditions-beat-natural-language ablation matters - it says the compile step is where the leverage is, but it is still a model doing the compiling. A gate is only as good as the conditions, and condition authoring is the skill we just called scarce, which means it is scarce for a reason.

CS-RNR is game-domain, and games are the friendliest possible deployment surface: the rules are written down. The confidence machinery transfers; the "checkable rules" part does not. Outside games, the certificate needs a verifier, and that is the whole problem again.

And the deepest one, which we keep returning to: structure can certify correctness, it cannot certify intent. The safety-benchmark audit's negative capability-safety correlation is not an argument for better safety judges, it is an argument that the misalignment axis is not checkable by the same structure that checks correctness. A gate certifies that the action observed what it needed. It does not certify that the action is what a human would want. The judge leaves the loop exactly where correctness is checkable, and stays - deservedly, expensively - where intent is the question. That is why the safety benchmarks are precisely where the numbers lie hardest: it is the hardest place to build structure.

What developers should do#

1. Before you add a judge to a loop, ask for the structure. What must be observed before this action fires? Can the task be transformed so the reward is verifiable, SpyRL-style? Can the release carry a certificate computed on the artifact you actually deploy? The papers above all started with those questions.

2. B-replay your repair loops. Replay each validation command against the buggy baseline state. A pass that does not discriminate the bug is not evidence of closure - and nearly a quarter of closures currently ship without any. It is a stateless, nearly free instrument, and it belongs in every CI gate that accepts agent patches.

3. Treat "it passed the tests" as a claim about the bug, not the patch. The audit wave taught us to demand breakdowns from vendors; the same discipline applies inside our own loops. Pass events need a discriminating-evidence column.

4. Keep the deterministic bottom first. Compile, typecheck, parity, tests - the free layer - before you spend a single judge token on the residual. The bottom is free because the toolchain already has it, and ECLoop is that principle applied to actions instead of artifacts.

5. When a vendor says "verified", ask what is structural and what is a model opinion. And when they say "safe", ask for benchmark, metric, behavior, and panel - an unqualified safety claim is now falsifiable on contact.

None of this means models stop mattering. It means the model's role in the loop is narrowing to the parts nobody has found structure for yet, and the people who get good at finding that structure are going to be the people building agent platforms in 2027. This is one thread of our developing long-range scenario, and the endgame is simple: the binding constraint in agent quality is not judgment, it is design. The judge is leaving the loop because the loop no longer needs one where it hurts most - and the residual, the part that genuinely needs judgment, is the product everyone will fight over.

FAQ#

What does it mean that the judge is leaving the agent loop?#

An agent loop needs a verdict at three points: the reward during training, the next action during execution, and the release at deployment. The argument here is that the fixes moving agent quality are not better LLM judges but structures that do not need one - verifiable reward games like SpyRL, evidence-conditioned action gates like ECLoop, and deploy-time certificates like CS-RNR.

What is ECLoop and how much did it help on SWE-bench Verified?#

ECLoop is an execution layer between the agent and the repository that compiles per-task conditions the agent must observe before each edit or patch submission, tracks which are satisfied, and postpones actions whose conditions are unmet. On all 500 SWE-bench Verified instances, across two models and two scaffolds, it raised Pass@1 by 4.8 to 11.8 points and cut token use by up to 12.1 percent, with no retraining.

What is B-replay and why should repair loops use it?#

B-replay means replaying each validation command against the original buggy code as well as the candidate state, so you can tell whether a passing test actually discriminates the reported bug. The BSG-VA study found nearly a quarter of repair runs closed with evidence that never touched the defect, and showing the agent what the original code does on the same test cut evidence-inadequate closure by 7.8 points.

Where do LLM judges still belong?#

Where intent, not correctness, is the question. Structure can certify that an action observed what it needed, but it cannot certify that the action is what a human would want, which is why safety and misalignment remain the hardest place to replace judgment with structure.

Continue Reading#

• Your Benchmark Is Lying to You
• The Fix for Broken Benchmarks Is Architecture, Not Smarter Models
• SWE-NFI: The Benchmark That Catches What Coding Agents Miss
• Agent Memory Is Moving Into the Model
• AI Agent Evaluation Tools Compared 2026
• The Plateau Was the Instrument
• The Response Looked Right. The Work Was Not Done. - the follow-up: completion claims decouple from delivery, and evidence-carrying termination now certifies the stop itself

Sources#

• ECLoop: evidence-conditioned execution gates - arXiv
• RLSVR/SpyRL: verifiable rewards by task transformation - arXiv
• CS-RNR: certificates on the deployed strategy - arXiv
• HALO: per-action admission gates - arXiv
• BSG-VA: validation evidence and bug-discriminating passes - arXiv
• Agent-safety benchmark validity audit - arXiv
• Interaction-centric agent failure taxonomy - arXiv
• LLM user simulations grounded in initial stances - arXiv
• Lean kernel soundness bug postmortem - Leo de Moura