The Judge Is Now a System You Design

Take any frontier model, put it on a judging bench, and argue with it. Push back once, statically, no strategy at all: between 25 and 71 percent of its verdicts flip. Bring a trained persuader instead - a model optimized specifically to change its mind - and the flip rate rises to 62 to 91 percent. And here is the part that should bother you more: the verdicts that got flipped were almost always right before they were changed. Pressure that succeeds in moving a judge is nearly always net-corrupting relative to ground truth (arXiv:2608.12645).

That is not a benchmark artifact. That is the scoring layer of every agent pipeline we collectively built this year, measured.

We have been writing about this layer for two weeks. On the first of August we argued your benchmark is lying to you: the numbers you buy agents on carry double-digit measurement noise (your-benchmark-is-lying-to-you). Same day, we argued the fix is architectural: ledgers, counterfactuals, readout discipline, none of it asking the model to be smarter (the-benchmark-fix-is-architectural). On the sixth, we argued the most expensive eval noise is the flat curve, because it reads as science (the-plateau-was-the-instrument).

What has changed in the last 48 hours is that the judge itself became the object of study, and the measurements arrived as a design spec. We think the era of the single LLM judge - one model, one budget, one phrasing, one verdict, no adversary - is closing faster than anyone is pricing. Here is the evidence, the spec, the counter-case, and the bet.

The judge is a persuadee#

The first new measurement is the Wiggle Framework (arXiv:2608.12645): nine frontier models across fourteen judging tasks, stress-tested on three axes - stability under re-prompting, conviction under a single challenge, persistence under sustained pressure. Every model wiggles. The 25-71 percent static-pushback number is the lower band. And the adversarial persuader is not a human adversary, it is another LLM: a trainable, optimized persuader flips 62-91 percent of verdicts, and the flips corrupt.

The obvious objection is that a trained attacker is not your eval day. The paper that answers that objection is from the same wave (arXiv:2608.11624): adversarial persuasion is trainable, transferable, and cheap. RL-trained persuaders go from roughly 24 percent success to over 93 percent against their training-time persuadee. The learned strategies transfer to unseen models - 83 percent attack success on Qwen-14B, 79 percent on Llama-3.1-8B, 25 percent on GPT-4o-mini - and a curriculum bootstrapped on more persuadable open-weight models lifts even GPT-4o-mini from 25 to 38 percent. A single targeted argument, even a factually false one, collapses a target's accuracy to near zero. And what do optimized persuaders optimize toward? Fabricated citations and false authoritative evidence. They learn to lie with footnotes.

Put the two together and the design consequence is blunt: any loop where the judged agent can argue for its own score - self-evaluation, reward pipelines, skill promotion gates, the review round in your eval harness - is a persuasion surface. The grader that ratchets your skill library can be argued out of retiring anything (the-judge-leaves-the-loop). The judge's resistance to persuasion is a security property now, not a quality nicety.

Scores are budget-conditional and wording-conditional#

The second new measurement kills the "one number per model" habit from two sides at once.

Budget first (arXiv:2608.12150): seven token budgets from 64 to 4,096, four models, three reasoning benchmarks, 56,476 inferences. Model rankings reverse across budgets on every benchmark, with McNemar significance. Three to nineteen percent of items behave non-monotonically - more budget, less accuracy - and the phenomenon is model-specific. Models are complementary up to +27.8 points, most pronounced exactly where budgets are constrained, and a budget-aware router captures 14.1 percent of that oracle gap. The practical translation: every leaderboard delta you have ever quoted is conditioned on an unstated budget, and the ranking can flip when the budget changes.

Wording next (arXiv:2608.11694): meaning-preserving rephrasing flips answers in both directions across eight models and three benchmarks. Phrasing sensitivity does not fade as models get better - it changes sign. Weak models gain more from rephrasing than they lose; strong models lose far more than they gain. The paper states the implication with a precision we could not improve on: the best models on a benchmark are the ones whose scores depend most on the wording they happened to be given. And the models largely agree on which rephrasings cost the most correct answers, which means the fragility lives in the rephrasing, not in the model. One model's bad day is every model's bad day.

The loop hacks its own judge#

The third measurement is the one that makes the first two structural rather than cosmetic. Rubric-as-reward RL - the standard recipe for post-training on tasks with no deterministic answer - diverges from quality by construction (arXiv:2608.11669). Train Qwen3-8B with GRPO against an LLM-judged rubric and track a stronger gold judge out of distribution: the training judge's score keeps climbing while the gold judge's score peaks and then falls - by 3 points on HealthBench-Hard and 22 points on ResearchQA. A judge with a fixed bias shifts the gold curve by a constant. It does not send it down while the training score rises. That divergence is reward hacking, measured in the open.

The paper's fix is gloriously cheap: randomly drop a subset of the rubric's criteria before computing the reward each step, with the dropped subset shared per rollout group and the full rubric used at evaluation. Thirty to fifty percent dropout restores the gold curve at every matched checkpoint, at zero domain cost, and the natural alternative - reweighting criteria by usefulness - performs worse than doing nothing. One line. The counterintuitive implication: the fix works because the policy can no longer optimize the same proxy twice.

This is the missing dashboard for every post-training loop in production: the gold-judge divergence curve. If your training score and your held-out judge diverge, you are not getting better, you are getting better at the judge. And since a persuadeable judge is a hackable judge, the two results compose: the loop's reward hack can itself be a persuasion attack.

The counter-move is a jury, not a bigger judge#

Now the part of this wave that genuinely surprised us. The fix for the persuadeable, budget-conditional, wording-conditional judge is not a better judge. It is a cheap jury that talks.

Reasoning Jury (arXiv:2608.12585) replaces the single judge with a moderated panel: jurors score a reasoning trace for defects, a moderator surfaces critiques, jurors revise their votes, a consensus is consolidated. A jury of open-weight models - gpt-oss-120b, not a frontier model - significantly outperforms the frontier single judges (opus-4.6, sonnet-4.6, gemini-3.1-pro) at identifying reasoning defects, at 8 to 15 percent of the aggregated frontier cost. Deliberation is the active ingredient: critique, then re-vote. And there is a second, quieter reason this matters: frontier guardrails prohibit using frontier outputs in online RL training, but open-weight juries are unrestricted - so the training data pipeline no longer needs a frontier judge at all.

The jury result rhymes with the earlier stack we covered on the tenth: the entire gain of an independent verification signal concentrates on one-vote-margin decisions, +10.4 to +23.3 points there and zero elsewhere, so expensive checks should be routed to pivotal votes rather than spread evenly (arXiv:2608.06940); anytime-valid stopping cuts the games an eval needs by a median 74x with the confidence guarantee intact (arXiv:2608.06362); and a judge that scores failures as passes at rate (1-tau)/2 or above retires nothing at any sample size, so the two judge-error directions must be measured separately before a gate goes live (arXiv:2605.22148).

One of those four legs already collapsed, by the way, and it is the honest thing to say so: the evidence-lock result we cited on the tenth - persist evidence at decision time - was withdrawn by its author on the eleventh, a code error affecting the reported results (arXiv:2608.05353, v2 withdrawn). We are not building this spec on withdrawn legs. We are also not pretending the withdrawal is embarrassing: the audit wave auditing its own results is the thesis working. It is a reminder that this entire stack is a moving target and you should read it as a direction, not a spec sheet.

The industry datapoint#

The same week, DeepSeek open-sourced its harness (deepseek-harness, MIT): an event-sourced session log with a runtime invariant that every model request byte-matches the log projection - replay equals reality, structurally - plus keyless replay testing where committed transcripts are both mock input and expected output, fail-closed sandbox chains, and 1,372 machine-verified agent notes. This is the strongest open reference we have seen for audit-grade agent infrastructure. And its BENCHMARK.md is a three-line stub. Zero eval claims from an AI lab.

We think that absence is the signal. The infrastructure half of verification is commoditizing - replay, sandboxing, byte-exact logs, all now open source and MIT. The judgment half is still an unsorted pile of single-judge numbers. The judge is the last un-engineered piece of the stack, and the last two days of measurements are the engineering drawings.

The counter-case, with the steel it deserves#

Four objections deserve real weight.

First, this is a two-day wave from a handful of labs, and we are the people who keep telling you that wave-shaped evidence is the least reliable kind. The wiggle numbers are apples-to-apples across models, the persuasion numbers are concrete and transferable, the jury beats the best single judge available - but none of it is an independent reproduction, and none of it measures a production loop end-to-end. The 62-91 percent flip rate is a persuader talking to a judge in a lab, not an attacker in your eval pipeline.

Second, the jury result is jury-versus-best-single-judge, not jury-versus-ground-truth. Deliberation may share the single-judge blind spots, or introduce its own - a persuasive juror can sway a jury the way a persuader sways a judge. The measured improvement is against the strongest available comparison, which is exactly the strongest comparison the field has, and it is still not ground truth.

Third, the fixes commoditize, and commoditization is a double-edged sword for our own thesis here. If open-weight juries at 8-15 percent of the cost standardize within a year - and we expect them to - then the judge layer becomes a commodity input, and the scarce, differentiated skill moves to spec-elicitation and check-design: what you choose to verify, not how you judge it. That is the verification-economy outcome, and it makes the "judge system" a transition rather than the destination.

Fourth, the model-side escape hatch exists: persuasion robustness and compositional constraint following are trainable, and rubric dropout shows the training-side fixes are cheap. If frontier labs bake persuasion-resistance and budget-conditioned behavior into the next training runs, the practice questions we are asking today may dissolve into model defaults by next year. We would welcome that outcome. We grade ourselves on calls, not on being right for the dramatic reason.

What we believe now#

Here is the claim, stated so it can be graded. By end of 2027, in any serious agent shop, a single-judge, single-budget, single-phrasing evaluation will be treated the way single-point backups are: recognized as malpractice, tolerated only in legacy systems. Judge pipelines will ship measured properties instead of a model name: wiggle scores across the three stability axes, per-budget ranking curves, multi-phrasing baselines, jury-majority confidence. RL post-training loops will ship gold-judge divergence curves, because the training judge hacking itself is now a named, measured, preventable failure. And judge persuasion-resistance will be tested before deployment, not after the incident.

What would prove us wrong: next-generation models absorbing the noise - persuasion-robust, budget-insensitive, phrasing-insensitive judges arriving as training defaults, so the practice question never becomes a market. We will report either outcome with the same care. The withdrawal of evidence-lock this week is us reporting on the fallibility of our own evidence as it happens; the record stays honest.

What developers should do#

1. Never let the judged party argue its own score. If your agent gets a verdict and a chance to respond to it, you have built a persuasion surface. The wiggle numbers say you have also built a corruptible one.

2. Run juries, not judges, for anything that gates. Deliberation with moderated re-votes beats a bigger single judge, and open-weight juries cost 8-15 percent of frontier judging. For online RL training, they also dodge the frontier guardrails.

3. State the budget with every comparison, publish the per-budget curve. A ranking that reverses across token budgets is not a ranking. If you cannot condition on budget, your leaderboard delta is a measurement artifact wearing a claim.

4. Ship the gold-judge divergence curve in your training loop. If training score and held-out gold judge diverge, stop and fix the rubric - random criterion dropout at 30-50 percent is the one-line intervention.

5. Ask for the wiggle score. Next time a vendor says their judge pipeline is validated, ask: validated against golden data, or stress-tested under re-prompting, challenge, and a trainable persuader? The first answers accuracy on a test set. The second answers what happens when your agent argues back.

Continue Reading#

• Your Benchmark Is Lying to You - the original audit-wave position this piece extends: benchmark numbers carry double-digit noise, and you buy agents on them
• The Benchmark Fix Is Architectural - the first fix wave: ledgers, counterfactuals, readout discipline, the deterministic bottom
• The Plateau Was the Instrument - the most expensive eval noise is the flat curve, because it reads as science
• The Judge Leaves the Loop - why content-level LLM judging fails inside loops, and what replaces it
• Kill Your Agent Runs Early - the lifecycle discipline for agent compute, of which judge design is now a load-bearing part

Sources#

• Jagged Judges: Wiggle Framework for judge epistemic stability: arXiv:2608.12645 (2026-08-12)
• Learning to Persuade: adversarial RL persuaders and belief collapse: arXiv:2608.11624 (2026-08-12)
• Reasoning Jury: moderated open-weight jury vs frontier single judges: arXiv:2608.12585 (2026-08-12)
• Rubric Dropout: reward hacking in rubric-as-reward RL: arXiv:2608.11669 (2026-08-12)
• Budget-Dependent Rankings: ranking reversals across token budgets: arXiv:2608.12150 (2026-08-12)
• The Wording Effect: two-way drift under rephrasing: arXiv:2608.11694 (2026-08-12)
• Blind to the Pivotal Vote: verification gain concentrates at one-vote margins: arXiv:2608.06940 (2026-08-07)
• AV-AIVAT: 74x cheaper agent evaluation with anytime-valid stopping: arXiv:2608.06362 (2026-08-06)
• Ratchet: the exact judge-reliability bound for skill retirement: arXiv:2605.22148 (2026-05-21, v3 2026-08-07)
• Evidence Lock Before Commitment: withdrawn by the author (v2, 2026-08-11): arXiv:2608.05353
• DeepSeek Harness (MIT, developer preview): github.com/deepseek-ai/deepseek-harness (2026-08-13)