Benchmarking Opus 5 on SlopCodeBench: AI Code Quality Under Iteration

Most coding benchmarks give the model the full spec up front. Solve a task, get a score, done. Real software engineering does not work that way - requirements emerge, codebases grow, and quality degrades across hundreds of incremental changes.

SlopCodeBench, a March 2026 benchmark from Gabe Orlanski's lab at UW Madison, was built to measure exactly this. Each challenge reveals requirements across multiple checkpoints. The model never knows what comes next. It has to evolve a codebase over time without breaking what it already built.

Dexter from humanlayer ran three Claude models - Opus 5, Opus 4.8, and Sonnet 5 - through a 17-checkpoint subset of SlopCodeBench and watched the results unfold over six hours. The findings are sobering for anyone hoping to run AI coding agents lights-off.

What SlopCodeBench Measures Differently#

The benchmark's key insight is that software maintenance is a longitudinal problem. Most benchmarks - SWE-bench, Frontier Code, DeepSWE - hand the model a complete task description and score the result. SlopCodeBench gives checkpoints one at a time. A model that passes checkpoint 1 with clean code might find itself in an unmaintainable mess by checkpoint 5, and the strict pass criteria means every regression test from every prior checkpoint must still pass.

The original paper showed that the best models available at publication - GPT-5.4 and Opus 4.6 - scored 11% and 17% strict pass rates. The benchmark was unsaturated, meaning there was genuine headroom for improvement.

The Results: Opus 5 Wins, But Not By Much#

Across three problems (circuit_eval, database_migration, dynamic_config_service_api) totaling 17 checkpoints, Opus 5 achieved a 24% strict pass rate (4 of 17). Opus 4.8 and Sonnet 5 each managed 6% (1 of 17).

Three of Opus 5's four strict passes came from the opening checkpoints of a single problem. No model reached the final checkpoint of any challenge with all tests passing - even on the problem labeled "easy."

The headline number is a 41% relative improvement over Opus 4.6's 17% in the original paper, but the absolute pass rate remains low enough that you would not trust any of these models to iterate on a production codebase without supervision.

Code Quality Degradation: The Slop Meter#

The benchmark tracks 41 code quality metrics across checkpoints: cyclomatic complexity, duplication, function count, lint errors, dependency propagation cost, and more. The trend across every model is upward.

Opus 5 wrote five times more functions than Opus 4.8 over the same challenges. Some of that was test code - Opus 5's output was 51% tests versus 11% for Opus 4.8 - but the sheer volume of code ballooned to 29,065 source lines against roughly 9,000 for each of the other two models.

Cyclomatic complexity rose across checkpoints for every model. Opus 4.8's mean complexity climbed 70% across eight checkpoints, with its worst single function hitting a cyclomatic complexity of 93. Opus 5 stayed flatter in complexity but achieved that by writing vastly more, smaller functions.

Code duplication told a sharper story. Opus 4.8 went from 4.6% duplicated lines to 16.8% across the circuit_eval challenge, with an inflection at checkpoint 3 - roughly where new requirements started fighting the initial design. Opus 5 stayed essentially flat at 2.4-2.6%.

Nearly all code written triggered the benchmark's slop detectors: Opus 4.8 at 98%, Opus 5 at 93%, Sonnet 5 at 89%. Lines flagged as verbose rose from roughly 65% at checkpoint 1 to 80% by checkpoint 8 for every model.

What HN Is Saying#

The Hacker News discussion split between appreciation for the benchmark design and debate about what the results actually mean.

Several commenters confirmed the finding from their own experience. One noted that Opus 5 is "a nice improvement over Opus 4.8, but not being revolutionary like Fable felt," and reported swapping Opus 4.8 xhigh with Opus 5 medium for a faster, cheaper experience. Another pushed back on the framing: a 41% improvement from 17% to 24% "is not much higher?" is arguably doomer framing.

The harness vs model debate surfaced repeatedly. Multiple commenters argued that "slop accumulates when the agent can touch anything" and that constraining the agent to one seam with add-only edits matters more than model choice. Another suggested an adversarial review pass - have the model scan its own output for complexity in a second pass before returning - as a practical mitigation.

Several people asked for Fable 5 and Sol results. The author noted those are coming in a follow-up. One commenter raised the idea of a hand-off benchmark where a frontier model builds checkpoints 1-7 and a smaller model like Sonnet attempts checkpoint 8, creating a measurable signal for codebase maintainability.

A few voiced skepticism about benchmark proliferation. "So many benchmarks more the models themselves," one wrote. "Just make one unified standard to benchmark all or stop calling it 'benchmarking.'"

Why This Matters#

SlopCodeBench fills a real gap. SWE-bench-style problems measure "can a model solve a task in one shot." They do not measure "can a model sustain codebase health across a week of PRs." The Frontier Code benchmark and SWE-Marathon push on scope but still hand the model a complete spec up front.

The practical implication is that running an agent on "lights-off" mode - where it iterates without human review - is still risky. Code quality degrades, complexity accumulates, and the model cannot see its own trajectory. The author's framing captures it well: "every dollar bought correctness. nobody bought enough of it."

The hand-off variant proposed in the post is especially promising. If you have a frontier model build the initial codebase and a smaller, cheaper model try to extend it, the smaller model's failure rate becomes a signal for how maintainable the frontier model's code actually is. That is testable today.

For developers using Claude Code, Codex, or Cursor, the takeaway is straightforward: review agent-generated code for structural health, not just correctness. Set up complexity guards in CI. Run periodic refactor passes. Do not assume that because a model solved the current ticket cleanly, the codebase is getting better.

FAQ#

What is SlopCodeBench?#

SlopCodeBench is a long-horizon coding benchmark from UW Madison that tests a model's ability to evolve a codebase across multiple checkpoints, where each checkpoint reveals new requirements the model did not see in advance.

How did Opus 5 perform on SlopCodeBench?#

Opus 5 scored a 24% strict pass rate (4 of 17 checkpoints), compared to Opus 4.6's 17% in the original paper and Opus 4.8 and Sonnet 5's 6% in this test run.

Is Opus 5 better than Opus 4.8 for coding?#

On SlopCodeBench, Opus 5 is clearly better - 24% vs 6% strict pass rate - but the improvement comes with 5x more functions and 3x more source lines written. The model is more capable but also more verbose.

Can I run AI coding agents lights-off based on these results?#

The results suggest not. No model reached the final checkpoint of any challenge with all tests passing. Code quality metrics degraded across every model over the course of each challenge.

What is the hand-off test variant?#

The hand-off test has a frontier model (Opus 5, Fable) build checkpoints 1-7, then hands the codebase to a smaller model (Sonnet) for checkpoint 8. If the smaller model fails, it signals that the frontier model's code is not maintainable.

Sources#

• Benchmarking Opus 5 on SlopCodeBench (humanlayer / Dexter)
• HN Discussion (490 points)
• SlopCodeBench Paper (arXiv, March 2026)
• SlopCodeBench Runner (SprocketLab)
• SlopCodeBench Problems (gabeorlanski/scb-problems)

Continue Reading#

• Claude Opus 5 vs Opus 4.8 vs Fable 5: Full Benchmark Comparison - Our full 7-eval shootout across the Anthropic model lineup
• FrontierCode Benchmark Explained: Why AI Coding Scores Are Wrong - Why mergeability beats pass rate as a signal
• Clean Code Makes AI Agents 34% More Efficient - Empirical data on how codebase quality affects token usage
• Why Software Factories Fail: Harness Engineering Is Not Enough - How models systematically degrade codebase quality over time
• Write Code Like a Human Will Maintain It - The HN debate on AI-generated slop and maintainability