xhigh vs Medium Effort: SlopCodeBench Oct 2026 Results

Last updated: October 21, 2026

If you have been defaulting your coding agent to xhigh reasoning effort on the assumption that more thinking means better code, the newest SlopCodeBench write-up is a reason to test that assumption. Dex Horthy of HumanLayer ran ten model and effort settings through 36 problems and 196 checkpoints, and xhigh cost between 2.11x and 4.22x as much as medium without reliably passing more of them. Opus 5.5 did worse at xhigh than at medium.

What the benchmark measures#

SlopCodeBench hands a coding agent a problem in stages. It writes a program, gets new requirements, and has to change its own code without breaking earlier behavior. We covered the earlier three-model run in Benchmarking Opus 5 on SlopCodeBench. The October run is bigger: 36 problems and 196 checkpoints per setting, using the pinned v1.0 catalog, the stock just-solve prompt and seed 42.

The headline metric is the strict pass rate: a checkpoint counts only if every test category passes, including regression tests. The looser core pass rate only needs the core category to pass. The author's own caution matters here: each model and effort setting was run once, and each checkpoint inherits the code from the one before it.

The numbers#

All figures below are from the write-up. Costs are the author's estimates.

Two readings stand out. First, the strict spread across all ten campaigns is only 12.24% to 16.33%, and Astra xhigh leads GPT-6.1 SOL medium by two checkpoints. Second, the cost spread is $178.51 to $1,550.07 for that same narrow band. The author notes that Fable 5.1's recorded estimate is about 8.68x GPT-6.1 Medium's with three fewer strict passes.

The effort comparison#

Three models have complete medium and xhigh pairs:

The author's own conclusion is careful: this does not prove more reasoning hurts coding, only that higher effort "was not a reliable way to buy more strict passes in these runs". With one campaign per setting, run-to-run noise could explain a three-checkpoint swing.

The angle: who wins and who loses#

Winners: teams that pay per token. A 2x to 4x cost multiplier with no dependable lift is the easiest budget line to cut, and the cheapest settings in the table (GPT-6.1 SOL medium at $178.51, Opus 5.5 medium at $193.47) sit within a few checkpoints of the leader.

Losers: the "always max" habit, and any pricing story built on selling the highest tier as the safe default. If your tool defaults to xhigh, your bill is carrying a bet the data does not yet back up.

The twist is quality, not passes. The write-up measures how much of the leftover code trips a slop rule, across 28 matched problems. Opus 5.5 xhigh scored 77.75% and Opus 5.5 medium 94.25%, so on that metric xhigh wrote code the author describes as close to 5x cleaner, while passing fewer strict checkpoints. The author flags that the scanner's linters are hand-curated patterns from earlier agent-authored PRs, so newer models may produce slop it does not know about. Medium also left the most code, 216,881 lines against 175,616 after filtering scratch folders. More effort can mean "solve harder problems" or "produce more stuff", and the benchmark does not separate them.

What a builder does Monday#

Do not copy the table into your config. Run the same experiment on your own work:

1. Pick five to ten representative tasks from your backlog, ones with tests so a pass is checkable.
2. Run each at your current effort and at medium, with the same prompt and branch state.
3. Compare tests passing, tokens spent and how much code you would have to review. The last one is the cost the benchmark's author flags most.
4. Keep the cheaper setting wherever results are tied, and reserve high effort for the task types where it actually won.

For how the effort settings differ in practice, see Fable 5 effort levels explained and effort versus switching models. The Ultracode effort level post covers what the top tier changes.

What people are actually saying#

The Hacker News thread was well past 200 points when read. Paraphrasing the comments:

• Medium is enough. Several commenters say medium gives them the best output for most coding, with one reserving xhigh for spatial reasoning or creative writing. Another argues effort can mean either "solve harder problems" or "produce more stuff", and they mostly do not want more stuff.
• The counter-case. One commenter says they leave GPT-6.1 on max and never touch it, while another reports max and ultra are slow and overengineer. The two camps disagree on the evidence, which fits the benchmark's own single-run caveat.
• Statistical skepticism. A commenter objects that reporting percentages to two decimals from 36 problems overstates precision, and others ask for repeated runs with confidence intervals.
• Reproduction. Commenters point to the author's note on how to repeat the runs, and to the scb-problems repository for a subset you can run yourself. The write-up itself notes that an earlier six-problem subset produced a 30% Fable score that is not comparable to the 36-problem results, so small subsets can mislead.
• Style complaints. Some readers disliked the write-up's tone. That says nothing about the numbers, but it is in the thread.

What to watch#

Part II from the SlopCodeBench team, which the author says will cover the qualitative slop patterns and the scanner's blind spots on newer models. Repeated runs per setting would tell us whether the Opus 5.5 reversal is signal or noise. Until then, the safe reading is narrow: in these single runs, paying for xhigh did not reliably buy more strict passes.

Continue Reading#

• Benchmarking Opus 5 on SlopCodeBench - the earlier three-model run
• Fable 5 effort levels explained - what each setting does
• Fable 5 effort vs model switching - which lever to pull first
• Ultracode effort level explained - the top tier
• Claude Opus 5.5 release guide - the model that did better at medium

Sources#

• SlopCodeBench update 10/2026 - Dex Horthy, October 15, 2026; all numbers author-reported, read October 21
• SlopCodeBench paper - benchmark definition
• scb-problems repository - problem set linked from the thread
• Hacker News: SlopCodeBench 10/2026 thread - community discussion