Blind Resampling Beats Self-Repair in Small Code Models: Retry Without the Failed Code

Self-repair is the default loop in nearly every coding agent: the model runs the tests, fails, gets the failing program and the test output fed back in, and is asked to fix it. A new preprint argues the field has been measuring the wrong baseline all along - and that for small models, the failed code should never go back in at all.

The paper, "Try Again, Don't Look Back: Blind Resampling Outperforms Self-Repair in Small Code Models" (arXiv:2607.26117, submitted July 28), makes the case with a placebo-controlled design on MBPP+ at three model scales (1.5B, 3B, 7B). The headline: blind resampling - retrying the same prompt with no information about the failure - is the strongest retry condition below 7B and stays statistically tied at 7B, while consuming 2.5-5.5x fewer tokens.

What the study did#

The critique of prior work is sharp: self-repair is almost always evaluated against a baseline that does not retry at all. That comparison confounds two things - the value of the extra attempt and the value of the execution feedback. To separate them, the author compares four retry conditions at matched budgets:

• Blind resampling: sample again from the same prompt, no failure information.
• Content-free failure notice: a placebo message saying the attempt failed, with no detail.
• Genuine execution feedback: the failed program plus the test output.
• Feedback plus verbal self-reflection: execution feedback, then a reflection pass.

At 1.5B, conditioning the retry on the model's own failed attempt costs 6.1 points of accuracy (p=0.006) versus blind resampling. The informational content of execution feedback adds nothing measurable over the placebo. The proposed mechanism is anchoring: when shown its previous attempt, the model reproduces a near-identical program in 33-68% of retries, versus 2-14% under blind resampling. The model is not fixing its bug; it is re-sampling from the same failure region.

Two control experiments delimit the effect. Retrieving solutions to other tasks into the context changes nothing (bounded to +/-3.5 points), which localizes the harm to self-conditioning rather than context length. And reflection, the only condition that measurably weakens the anchor, remains dominated on cost: it spends the most tokens and still does not beat resampling on accuracy.

Why the anchoring result matters#

The mechanism is the finding. If repair loops only work by re-rolling the dice, then most agent harnesses are spending 2.5-5.5x the tokens to do what a plain retry does better. For anyone running agents on small local models - the 1.5B-7B tier that powers cheap local coding loops - this is a directly actionable number: when the model fails, drop the code, re-prompt, and move on.

The result also explains a pattern that shows up in our earlier coverage of agent loops. The reliability cliff most 10-step agent chains hit is not about the model's reasoning, it is about how each step conditions on the previous one's output. A repair loop is just a two-step chain where the second step is anchored to the first. Our loop engineering post on designing agent loops that converge walks through exactly this failure mode from the harness side, and the paper supplies the mechanism for why it happens.

The replication work adds weight. The penalty is unchanged at full precision, and it reproduces on an independent model family - six configurations across two families and two precisions. The author also reports the magnitude of the anchoring cost is predicted by baseline quality alone (r=0.96), which is the paper's most useful sentence: the cost of anchoring is the cost of committing to a bad first attempt. Stronger models anchor less, which is consistent with the gap narrowing at 7B and raises a real question about whether frontier models with long-context memory are affected the same way.

What it means for agent harness design#

For builders, the practical guidance is a decision table, not a dogma:

• Small models (<=7B), single-attempt tasks: blind resampling. Do not feed the failed code back. If your harness has a self-repair loop, measure it against a no-feedback retry baseline before keeping it.
• Frontier models: keep the repair loop, but treat it as a second attempt, not a guaranteed fix. The paper's placebo comparison is exactly the kind of baseline-receipt discipline our evals coverage has pushed for - without a no-retry control, a repair loop's apparent gains are just the value of trying twice.
• Token budgets: because resampling spends 2.5-5.5x fewer tokens per retry, switching to blind retries is a cost win even where accuracy ties. This pairs with the cache-discipline argument from our Reasonix coverage: the cheap retry is the right retry.
• The reflection caveat: reflection weakens the anchor but is dominated on cost in this study. If you run reflection loops anyway (for example, agent routines that plan before acting), this is a reminder to audit whether the reflection pass is changing the code or just the tokens spent.

The study is one author, one benchmark (MBPP+, Python), and small-model scales - it is a preprint, not a law. The honest takeaway is not "self-repair is dead." It is that repair loops have been running without a proper control group, and the first placebo-controlled measurement says the feedback itself may be worth much less than the retry it rides on. Harness authors should re-baseline their repair loops this week.

Continue Reading#

• Loop Engineering: How to Design Agent Loops That Actually Converge - harness-side patterns for the retry loops this paper measures
• The Agent Reliability Cliff: Why Your 10-Step Chain Only Succeeds 20% of the Time - why conditioning on prior steps degrades chains, in production terms
• Codex Loops: What Boris Cherny Gets Right About Managing Agent Work - how frontier-loop designers think about retries and budgets
• Cheap subagents are better when their work is visible - token economics of agent fleets, where retry cost compounds
• SWE-NFI: The Benchmark That Catches What Coding Agents Miss - another fresh look at what coding agents get wrong and how we measure it

Sources#

• Paper (abstract, primary): Try Again, Don't Look Back: Blind Resampling Outperforms Self-Repair in Small Code Models
• Code, pre-registrations and run traces: github.com/vermayuvraj/self-improving-agent
• MBPP+ benchmark: github.com/evalplus/evalplus