SWE-NFI: The Benchmark That Catches What Coding Agents Miss

SWE-bench-style benchmarks proved coding agents can fix bugs. A new paper from the software engineering research community asks a question those benchmarks never do: can agents make code better without changing what it does?

The answer, from SWE-NFI (arXiv 2607.27409, submitted July 29), is a careful "not yet". The best agent in the study hits 70.0% functional correctness, yet every agent evaluated falls short of a human reference on non-functional improvements (NFIs). The gap is widest exactly where technical debt lives: structural code improvements.

What the benchmark is#

SWE-NFI is a benchmark for evaluating coding agents on behavior-preserving improvements - refactors, readability, maintainability, and performance work that changes no observable behavior. The authors built it from 188 tasks extracted from real merged pull requests in open-source Python projects, so the targets are improvements humans actually shipped, not synthetic ones.

The benchmark operationalizes developer-oriented NFIs into 92 executable rules, and the evaluation combines two gates: functional correctness tests (did the agent break anything) plus rule-based NFI scoring (did the code actually get better). That two-sided design matters. A refactor that passes tests but does not improve the code scores nothing, and a change that improves style but breaks a test fails outright.

What it found#

The headline numbers:

• Best agent functional correctness: 70.0%, comparable to what agents score on correctness-only benchmarks
• Overall NFI capability: every agent falls short of the human reference
• Structural code improvements: agents score 0.0 to 1.3, versus 1.5 for the human reference

The structural gap is the interesting one. Structural improvements are the changes that reshape code organization: extracting functions, removing duplication, simplifying control flow. They are also the hardest to verify mechanically, which is why most agent harnesses do not try. The paper's rule-based approach is an attempt to make that verification executable, and the wide 0.0 to 1.3 range across agents suggests this is not a uniform weakness - some models have genuinely better machinery for it than others.

The evaluation targets are behavior-preserving, which makes failures informative. When an agent cannot refactor without breaking behavior, the failure mode is not "agent can't write code". It is "agent can't prove the change is safe", which is a different and more tractable problem.

Why it matters#

Correctness benchmarks have saturated as a proxy. Agents score well on bug-fix benchmarks and teams have adopted them accordingly - but production code spends most of its life being improved, not fixed. We covered the same signal from a different angle in Agent Evals Need Baseline Receipts: an eval that measures one outcome teaches you nothing about the others. SWE-NFI is the complementary move - measure the outcome that reviews and maintenance actually care about.

Refactoring is where the agent value proposition gets murky. An agent that lands a bug fix saves a developer an hour of debugging. An agent that lands a half-done refactor creates a follow-up task for someone else. The 0.0 to 1.3 structural scores suggest that, at the margin, teams should be more conservative about delegating restructuring work to agents than bug fixes - which is the opposite of the natural instinct to hand over the "boring" cleanup tasks first. This connects to the review-queue problem in AI Code Review Is the New Bottleneck: the risk is not agent volume, it is agent output that needs human repair.

Rule-based verification is the real contribution. The 92 executable rules are a template for what a refactor-checking harness looks like: assert behavior preservation with tests, then score improvement against explicit rules. That is a pattern any team can borrow for its own agent workflows, and it pairs naturally with the case for why skills beat prompts for coding agents - a skill that cannot verify its own output is a skill you cannot trust on structural work.

My take#

Three things to take from this paper.

First, treat it as a division of labor argument, not a verdict on agents. The evidence says: agents for functional work, humans for structural work, with rule-checked automation in between. Repository context engineering keeps pushing on how to give agents more of the project around a task; SWE-NFI is the reminder that context alone does not close the structural gap.

Second, the 70.0% correctness number means half-decent refactors are landing everywhere, invisibly, inside bigger agent diffs. If you use coding agents on production repos, some of their output is already structural change that no benchmark would have caught. This is an argument for keeping agent PRs small and reviewable - the exact case GitHub made with stacked PRs - so structural drift is visible per layer instead of buried in a 2,000-line diff.

Third, expect this benchmark shape to become the norm. Correctness-only evaluations are cheap and have driven a year of improvements, but every agent vendor now needs an answer to "does your model make code better", because that is what developers ask of code review today. The paper's executable-rules approach is a credible answer format, and it is reproducible by construction.

The honest summary: agents are still better at fixing your bugs than improving your code, and the gap is measurable. If your team's agent adoption feels good on issue fixes and awkward on refactors, that is not your setup - it is the current state of the art, now quantified.

Continue Reading#

• Agent Evals Need Baseline Receipts
• Codenib: Repository Context for Coding Agents
• DeepSeek V4 Economics: Cost, Quality, and Frontier Agentic Coding
• What Hacker News Gets Right About AI Coding Agents
• AI Code Review Is the New Bottleneck

Sources#

• SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements - arXiv