SWE-NFI: The Benchmark That Catches What Coding Agents Miss

Last updated: July 31, 2026

SWE-bench-style benchmarks proved coding agents can fix bugs. A new paper from the software engineering research community asks a question those benchmarks never do: can agents make code better without changing what it does?

The answer, from SWE-NFI (arXiv 2607.27409, submitted July 29), is a careful "not yet". The best agent in the study hits 70.0% functional correctness, yet every agent evaluated falls short of a human reference on non-functional improvements (NFIs). The gap is widest exactly where technical debt lives: structural code improvements.

What the benchmark is#

SWE-NFI is a benchmark for evaluating coding agents on behavior-preserving improvements - refactors, readability, maintainability, and performance work that changes no observable behavior. The authors built it from 188 tasks extracted from real merged pull requests in open-source Python projects, so the targets are improvements humans actually shipped, not synthetic ones.

The benchmark operationalizes developer-oriented NFIs into 92 executable rules, and the evaluation combines two gates: functional correctness tests (did the agent break anything) plus rule-based NFI scoring (did the code actually get better). That two-sided design matters. A refactor that passes tests but does not improve the code scores nothing, and a change that improves style but breaks a test fails outright.

What it found#

The headline numbers:

• Best agent functional correctness: 70.0%, comparable to what agents score on correctness-only benchmarks
• Overall NFI capability: every agent falls short of the human reference
• Structural code improvements: agents score 0.0 to 1.3, versus 1.5 for the human reference

The structural gap is the interesting one. Structural improvements are the changes that reshape code organization: extracting functions, removing duplication, simplifying control flow. They are also the hardest to verify mechanically, which is why most agent harnesses do not try. The paper's rule-based approach is an attempt to make that verification executable, and the wide 0.0 to 1.3 range across agents suggests this is not a uniform weakness - some models have genuinely better machinery for it than others.

The evaluation targets are behavior-preserving, which makes failures informative. When an agent cannot refactor without breaking behavior, the failure mode is not "agent can't write code". It is "agent can't prove the change is safe", which is a different and more tractable problem.

Why it matters#

Correctness benchmarks have saturated as a proxy. Agents score well on bug-fix benchmarks and teams have adopted them accordingly - but production code spends most of its life being improved, not fixed. We covered the same signal from a different angle in Agent Evals Need Baseline Receipts: an eval that measures one outcome teaches you nothing about the others. SWE-NFI is the complementary move - measure the outcome that reviews and maintenance actually care about.

Refactoring is where the agent value proposition gets murky. An agent that lands a bug fix saves a developer an hour of debugging. An agent that lands a half-done refactor creates a follow-up task for someone else. The 0.0 to 1.3 structural scores suggest that, at the margin, teams should be more conservative about delegating restructuring work to agents than bug fixes - which is the opposite of the natural instinct to hand over the "boring" cleanup tasks first. This connects to the review-queue problem in AI Code Review Is the New Bottleneck: the risk is not agent volume, it is agent output that needs human repair.

Rule-based verification is the real contribution. The 92 executable rules are a template for what a refactor-checking harness looks like: assert behavior preservation with tests, then score improvement against explicit rules. That is a pattern any team can borrow for its own agent workflows, and it pairs naturally with the case for why skills beat prompts for coding agents - a skill that cannot verify its own output is a skill you cannot trust on structural work.

My take#

Three things to take from this paper.

First, treat it as a division of labor argument, not a verdict on agents. The evidence says: agents for functional work, humans for structural work, with rule-checked automation in between. Repository context engineering keeps pushing on how to give agents more of the project around a task; SWE-NFI is the reminder that context alone does not close the structural gap.

Second, the 70.0% correctness number means half-decent refactors are landing everywhere, invisibly, inside bigger agent diffs. If you use coding agents on production repos, some of their output is already structural change that no benchmark would have caught. This is an argument for keeping agent PRs small and reviewable - the exact case GitHub made with stacked PRs - so structural drift is visible per layer instead of buried in a 2,000-line diff.

Third, expect this benchmark shape to become the norm. Correctness-only evaluations are cheap and have driven a year of improvements, but every agent vendor now needs an answer to "does your model make code better", because that is what developers ask of code review today. The paper's executable-rules approach is a credible answer format, and it is reproducible by construction.

The honest summary: agents are still better at fixing your bugs than improving your code, and the gap is measurable. If your team's agent adoption feels good on issue fixes and awkward on refactors, that is not your setup - it is the current state of the art, now quantified.

FAQ#

What is SWE-NFI?#

SWE-NFI (arXiv 2607.27409) is a benchmark for evaluating coding agents on non-functional improvements: behavior-preserving refactors, readability, maintainability and performance work. It has 188 tasks extracted from real merged pull requests in open-source Python projects and scores them with 92 executable rules plus functional correctness tests.

How do coding agents score on SWE-NFI?#

The best agent reaches 70.0% functional correctness, but every agent evaluated falls short of the human reference on non-functional improvements. On structural code improvements agents score 0.0 to 1.3, versus 1.5 for the human reference.

Why are structural refactors hard for coding agents?#

Structural improvements such as extracting functions, removing duplication and simplifying control flow are the hardest changes to verify mechanically, so most agent harnesses do not try. The post frames the failure as the agent being unable to prove a change is safe, which is a different and more tractable problem than writing code.

Should teams delegate refactoring to coding agents?#

The post argues for caution: use agents for functional work such as bug fixes, keep humans on structural work with rule-checked automation in between, and keep agent PRs small and reviewable so structural drift stays visible.

Continue Reading#

• Agent Evals Need Baseline Receipts
• The ICML 2026 Agent Reproduction Audit: 23% of Examined Papers Had Falsified or Contested Claims
• Codenib: Repository Context for Coding Agents
• DeepSeek V4 Economics: Cost, Quality, and Frontier Agentic Coding
• What Hacker News Gets Right About AI Coding Agents
• AI Code Review Is the New Bottleneck
• OwlPath: Ontology-Based Code Retrieval Cuts Agent Tokens 29%

Sources#

• SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements - arXiv