Agent Retrieval Bench Finds The Files Before The Fix

Most coding-agent benchmarks grade the ending: did the patch pass, did the issue close, did the model reach the final answer?

That hides an earlier failure mode. Before an agent can edit the right code, it has to find the right files.

Last updated: September 21, 2026

Agent Retrieval Bench is useful because it isolates that upstream context-acquisition step. The paper asks a narrower question than SWE-bench or Terminal-Bench: given a real workflow signal and a frozen repository, can a retriever rank the files a coding agent needs to read next?

That makes it a natural follow-up to CodeNib treating repository context as a data system. CodeNib says repo context should be compiled, maintained, and served. Agent Retrieval Bench asks how you tell whether that context layer found the right evidence in the first place.

The Signal Check#

Hugging Face's July 2026 papers page included Agent Retrieval Bench in the same developer-relevant cluster as CodeNib, Dockerless, Resource2Skill, Long-Horizon-Terminal-Bench, and other agent-infrastructure papers. The official paper page links the arXiv, project site, Hugging Face dataset, and evaluation repository, which gives this topic enough primary-source depth to work with.

Google Trends was mandatory for this automation run. The check was attempted on September 21, 2026 for the query cluster coding agents, AI coding agents, repo context, repository context, and context engineering across a US 12-month window. Google returned HTTP 429, so no reliable Trends rows were available and no numeric Trends values are used in this article. The topic is selected on developer relevance, primary-source quality, Hugging Face paper visibility, existing DevDigest search-intent adjacency, and duplicate-risk review.

Duplicate risk was checked locally across the existing blog corpus. We already have a CodeNib article, plus adjacent posts on context reduction, codebase knowledge graphs, and long-horizon terminal evals. This post should not duplicate those. It should answer a sharper question: how do you evaluate the retrieval layer before the edit layer gets blamed?

What Agent Retrieval Bench Measures#

Agent Retrieval Bench, or ARB, is a file-level benchmark for repository context retrieval. Each sample gives a system a workflow-derived query and a repository frozen at a base commit. The retriever ranks candidate files, or abstains when the repo should not contain useful local context.

The key phrase is "agentic relevance." A file is relevant when reading it would materially help the agent take the next correct step, even if the query does not look semantically similar to that file.

That matters because coding-agent context is often indirect:

• A PR implementation signal may require the related tests, not only the implementation file.
• A code review comment may point at one file while the real constraint lives in another module.
• A failure trace may mention a test or stack frame while the root cause sits in a different source file.
• An anchored edit may ripple into files the initial query never names.

Traditional code search can miss those cases because it rewards text similarity. Coding agents need the file that helps the next action.

The Four Positive Tasks#

The benchmark covers four positive retrieval tasks.

code2test asks for tests related to an implementation or PR signal. This is the thing agents need before they confidently change behavior.

comment2context starts from a code review comment and the reviewed file, then asks for additional files needed to understand or satisfy the comment. The reviewed file is given context, not the target.

trace2code starts from reproduced failure output and asks for root-cause source files. Visible tests and stack frames are evidence, but they are not automatically counted as the main answer.

edit2ripple gives an anchored change and asks for additional source or test files affected by that change. This is close to the review problem every multi-file agent run creates.

There is also a selective retrieval subset with natural no-gold cases and counterfactual wrong-repository controls. That subset matters because a retrieval system that always returns something can look helpful while injecting irrelevant context.

The Numbers Worth Remembering#

ARB contains 427 samples across 25 repositories. The positive set has 345 examples across the four retrieval tasks. The selective set has 50 natural no-gold examples and 32 counterfactual controls. The combined corpus manifest covers 308 base-commit snapshots, 392,000 files, and 7.9 million chunks.

The paper evaluates lexical retrieval, RepoMap, open-source embeddings, selective abstention, and logged agent context selection. No single retrieval family dominates:

The last point is the one to underline. Agents can compensate for weak initial context by exploring, but exploration costs tool calls, latency, tokens, and human patience. In the paper's logged trajectory track, agents still never touch any gold file on a large minority of samples.

That lines up with the practical lesson from diffs versus whole-file editing: agent systems need routing policies before they generate output. First choose the right evidence. Then choose the right edit mode.

The Opposing View#

The skeptical read is fair: retrieval quality is not patch quality.

The paper says this directly. ARB does not claim that retrieval is the dominant cause of patch failure, that file F1 replaces test-passing repair, or that the reported numbers estimate the effect of retrieval on final patch success. It isolates one component.

That boundary is a feature, not a weakness.

End-to-end benchmarks are necessary, but they can make every failure look like a model failure. Maybe the model reasoned poorly. Maybe the edit format was wrong. Maybe validation was weak. Or maybe the agent simply never found the file it needed. ARB gives teams a way to test that last hypothesis without running a full repair loop every time.

There is also a product risk. If teams over-optimize retrieval metrics, they may pack the model context with files that score well but do not help the task. That is why ARB includes budgeted context yield and no-gold abstention instead of only Recall@20. A context layer should know when to return less.

What To Copy Into Your Agent Stack#

You do not need to adopt ARB as-is to use the idea.

Start by logging a few retrieval questions for your own agent runs:

• Which files did the agent read before its first edit?
• Which files were edited, tested, or cited in the final answer?
• Did the agent read the relevant test before changing implementation?
• Did a review comment require context outside the commented file?
• Did the agent keep searching after receiving initial retrieved context?
• Did the retrieval layer ever abstain?

Then build a small internal benchmark from your own workflows. Use past PRs, review comments, failing test traces, and issue-to-fix histories. Freeze the repo at the base commit. Remove exact final patch paths and obvious leakage. Score whether your context layer ranks useful files before the agent edits.

The point is not to produce a leaderboard. The point is to stop treating "agent got lost" as one undifferentiated failure.

For a production harness, I would track four numbers:

That last metric is especially valuable. If the agent receives "good" retrieval results and immediately runs broad grep anyway, your retrieval layer may not be trusted, usable, or formatted well enough.

The Takeaway#

Repository context is becoming a measurable subsystem.

That is the real story behind ARB, CodeNib, repo maps, context ledgers, and long-horizon agent evals. The coding agent is no longer one model call with a giant prompt. It is a pipeline: context acquisition, planning, editing, verification, review, and memory. Each layer can fail separately.

Agent Retrieval Bench gives builders a clean way to ask the upstream question: before the agent wrote code, did it find the files it needed?

That question is going to matter more as teams run more parallel agents. The bottleneck will not only be model quality. It will be evidence quality: the right file, from the right commit, within the right budget, with enough receipts for a human to trust the edit.

FAQ#

What is Agent Retrieval Bench?#

Agent Retrieval Bench is a file-level benchmark for repository context retrieval in coding-agent workflows. It tests whether a retriever can rank the files an agent needs to read next, using workflow signals such as PR summaries, review comments, failure traces, and anchored edits.

How is Agent Retrieval Bench different from SWE-bench?#

SWE-bench evaluates full issue-resolution workflows. Agent Retrieval Bench isolates the upstream context-acquisition step. It does not ask whether an agent can write the final patch; it asks whether the context layer can find useful files before editing starts.

Why does retrieval matter for coding agents?#

Coding agents often fail before reasoning begins because they read the wrong files, miss relevant tests, or chase semantically similar but operationally irrelevant code. Better retrieval can reduce exploration cost and make the edit loop easier to supervise.

Does Agent Retrieval Bench prove better retrieval improves patch success?#

No. The paper is careful about that boundary. It shows that context acquisition is measurable and that current agents miss gold files on many samples. Proving downstream patch-success gains requires aligned executable repair benchmarks.

What should teams measure first?#

Start with first useful file rank, gold-in-budget at a realistic context size, abstention quality, and whether the agent searches less after receiving retrieved context. Those metrics are closer to real harness behavior than a single aggregate retrieval score.

Continue Reading#

• CodeNib Makes Repository Context a Data System - the infrastructure side of compiled repository context.
• The 98% Context Reduction Pattern - why agents should keep raw evidence outside the active model window.
• Coding Agents Need Codebase Maps, Not Bigger Prompts - the broader repository-map product shape.
• Long-Horizon-Terminal-Bench Shows Agents Need Progress Receipts - how long tasks expose missing eval signals.
• Diffs vs Whole Files: What Code Editing Agents Should Rewrite - why edit strategy should depend on task locality.

Sources#

• Agent Retrieval Bench arXiv HTML, fetched September 21, 2026.
• Agent Retrieval Bench arXiv abstract, fetched September 21, 2026.
• Hugging Face paper page for Agent Retrieval Bench, fetched September 21, 2026.
• Agent Retrieval Bench GitHub repository, fetched September 21, 2026.
• Agent Retrieval Bench dataset on Hugging Face, linked from the paper and checked September 21, 2026.
• Google Trends check attempted September 21, 2026 for coding agents, AI coding agents, repo context, repository context, and context engineering; Google returned HTTP 429, so no reliable Trends rows or numbers are reported.