Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories

Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories

Authors: Prakhar Khatri

arXiv ID: 2607.27250

Problem: Persistent context files (AGENTS.md, CLAUDE.md) are standard practice for guiding AI coding agents, yet the evidence for their effectiveness is contradictory. Single-agent studies reach opposite conclusions, and nobody has isolated whether context files help because agents lack repository knowledge or because of something else entirely.

Key Methodology:

• Controlled ablation of context-injection strategy across two frontier agents (Claude Code and Codex) on 17 real tasks from 3 repositories (15 shared, 2 Codex-only)
• 288 evaluated runs with gold-test evaluation, plus equivalence testing to bound the size of any effect
• Failure-mode triage categorizing why agents fail (implementation skill vs missing knowledge)
• Manipulation probe: inject task-answer-relevant context into a real AGENTS.md to test whether context can convert a near-miss to a pass

Key Results:

• Context strategy does not measurably move correctness on either agent, bounded to <=10-15pp by equivalence testing
• Agents fail on implementation skill (feature design, pattern selection, exact wiring), not missing repository knowledge that a context file could supply
• The manipulation probe confirms a real AGENTS.md never converts a near-miss to a pass on either agent
• Borderline task difficulty is agent-specific (Spearman rho = 0.75), offering a candidate explanation for prior contradictory results: single-agent studies draw tasks from different agents' informative bands

Applied Context: For teams that treat context files as the fix for underperforming agents: the file is probably not the bottleneck. The actionable move is to classify the agent's actual failure mode first (skill gap vs knowledge gap) before investing in more documentation, and to be skeptical of any single-agent claim about context-file efficacy, since task difficulty itself is agent-dependent. All code, data, and analysis are released.

Paper: arXiv:2607.27250