CLAUDE.md Files Never Stop Growing: A New Paper Names the Mechanism

Every AGENTS.md, CLAUDE.md, or copilot-instructions.md you have ever maintained has the same lifecycle: it grows, someone rewrites it from scratch, and then it grows again. A paper posted August 11 (arXiv:2608.11095) gives the phenomenon a name and, for the first time, identifies the root cause: "catastrophic remembering," the mirror image of the catastrophic forgetting studied in continual learning.

The claim is not that agentic prompt files grow. That has been known. The contribution is showing why: maintainers cannot delete instructions because the reasoning behind each one decays, so the only safe operation is appending. The author, Kushal Chakrabarti, tracks 247,694 individual instruction lifetimes across 1,867 GitHub repositories and then runs controlled maintenance experiments with a known-optimal prompt to prove the mechanism and test a fix.

The numbers: a ratchet, not a trend#

Across 1,801 multi-version files, the average prompt more than triples its instruction count over its own lifetime (+226%), with total size up +140%. Each commit adds a net +4.9 instructions across 19,267 commits tracked. The median file ends its life at 39 instructions, and the 90th percentile sits at 131 - well past the range where instruction-following measurably degrades.

Growth is not gentle, either. 77.3% of instruction "deaths" arrive in a single commit that bulldozes the file wholesale or migrates it to a sibling file, not in careful pruning. And the ratchet survives the bulldoze: a file drops to 59.5% of its pre-rewrite count, then recovers to 91.5% within 10 commits. It regrows faster afterward, at 4.9% per commit versus 4.1% before.

Why deletion is hard: O(2^|D|) and falling#

The paper frames prompt maintenance as estimating an unobservable constraint set from censored feedback. Deleting an instruction safely requires proving it is excess, which means probing every subset of the remaining instructions - O(2^|D|) for a prompt of |D| instructions - because two instructions can each look free alone while both are needed together.

The one thing that collapses that cost to O(1) is knowing why an instruction was added. That latent reasoning decays. The empirical signature is a deletion hazard that falls with instruction age at -0.032 per commit (95% CI [-0.047, -0.019]) - the opposite of what instruction staleness predicts. And the multi-author interaction confirms the mechanism: hazard decays faster the more authors touch a file (beta -0.021, z = -11.7). More authors means more undocumented handoffs, meaning less surviving rationale.

This connects directly to what we already know about agent config files. A SCAM 2026 study found 91% of popular repos carry at least one of six configuration smells, with Context Bloat (files over 200 lines) second most common - see our AGENTS.md Configuration Smells write-up. It also squares with Anthropic's own finding that cutting 80% of Claude Code's system prompt produced zero regression, which we covered in Anthropic Removed 80% of Claude Code's System Prompt. Both papers show the same underlying truth: prompts carry more instructions than they need, and nobody can tell which ones are excess.

The fix: comments on prompt instructions#

Software engineering solved this problem decades ago with the comment. The paper's intervention is the same move applied to agentic prompts: annotate each instruction with the failure that motivated it, a hypothesis, and how it has fared. Comments are stripped before the prompt reaches the model, so they cost nothing at inference time and are visible only to the next maintainer.

In the controlled testbed (552 maintenance histories over an inverted IFEval, where the optimal prompt is known), the effect is dramatic:

Informative comments remove 99.3% of the excess size (+211.3% to +1.4% at 51 steps) at parity constraint satisfaction. Two controls matter here. Comment-shaped noise lands within noise of the no-comment arm, so the mechanism is the content, not the annotation itself. And the ablation on comment payloads shows a narrative of attempts without outcomes is the worst arm of all (+70.0%), worse than no comments: an unvalidated premise handed to the next maintainer is worse than none.

The effect compounds as maintainers get more capable. Across three maintainer tiers, the uncommented arm's excess rises from +67.7% to +571.9% - stronger agents ratchet harder - while commented prompts hold near their cover.

Real prompts: noisy instructions cost correctness#

The WildIFEval replication moves from synthetic worlds to real prompts. Seeding a prompt with 16 noisy instructions drawn from other tasks costs 24.1pp of correctness on the true instructions already present (satisfaction drops from 65.6% to 41.5%). Comments recover most of it: satisfaction rises from 50.4% to 62.0% over three maintenance rounds, an 11.6pp gain. The magnitude is judge-dependent - a second judge measures 7.8pp - so treat the exact number as approximate, but the direction is consistent.

This is the missing half of the context-file ablation story we covered in July, where adding context files did not move correctness. That study tested presence versus absence; this one shows that the content and maintenance of those files is what matters. A bloated, uncommented prompt actively degrades instruction-following, and the degradation is recoverable.

What this means for your AGENTS.md#

The practical takeaway is cheap and immediate: when you add a rule to your agent config file, write the why next to it. One line naming the failure it prevents and the outcome it produced. The paper's closing question is the whole argument: "If English is the new code, why don't we have comments yet?"

Three things worth doing this week:

• Audit your AGENTS.md against the six configuration smells and the 200-line guideline Anthropic recommends for CLAUDE.md files.
• When you append a rule, append its rationale. If the file has grown past a few dozen instructions and nobody remembers why half of them exist, the model is in the same position as the next maintainer: guessing.
• Treat wholesale rewrites as a smell, not a reset. The paper shows growth resumes at a higher rate after every bulldoze.

For more on keeping agent context lean, see our 98% Context Reduction Pattern, the case for skills over prompts, and the production checklist for agent skills. The pattern across all of them is the same: keep the always-loaded file small, and push task-specific knowledge into structures that load on demand.

Continue Reading#

• AGENTS.md Configuration Smells: 91% of Popular Repos Get One of Six Wrong - the SCAM 2026 taxonomy of what goes wrong in agent config files
• AGENTS.md Files Don't Move Coding Agent Correctness - the 288-run ablation that found context injection strategy does not shift correctness
• Anthropic Removed 80% of Claude Code's System Prompt - what the cut taught the industry about prompt weight
• The 98% Context Reduction Pattern - keeping intermediate state out of the model context
• Why Skills Beat Prompts for Coding Agents in 2026 - the control-stack workflow maturing past giant hand-written prompts

Sources#

• arXiv:2608.11095 - Why Does CLAUDE.md Keep Growing? Catastrophic Remembering in Agentic Coding (abstract)
• Full text (arXiv HTML)