CAPA Benchmark: Why Coding Agents Should Learn Your Habits Across Sessions

Last updated: July 31, 2026

What changed#

A new arXiv paper (2607.26611, submitted July 29) formalizes the problem every heavy coding-agent user hits within a week: your assistant asks the same clarifying questions over and over because it treats every session as a fresh conversation.

The authors call the task "personalized ambiguity adaptation" and ship CAPA (Cross-Session Adaptation to Personalized Ambiguity), a benchmark of 600 executable coding sessions built from HumanEval tasks. Sixty synthetic users each have a recurring ambiguity mechanism - a stable way they leave required implementation detail out of a request - plus a stable way they resolve it. Each user contributes five resolved history sessions and five held-out evaluation sessions, and the assistant's job is to infer the pattern and apply it to the held-out requests.

The six mechanisms are grounded in real coding conversations from WildChat: domain-cognitive polysemy (your own shorthand terms), structural logic misalignment (omitted scope or thresholds), habitual context omission (assumed conventions like "the usual entry point"), system-boundary misconception (presupposing file or runtime state you never fetched), conversational context misalignment (references to earlier decisions), and implicit constraint under-specification (unstated validation or edge-case requirements).

The running example is "normalize" meaning min-max scaling to one collaborator and z-score standardization to another. A colleague who has worked with you knows which one you mean; a coding assistant with no memory of your sessions guesses, or asks.

What the results say#

Twelve models across closed frontier, open-access frontier, and compact open-source tiers were tested under no-history and same-user-history conditions, scored on Executable Success (ES), First-Turn Executable Success (FT-ES), and Turns-to-Completion (TTC):

• Same-user history improved ES for 11 of 12 models, FT-ES for all 12, and reduced average TTC for every model
• Across models, history added 6.8 points to ES but 15.6 points to FT-ES, and cut 0.81 turns per task
• Claude Opus 4.8 went from 24.3% to 60.3% FT-ES with history (ES 88.0% to 90.0%)
• GPT-5.5 improved ES 74.3% to 84.3% and FT-ES 2.3% to 31.0%
• GLM-5.2 hit 89.7% ES and 46.7% FT-ES, competitive with the best closed models
• Compact models averaged 44.0% ES, with Qwen3.5-27B gaining the most from history (+18.3 points ES)
• The ceiling is far off: with no ambiguity at all, GPT-5.5 and DeepSeek V4 Pro both hit 100% ES and about 1.1 TTC

Two control experiments make the paper worth reading in full. First, shuffling history from other users still helped models (generic resolved sessions contain useful coding and dialogue patterns), but matched same-user history added consistent extra gains in FT-ES and TTC - the personalization signal is real, not just in-context learning from extra text. Second, general-purpose memory systems (mem0, A-mem) underperformed raw same-user history across all three metrics on two of three models tested. The authors' diagnosis is an objective mismatch: memory tools are built to store and retrieve facts, not to identify a user's recurring ambiguity-resolution pattern and decide whether the evidence justifies direct implementation.

Their lightweight fix is a "same-user history gating" workflow: a cheap gate LLM reviews resolved history for consistent ambiguity-resolution evidence, then either surfaces the single most informative prior session or states what remains unresolved. On GPT-5.5, DeepSeek V4 Pro, and GLM-5.2 it improved FT-ES by up to 13.3 points over raw history while holding ES steady.

Why it matters to developers#

This is the measurable version of a UX annoyance you already live with. Every clarifying question is a context switch: you leave flow, explain yourself, wait for a turn. The paper shows the gap between what models can do and what they do by default is partly a memory-retrieval problem, not a capability problem. Claude Opus 4.8 is 2.5x better at first-turn success with five prior sessions of the same user's history in reach. The model knows how to use the evidence - the missing piece is deciding which evidence to use.

It also reframes what "agent memory" should store. The agent-memory space is crowded with tools competing on fact retrieval (preferences, credentials, project trivia). This paper argues the higher-value memory is procedural: how this user typically leaves requirements underspecified and what they usually mean. That maps directly to patterns the site has covered before, from Agent Memory Benchmarks Are Not Enough (benchmarks measuring recall miss what matters) to the Context Ledger argument that memory should be an auditable, append-only record rather than a silent profile.

For practical setups, the takeaway is cheaper than it sounds: the winning method is a gate prompt over your existing session logs, not a new memory database. If you run Claude Code, Codex, or an opencode-style agent with a session store, you can approximate same-user history gating today by having the agent skim the user's last few resolved sessions before planning a new ambiguous task - which is also consistent with the recent AGENTS.md ablation finding that what you put in context moves correctness more than the model's default behavior.

FAQ#

What is the CAPA benchmark?#

CAPA (Cross-Session Adaptation to Personalized Ambiguity, arXiv 2607.26611) is a benchmark of 600 executable coding sessions built from HumanEval tasks. Sixty synthetic users each have a recurring way of leaving implementation detail out of a request and a stable way of resolving it, and the assistant has to infer that pattern from five resolved history sessions and apply it to five held-out sessions.

Does session history help coding agents ask fewer clarifying questions?#

Yes. Same-user history improved first-turn executable success for all 12 models tested and reduced average turns-to-completion for every model. Across models it added 15.6 points to first-turn success and cut 0.81 turns per task, and Claude Opus 4.8 went from 24.3% to 60.3% first-turn success with history.

Do memory tools like mem0 solve personalized ambiguity?#

Not in this study. General-purpose memory systems (mem0, A-mem) underperformed raw same-user history across all three metrics on two of three models tested. The authors attribute it to an objective mismatch: memory tools are built to store and retrieve facts, not to identify a user's recurring ambiguity-resolution pattern.

What is same-user history gating?#

It is the paper's lightweight fix: a cheap gate LLM reviews a user's resolved history for consistent ambiguity-resolution evidence, then either surfaces the single most informative prior session or states what remains unresolved. On GPT-5.5, DeepSeek V4 Pro, and GLM-5.2 it improved first-turn success by up to 13.3 points over raw history while holding overall success steady.

Continue Reading#

• Agent Memory Benchmarks Are Not Enough - why recall-based memory evals miss the working-value question
• AI Agent Memory Needs a Context Ledger - a design for auditable cross-session memory
• AI Agent Memory Patterns - the pattern catalog behind long-horizon agents
• AGENTS.md Files Don't Move Coding Agent Correctness - what context actually changes in agent output
• Intent Debt: The AI-Era Debt Nobody Is Tracking - the cost of ambiguity that never gets resolved
• AgentMemory Is Useful Only If You Audit What It Remembers

Sources#

• Fewer Clarifications, Better Code: Benchmarking Cross-Session Personalized Ambiguity Adaptation in Coding Assistants (arXiv 2607.26611) - fetched July 31, 2026
• Full text (arXiv HTML) - main results Table 1, shuffled-history control Table 3, memory-method comparison Table 4, history gating Table 5