ACE vs ALTK-Evolve: How You Deliver Agent Memory Determines the Token Bill

Give an LLM agent a realistic multi-step task and when it fails, it is usually not for lack of knowledge. It mis-paginates an API, resolves the wrong person, or returns a value when none was asked for. Two research systems attack exactly this failure mode by having the agent learn from its own trajectories, with no weight updates and no human labels: Agentic Context Engineering (ACE, arXiv 2510.04618) and IBM Research's ALTK-Evolve (arXiv 2603.10600). A new IBM Research post published today runs both head-to-head on the same models and harness, and the headline is the token bill: on a strong model, ALTK-Evolve matches or beats ACE's accuracy at roughly 40% of its inference cost, and on a weaker model at about one-seventh.

Both systems turn an agent's past trajectories into reusable lessons and feed them back at inference time. Where they disagree is delivery, and that is what shows up in the cost numbers.

What they agree on: never compress the lessons#

ACE names the two failure modes of naive agent memory: brevity bias, where optimization collapses toward short generic instructions, and context collapse, where a model asked to rewrite its whole context each step summarizes the detail away. Its answer is a rich itemized playbook with a helpful/harmful counter on every bullet, and letting the model distill relevance at read time.

IBM Research reached the same conclusion from the other direction. Every ALTK-Evolve guideline keeps a support count, the number of independent episodes that produced it, and the store is never summarized down to a handful of rules. A lesson five tasks discovered is a different object from one that appeared once. The post calls the shared principle "count them, don't collapse them": ACE's per-bullet counters and ALTK-Evolve's support counts are two spellings of the same idea.

The systems also both refuse to hand the agent its memory as a prompt-adjacent summary, a design position our own coverage has been moving toward. We wrote about the context ledger model of agent memory (source-linked, scoped, expiring entries) and about why memory benchmarks alone are not enough. These two papers are the same argument, now with controlled numbers behind it.

Where they differ: fixed injection vs calibrated retrieval#

Consolidation. ACE grows one playbook through a Generator to Reflector to Curator loop, applying incremental delta updates and de-duplicating by embedding. ALTK-Evolve clusters near-duplicate lessons and merges within a cluster, support-conserving: when several lessons merge, the survivor inherits their combined count, so the store shrinks without losing how much experience backs each guideline. It also extracts typed guidelines (strategy, recovery, optimization) with causal attribution and provenance back to the source trajectory, at subtask granularity so a lesson learned on one app transfers to another.

Delivery. ACE injects the comprehensive playbook on every step, the same way regardless of model or task. ALTK-Evolve treats delivery as a dial: a small fixed core of high-support guidelines, extended per task with a handful selected for that task (cosine or LLM-guided, priority-weighted), or the full consolidated set when the model has headroom. The same lessons are available to both agents; ALTK-Evolve just sends however many a given model can actually use.

The numbers on AppWorld#

Both systems ran on the same base ReAct agent (each step writes Python, the environment returns output) on AppWorld test_normal, 168 tasks. ACE's paper used DeepSeek-V3.1, so IBM re-ran both systems in-house on identical models and harness to keep the comparison controlled. Memory was mined from train/dev only, scored pass@1.

DeepSeek-V3.2 (the stronger model):

gpt-oss-120b (the weaker model):

On the strong model ALTK-Evolve wins both metrics at about 40% of ACE's inference cost. On the weak model it is a near-tie on accuracy (56.0 vs 54.8, which IBM calls within the benchmark's run-to-run noise) at about one-seventh the cost.

The by-difficulty breakdown explains why. On gpt-oss-120b, ACE's full playbook edges ahead on Easy and Medium tasks, where generic instruction-following gets most of the way there and a comprehensive prompt helps more than it distracts. On Hard tasks, curated retrieval pulls ahead decisively: 31.8 TGC for ALTK-Evolve vs 23.8 for ACE vs a 19.1 no-memory baseline, and the hard tier decides the aggregate. On DeepSeek-V3.2 the stronger model absorbs the full playbook well enough to edge ACE on Medium, but ALTK-Evolve leads Easy, Hard, and Overall.

A fair caveat: the two no-memory baselines differ (72.0 vs 79.8 TGC) because the systems use different prompt templates, and the comparison rests on what a prompt tweak cannot explain: same-or-better accuracy at a fraction of the tokens.

Why it matters to developers#

Two practical lessons land here for anyone building agent memory.

First, context is the new line item. Injecting 634K tokens per task instead of 263K is not a micro-optimization; it is the difference between an agent pipeline that fits a budget and one that does not. This is the same conclusion as SkillSV's pruning results, where attribution cut skill files to 69% of their tokens with no significant accuracy loss, and SIGIL's compiled harnesses, which hit 86% procedural compliance at 0.58x the tokens. The pattern across all three: most of what you feed an agent costs tokens without buying accuracy, and measuring that is a design task, not an afterthought.

Second, one-size-fits-all injection is the wrong default. A large context overwhelms a weaker model rather than helping it, and it crowds out the specific lesson a hard task needs. Calibrated delivery, a small core of high-support guidelines plus per-task selection, is a cheap mechanism that shows up in the numbers. The agent memory tooling comparison we maintain lists tools that store context; the harder question this post raises is how much of that context should reach the model on any given step.

Continue Reading#

• AI Agent Memory Needs a Context Ledger - what agent memory should look like beyond magic recall
• SkillSV: A Shapley Framework That Values the Lines Inside an Agent Skill - pruning skill files to 69% of tokens without accuracy loss
• SIGIL Compiles Agent Skills into Harnesses - procedural compliance and token costs of skill delivery
• Context Files and Coding Agents: An Ablation Study - what actually happens when you add context to agents
• Agent Plugins 1.0 - how the same lesson sets get distributed and injected into real agents

Sources#

• Thinking of ACE? We Can Do It with Fewer Tokens - IBM Research on Hugging Face - published August 11, 2026, includes full benchmark tables and method notes
• Trajectory-Informed Memory Generation for Self-Improving Agent Systems (ALTK-Evolve) - the technical report
• Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models (ACE) - the ACE paper
• ALTK-Evolve on GitHub - the extraction, consolidation, and retrieval pipeline
• AppWorld Benchmark - the evaluation environment (168 tasks, test_normal)