The Quiet Tax on Your Cheap Agent Tier

Somewhere in your serving telemetry there is a cache hit rate, and it is likely the most honest number in your agent cost model. The largest published production trace of agentic coding - GitHub Copilot, June 2026, 3.2 million users, 761 million LLM calls, 95 trillion tokens - shows KV-cache hits averaging 90 percent inside a turn and 55 percent across turn boundaries (arXiv:2608.00101). Inside a conversation with the model, the cache does its job. At the seam between one turn and the next - where the agent has actually done something and is about to look at the result - the fleet re-encodes half of what it already paid for, because schemas recur in different orders and the prefix cache cannot see them.

When the bill lands, the default reflex is to shrink the model. This post is the argument that the reflex is pointed at the wrong end of the pipeline. Compression is the only major cost lever whose tax you cannot see, and this month's audit wave finally priced it. The same week, the serving side produced cost cuts that do not touch a single weight. We think the ordering matters: serve, don't compress.

Where this argument comes from#

This desk has been grading the layers of the agent pipeline for three weeks. On the first of August we argued your benchmark is lying to you: published agent numbers carry double-digit systematic noise, not random noise (your-benchmark-is-lying-to-you). On the sixth we argued the most expensive eval noise is the flat curve, because a plateau reads as science and gets funded like one (the-plateau-was-the-instrument). On the fifth we argued the durable unit of an agent run is shifting from tokens to explicit state, and we used the 90-to-55 cache split as the evidence that the turn boundary, not the token stream, is what serving economics organize around (kill-your-agent-runs-early).

The compressed model is the same storyline wearing a different costume. It is the layer of the fleet nobody argues with, because a quantized checkpoint looks deterministic: same weights, fewer bits, benchmarks inside the confidence interval. The judge needed an adversarial persuader before anyone measured its flips. The compressed model needs nothing, because the guardrails that exist were built so the tax slips past them.

The audit, in three findings#

A systematic audit that landed this week covers 3 LLMs across 11 compression methods, from both major families, quantization and pruning, and grades what aggregate metrics cannot see: knowledge retention, model confidence, and social bias (arXiv:2608.19670). Three findings, each one a category of hidden cost.

First, the knowledge loss is asymmetric. Compression disproportionately reduces the relative retention of head knowledge compared to tail knowledge - the well-learned knowledge the model holds best, which is exactly the knowledge a production assistant is leaned on for, erodes first. This is the margin-collapse result we covered in August taken one level deeper: median decision margin collapses to 0.86 at 4 bits, 0.33 at 3, and 0.00 at 2, while benchmark scores barely move; at 3 bits the decision to call a tool collapses toward inaction and roughly half the safety refusals vanish (arXiv:2608.06564). The aggregate number does not move because the aggregate is exactly the wrong instrument.

Second, the model stays confident about what it lost. Compressed models remain substantially confident in their incorrect answers on newly lost knowledge. That is the expensive failure mode: a model that quietly forgets costs you a miss; a model that quietly forgets while sounding certain costs you a decision made on a confident wrong fact. The margin paper's instrument now has company - a second, independent audit arriving at the same object from the knowledge side.

Third, the bias story is worse than flat. Stable aggregate bias scores conceal substantial, opposing shifts in stereotypical preferences across demographic subgroups. Flat is not evidence of safety; it is a sum that cancels. We have seen this exact shape before - a flat curve that turned out to be 263 benchmark bugs, and a plateau that read as science. At subgroup level it reads as nothing at all.

And the layer underneath, from the end of July: gently-compressed models pass every data-free quality guard - perplexity, accuracy, output-fidelity probes - and then invent procedure steps that were never in the instructions when they execute a standard operating procedure as an agent (arXiv:2607.28196). The effect is operator-specific: coherent low-rank truncation induces it, magnitude pruning at matched perplexity does not. The damage cannot even be predicted from the damage size; the axis is the coherence of the compression error, which every shipped guard is structurally blind to.

Three masks, one tax: passing aggregates while the head erodes, passing fidelity probes while procedure steps get invented, passing bias checks while subgroup preferences flip. None of these failures moves your dashboard. All of them are decision-grade.

The counter-case, with the steel it deserves#

Four objections deserve real weight.

First: this is one lab's audit on three models with synthetic evals. Real traffic may not reproduce the head-tail split, and a narrow task profile may sit entirely in knowledge that survives compression. Fair. The margin paper's constants, likewise, are per-model and do not transfer - that is the point of measuring. We are not claiming every compressed deployment is broken; we are claiming no aggregate metric can tell you whether yours is.

Second: compression is improving. Quantization-aware training, distillation backfill, and the general efficiency wave could erase the asymmetry inside a year. We think that is the actual race - not "compression is bad" but "the acceptance bar must live where the tax lives" - and a compression method that clears the head-knowledge bar at 4 bits would prove the ordering rule wrong in the good direction.

Third: the serving-side levers have their own ceilings. ReCache's benchmark is tool- and skill-schema workloads; it does not fix cold long-context prefill. The LFU result says no eviction policy is worth more than a fraction of a point - which also means caches are not free money; the CLEVER audit cuts raw hit rates of 51-60 percent down to 1.1-2.2 percent answer-substitutable hits at one encoder's threshold, with thresholds that do not transfer between encoders. Caching is the right lever, and it needs the same honest measurement the compression tax got.

Fourth: sometimes compression is the only budget answer. A 2-bit tier on aging hardware beats no tier. We are not banning it; we are pricing it. The one-bit repair from the margin paper is the cheapest measured mitigation, and the acceptance bar below is what you are buying with it.

None of these objections survives contact with the headline shape. All four defend compression by citing the instruments the tax is built to dodge.

What the serving side shipped this week#

While the audit was landing, the serving side answered with a cost cut aimed at exactly the measured seam. ReCache's starting observation is the tool-agent version of the 55 percent number: agents repeatedly encode the same tool and skill schemas, but schemas recur in different combinations and orders, which prevents standard prefix caching from reusing their key-value states (arXiv:2608.19662). The fix separates the schema encoding from the composition: resource-wise attention assigns resource-local positions so KV blocks become composition-invariant, restricts visibility to contribution-selected routes, and prunes to invocation-critical fields. The measured result: matched task performance - 82.3 versus 82.4 Inv-F1 on a benchmark assembled from seven public tool- and skill-use datasets - with a 3.655x time-to-first-token speedup and 92.43 percent less allocated KV-tensor memory. Critically for this argument: ReCache does not rewrite the sampling distribution. It changes which KV entries get materialized and which schema fields stay visible, the weights answer as before, and measured task performance holds - the two families of cost cutting differ exactly on this property.

Two more results same week, both downhill: eviction policy is a solved axis. Across three corpora, three capacities, two encoders, and 18 total settings, no semantic-cache eviction policy improves on LFU by more than 0.041 percentage points, with a structural reason why geometry-aware eviction cannot win - under insert-on-miss a new entry cannot have a resident neighbor inside the hit radius (arXiv:2608.20280). And from the end of July, Hybrid-model caching with a single cached linear state beats exact state composition on the Mamba-2 class - 86.8 versus 46.6 percent of full quality under one selector, at 0.46x the prefill TTFT (arXiv:2608.11231). The cheap and simple default keeps winning. Cache the schema, not the composed prompt. LFU, not the clever policy. One state, not the exact composition.

The ordering rule#

Put the two families side by side and the decision rule writes itself. Cheapest first: fix the serving shape - composition-invariant schema caching for tool workloads, LFU-clean eviction defaults, single-state initializers on hybrid backends, and from the same week, effort-adaptive decoding where a 1.5B model learned to choose NoThink, Short, or Long as its first token and cut response length 41 percent at 1.4 points on MATH500, 76 percent on easy GSM8K (arXiv:2608.20256). All of these change what the fleet costs. None of them changes what the model knows, or how confidently it is wrong, or which subgroup it prejudges.

Only then compress, and only against a bar that lives where the tax lives. Head-knowledge retention relative to the full model, measured per model and per bit-width, because the margin constants do not transfer. Calibration on lost knowledge: when the compressed model is wrong on a fact the full model knew, is it wrong loudly? Subgroup-level bias audits, because the aggregate will look fine. Never aggregate accuracy. Aggregate accuracy is the tax's hiding place, not its detector.

Here is the bet, stated so it can be graded. At least one of three failures will show up in the first quarter of traffic on any newly deployed compressed agent tier, relative to the full model: disproportionate head-knowledge misses, confident wrong answers on facts the full model held, or a subgroup preference flip that the aggregate hides. If a fleet audit finds none of the three on a 4-bit deployment, the tax is smaller than this audit suggests, the ordering rule is wrong, and we will grade ourselves accordingly. We give the same weight to the reverse: a vendor shipping margin sets and head-retention curves instead of rounded benchmark deltas will have our attention.

What would prove the ordering rule wrong in the strong sense: a compression family that clears the head-knowledge and calibration bars at 4 bits or lower on real traffic, or ReCache-style gains that fail to reproduce outside tool-schema workloads. Both are falsifiable, both would be reported here with the same care.

What developers should do#

1. Read your cross-turn cache hit rate before you read your weights. If it sits near the 55 percent zone, you are paying to re-encode compositions; try the serving levers first and measure the TTFT and memory deltas. The seam is where the money is.

2. Give every compressed tier an acceptance bar in the shape of the tax: head-knowledge probes, confidence on known-fact questions, bias audits split by subgroup. If you cannot run those three, you have not accepted the tier, you have shipped the tax.

3. Prefer distribution-preserving levers. Caching, eviction defaults, and effort modes change the cost, not the probabilities; lossy levers are a distribution change wearing a speed flag, and every one of them needs regression testing at the level of the behavior it touches.

4. When you must compress, take the one-bit repair and the per-model margin set. Constants do not transfer; the two-bit floor is where the instrument stops measuring; the cheapest known mitigation is the next bit.

5. If you sell compressed tiers, ship margin sets and head-retention curves. The labs measured them. The buyers are next.

Continue Reading#

• Your Benchmark Is Lying to You - the original position: agent benchmark numbers carry double-digit systematic noise
• The Plateau Was the Instrument - why flat curves are the most expensive eval noise, and the same shape at subgroup level
• Kill Your Agent Runs Early - the run lifecycle, the execution-state ledger, and the 90-to-55 cache split this post builds on
• The Judge Is Now a System You Design - the judge layer flipped by persuasion, and the jury that beats the frontier at 8 to 15 percent of the cost
• The Oracle Is Agreeing With Itself - the independence audit, the placebo arm, and why correlated checks collapse to self-agreement

Sources#

• The Asymmetric Harms of LLM Compression: arXiv:2608.19670 (2026-08-20)
• ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents: arXiv:2608.19662 (2026-08-20)
• Which Eviction Policy Should an LLM Cache Use? A Systematic Study Across Workloads, Capacities, and Encoders: arXiv:2608.20280 (2026-08-20)
• Which Decisions Low-Bit Quantization Breaks, and How to Predict Them: arXiv:2608.06564 (2026-08-06, rev. 2026-08-13)
• Fidelity Is Not Safety: Gently-Compressed LLMs Pass Every Data-Free Quality Guard Yet Invent Procedure Steps in Agentic Execution: arXiv:2607.28196 (2026-07-30)
• LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMs: arXiv:2608.11231 (2026-07-31)
• Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale: arXiv:2608.00101 (2026-07-30)
• Learning When to Think: arXiv:2608.20256 (2026-08-20)