ANALYSIS
18 items
18 posts
The text agents read is a control surface: bad MCP error messages cost more on stronger models, and context hygiene gains value with every model upgrade.
Approve effects, not invocations: why agent approval records miss harm in install hooks and skill combinations, and why pipelines need closure metrics.
An agent's report on its own run is unreliable, and eight studies show why. The fix: append-only traces, obligation ledgers, and records the agent cannot write.
AI scribe audits show omission is the dominant error, and LLM judges are near coin-flip at catching it. The fix: enumerate the facts, then check each one.
Agent instruction following collapses past five or six simultaneous constraints. Why rules also die in compaction and handoffs, and why side channels help.
AI agents often claim completion they did not earn. New benchmarks measure the gap, and evidence-carrying termination makes done a checked artifact.
Agent model compression has a hidden tax: quantized and pruned agents lose knowledge first and stay confidently wrong. Why cheaper agents start at the cache.
Self-improving test generation loops can grade against an oracle that agrees with itself. An audit shows the inflated gain and why independent checks count.
LLM judges flip verdicts under pushback and rankings shift with token budget. The design spec for a judge system, including cheap deliberating juries.
A benchmark plateau can be a broken yardstick: a SciCode audit found 263 defects, and corrected scores jump to 84-98 percent. How to read flat leaderboards.
Kill your agent runs early: production traces show context goes stale at turn boundaries, and the fixes that moved numbers are kill, carry the state, restart.
The LLM judge is leaving the agent loop: evidence gates, verifiable rewards and deploy-time certificates replace verdicts with structure. Here is our bet.
Agent benchmark noise is being fixed by architecture - ledgers, counterfactuals, decomposition, personas, readout discipline - not by smarter judges.
Agent benchmarks are noisier than their scores admit: audits found misaligned ground truth, lenient model judges and scalars that hide failures. What to trust.
Agent memory is moving into the model: in-backbone memory, parametric memory, reconstruction and transactional writes challenge external stores. What to do now.
What happens when AI tokens are too cheap to meter: five scenarios for developers and knowledge work, each with the signal that tells you it is arriving.
What if AI was free tomorrow at today's capabilities? A thought experiment on what changes for developers when price hits zero, and what stays scarce.

Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.