Agents / Skills / Evaluation
Agent Skill Evolution: How Revisions Affect Coding Agents
Jiajie Wang, Yutong Zhao, Tianlin Li, Huashan Chen, Jinfu Chen, Kebin Peng, Sen He · University of Arizona; California State University, Long Beach; Chinese Academy of Sciences
Agent Skill Evolution: How Revisions Affect Coding Agents
Authors: Jiajie Wang, Yutong Zhao, Tianlin Li, Huashan Chen, Jinfu Chen, Kebin Peng, Sen He
arXiv ID: 2610.04832
Problem: Agent Skills, the SKILL.md files that tell a coding agent how a project works, are revised like code, but what a revision actually does to the agent's behavior is unknown. Teams edit skills on intuition, with no measured account of which changes move compliance and what they cost.
Key Methodology:
- Analysis of 2,608 first/last revision pairs from 3,159 Skills to characterize how skills evolve and how commits that revise a skill correlate with changes to harness files such as CLAUDE.md
- Focus on rule changes: revisions that add or remove an automatically checkable rule, for example "run allium check"
- Effect measured on 21 models in single answers and four agents in a sandbox, with cost measured on 20 of the models and the four agents
- Three of the agents assessed by blind judges for final correctness, plus an on-demand loading setting that matches how real tools load a skill body only when the agent decides it needs it
Key Results:
- Most revisions (55%) change a rule or procedure, and skill-revising commits change harness files more often than other commits of the same size
- Across 16 open-weight models, an added rule raises single-answer compliance by +0.41 on average
- Across the four agents, the rate at which the agent takes the required action rises by +0.23 on average (+0.16 to +0.36), and final correctness rises by +0.10 (+0.06 to +0.14)
- The gain comes mainly from rules that name a command or path the old skill did not mention
- Under on-demand loading, agents keep about half the action gain (51%), and three open models about 38%
- A revision adds 18-19% input tokens to a single answer and no detectable cost to an agent episode, while loading a skill body raises episode tokens by 50% on average
What it means for developers: The revision that pays is the one that hands the agent an executable handle: a concrete command or path it did not previously have. Prose guidance about intent moves much less. That makes skill review a mechanical question, whether a new line adds a new named action, and it prices progressive disclosure: editing a skill is cheap at the episode level, but loading one is not, so revision effort belongs where the skill will actually be loaded.
Paper: arXiv:2610.04832