Skip to main content
Watch: Grok Bot Built My AI Video Arena While I Tested 8 Models

Agents / Skills / Evaluation

Agent Skill Evolution: How Revisions Affect Coding Agents

Jiajie Wang, Yutong Zhao, Tianlin Li, Huashan Chen, Jinfu Chen, Kebin Peng, Sen He · University of Arizona; California State University, Long Beach; Chinese Academy of Sciences

arXiv:2610.04832

Agent Skill Evolution: How Revisions Affect Coding Agents

Authors: Jiajie Wang, Yutong Zhao, Tianlin Li, Huashan Chen, Jinfu Chen, Kebin Peng, Sen He

arXiv ID: 2610.04832

Problem: Agent Skills, the SKILL.md files that tell a coding agent how a project works, are revised like code, but what a revision actually does to the agent's behavior is unknown. Teams edit skills on intuition, with no measured account of which changes move compliance and what they cost.

Key Methodology:

  • Analysis of 2,608 first/last revision pairs from 3,159 Skills to characterize how skills evolve and how commits that revise a skill correlate with changes to harness files such as CLAUDE.md
  • Focus on rule changes: revisions that add or remove an automatically checkable rule, for example "run allium check"
  • Effect measured on 21 models in single answers and four agents in a sandbox, with cost measured on 20 of the models and the four agents
  • Three of the agents assessed by blind judges for final correctness, plus an on-demand loading setting that matches how real tools load a skill body only when the agent decides it needs it

Key Results:

  • Most revisions (55%) change a rule or procedure, and skill-revising commits change harness files more often than other commits of the same size
  • Across 16 open-weight models, an added rule raises single-answer compliance by +0.41 on average
  • Across the four agents, the rate at which the agent takes the required action rises by +0.23 on average (+0.16 to +0.36), and final correctness rises by +0.10 (+0.06 to +0.14)
  • The gain comes mainly from rules that name a command or path the old skill did not mention
  • Under on-demand loading, agents keep about half the action gain (51%), and three open models about 38%
  • A revision adds 18-19% input tokens to a single answer and no detectable cost to an agent episode, while loading a skill body raises episode tokens by 50% on average

What it means for developers: The revision that pays is the one that hands the agent an executable handle: a concrete command or path it did not previously have. Prose guidance about intent moves much less. That makes skill review a mechanical question, whether a new line adds a new named action, and it prices progressive disclosure: editing a skill is cheap at the episode level, but loading one is not, so revision effort belongs where the skill will actually be loaded.

Paper: arXiv:2610.04832