SkillSV: A Shapley Framework That Values the Lines Inside an Agent Skill

Agent skills are increasingly written by automated feedback loops, and those loops raise aggregate scores while leaving one thing unexplained: which lines actually did the work. A new arXiv paper from Nanjing University of Aeronautics and Astronautics, Pengcheng Laboratory, Hefei University, and Microsoft introduces SkillSV, a structure-aware Shapley-style framework that attributes a skill's performance to its internal units - the rules, examples, scripts, and heuristics that make up a skill.md file.

The results make a strong case that skill files carry substantial dead weight: an attribution-guided refinement pass cut skills to 69% of their original tokens on average with no significant performance change across all four benchmarks tested.

The problem: skills are structured, so flat ablation breaks#

Skill valuation differs from data or prompt-span valuation because skill units are not independent sentences. They depend on other units, belong to a document hierarchy, trigger agent behavior, and consume limited prompt context. Delete a rule that a script depends on and you have not measured the rule, you have measured a broken skill.

Existing attribution methods miss this. Closure-LOO (leave-one-out) ablates each unit with its dependency closure, and the paper shows it is often miscalibrated: on LiveMath, LOO's summed values came out at -0.83x the measured content lift, a sign error, while SkillSV's totals recovered the lift at 0.97x. An LLM judge scoring units independently fared better but still worse than SkillSV in pruning tests. The core finding is that redundancy masks value: in the full context, units that duplicate each other's effect collapse to near-zero estimates, so true sparsity only emerges through multi-context evaluation.

How SkillSV works#

SkillSV compiles a skill into three artifacts before any valuation happens:

• Units - the editable pieces (a rule, an example, a script call, a heuristic), extracted by a compiler specification detailed in the paper's appendix.
• Dependencies - edges between units, so counterfactual skills are only evaluated when they are valid skills, not mangled documents.
• Hierarchy - the document structure the skill ships in.

Valuation then runs as a Shapley-style game over feasible insertion orders, with two tricks that separate content value from context cost. Paired deletion measures what removing a unit does; length-neutral padding replaces removed content with filler of equal length, so the prompt-occupancy cost is not confused with the content contribution. Because agent rollouts are noisy and expensive, estimates come from a budgeted estimator (K=12 rollouts, b=8, noise gate tau=0.05) rather than exhaustive enumeration.

The framework is optimizer-agnostic: it values only the compiled skill document, regardless of whether it came from Trace2Skill, TextGrad, GEPA, or SkillOpt, the four optimizers whose converged skills the paper evaluates on LiveMath, OfficeQA, SpreadsheetBench, and ALFWorld with a frozen GPT-5.5 agent.

What the numbers say#

The skill lift itself is worth noting before the attribution results. Full skills versus no skill on the target agent: +16.7 points on LiveMath, +18.6 on OfficeQA, +51.2 on SpreadsheetBench, +8.3 on ALFWorld. Skills clearly carry real value - the question is how much of that value sits in how many lines.

The answer concentrates sharply. The top 10% of units account for 21% of the value mass on OfficeQA, 35% on LiveMath, 60% on SpreadsheetBench, and 100% on ALFWorld. Value is not distributed across the file; it is concentrated in a small number of units, with the rest near the estimator's noise floor.

That concentration makes safe pruning possible. In a sequential pruning test (remove units from lowest to highest value, re-score at each step), SkillSV produced a significantly higher AUC than all baselines: +0.026 over Closure-LOO, +0.049 over the LLM judge, +0.082 over random ranking, with no 95% confidence intervals containing zero. Random ranking degraded rapidly and even collapsed below the minimal-skill floor on OfficeQA.

The most practical result is a single attribution-guided refinement step: an editor receives the skill plus the SkillSV report (content value, context cost, net effect per unit) and is told to preserve high-value content, remove harmful or near-zero units, and compress units whose content value is positive but whose context cost is large. The revised skills retained 69% of original tokens on average, with no significant performance change on any of the four benchmarks. "Lossless compression" of a skill file is now a measured result, not a hope.

Why this matters#

This paper lands in the middle of the skills debate this site has tracked all year. The skills-over-prompts argument is settled; the skills-governance argument is being settled; and this adds the missing instrument: how do you know what a skill is worth before you trust it in production?

Two implications stand out. First, skill authors should expect redundancy. Automated optimizers iterate until scores plateau, and they have no incentive to keep files lean, so converged skills accrete examples and heuristics that protect against failures the optimizer already handled. The paper's 69%-token result is effectively a measure of how much of that accretion is removable. Our coverage of SkillForge and Cost Tape has been pushing CI and cost instrumentation for skills; SkillSV supplies the per-unit signal those systems currently lack.

Second, it changes how to edit a skill by hand. The classic instinct is to prune what reads redundant, which is exactly what Closure-LOO-like reasoning does, and the paper shows that instinct collapses redundant units to zero and hides the units that actually matter. Context-cost separation is the key idea: a unit can be valuable content yet costly to keep, and only paired deletion plus length-neutral padding sees both numbers at once.

The honest limits: the paper values units within a fixed skill under a fixed agent, so the values are agent-specific, and the four benchmarks are tool-based rather than coding-heavy (ALFWorld is the most interactive). It does not claim to optimize skills, only to explain them. But as a diagnostic layer, "not only whether a skill works, but which parts work" is exactly the framing skill runtimes and registries need next.

Continue Reading#

• SIGIL Compiles Agent Skills into Harnesses - a sibling paper measuring how faithfully agents execute skill procedures at all
• Skills Are the New Agent Operating System - why prose skills won the authoring surface
• Agent Skills Need a Package Manager - versioning, distribution, and governance for skill files
• Two Small Devtools: SkillForge CI and Cost Tape - the CI and cost-instrumentation layer skills are missing
• Self-Improving Skills for Claude Code - what happens when agents edit their own skills

Sources#

• What Is a Skill Worth? Structure-Aware Shapley Valuation of Agent Skills (arXiv:2608.04562) - abstract and authors fetched August 6, 2026
• SkillSV full text (arXiv HTML) - experimental setup, Tables 1-3, Sections 4.3-4.4, and conclusion fetched August 6, 2026