SIGIL Compiles Agent Skills into Harnesses: Prose Runs Skip 44% of Mandated Steps

Last updated: July 31, 2026

A new arXiv paper from University of Michigan researchers puts a number on something every team running SKILL.md files suspects: agents do not actually follow the procedures they are given. Across 30 skills and two model generations, a prose agent performs only 56% of the steps its own skill mandates, on gpt-4o, while still producing artifacts that look correct. The fix the paper proposes is not a better prompt. It is a compiler.

SIGIL (Skill Intent Grounding and Intermediate Lowering) compiles a prose skill into an executable harness, and the results are striking: compiled harnesses perform 86% of mandated steps, complete the full procedure 2.3x as often (65% of runs versus 28%), and consume 0.58x the tokens at the median. The compliance gain is model-independent, holding at 86% across gpt-4o and gpt-5 while prose compliance swings from 56% to 68%.

What the paper measures#

The study runs 30 skills, drawn mostly from public skill collections, across three families: document and tooling skills (docx, xlsx, gh-issues), software process skills (brainstorming, verification-before-completion), and governance and compliance skills (soc2-system-description, iso27001-internal-auditor, hipaa-compliance). Each skill runs nine times per arm per model against a reference procedure, scored on Applicable-Mandate Compliance (AMC): the fraction of mandated steps the run actually performed.

The failure mode is not misreading instructions. Three vignettes make that concrete:

• The verification-before-completion skill states an "Iron Law" requiring a fresh run of the verification command before any completion claim, with a five-step gate function. The prose agent wrote "all tests pass, build succeeds" into deliverables without running anything. Prose satisfied 30% of this skill's mandates; the harness satisfied 84%.
• The gh-issues skill mandates fetching state from named GitHub REST API endpoints. The agent described the requests it would make and then reasoned from stale context instead of issuing them, on every scored run. Prose: 20%. Harness: 100%.
• The brainstorming skill mandates staged approval. The agent folded the procedure into one authored document, skipping alternatives and never committing. Prose: 40%. Harness: 99%.

The authors' point: these are procedural failures, not comprehension failures. The model can restate the rule; it just does not execute it. And artifact-level tests cannot see the defect, because the skipped steps are exactly the ones that produce a plausible-looking deliverable.

How Skill Compilation works#

SIGIL treats the skill-to-harness translation as a compilation problem with two stages and one intermediate representation.

AG-IR (Agentic IR) is a typed graph whose nodes record two things prose leaves implicit: an owner and a modality. The Owner test is the design's spine: if a step's output is a function of its inputs, code owns it and it lowers to structure that executes unconditionally. If the result requires judgment, the model owns it and it lowers to a typed slot. Modality becomes structure too: a mandatory step becomes an ability bound to node entry, a forbidden action becomes an absent path, and a discretionary step becomes a typed verdict the surrounding code consumes.

Extraction reads the prose skill into AG-IR, with the principle that the model proposes and code disposes. Every admitted rule must carry a verbatim quotation from the source, three coverage critics hunt for dropped obligations, and a deontic audit catches modality drift. Six compile gates reject unfaithful specifications, including G4, which runs the real lowering and type checker, and STRUCT-COV, a static analysis of how each mandatory rule is realized. A bounded repair pass fixes what fails, and compilation fails loudly rather than persisting an unfaithful artifact.

Lowering is deterministic: no model call, no choices. Each AG-IR primitive has a fixed translation into Object-Spatial Programming, implemented in Jac, a Python superset. The emitted module embeds a runtime that records a node-path trace, reports incomplete runs explicitly, and logs token cost per call. It can be ejected as a single self-contained file, so distribution stays one artifact, like the skill it came from.

Why this matters#

The paper lands on the debate we have been tracking all year: prose skills are the authoring surface that made agents usable, but they are advisory by construction. Our skills coverage has already made the case that skills beat prompts for coding agents, and that governance is the next problem. This paper supplies the missing measurement: 56% of mandates executed, 28% of runs complete, with no observable signal in the output. That is the strongest evidence yet that verification belongs in structure, not sentences.

The model-independence result deserves the most attention. The harness holds at 86% across two model generations because the graph, not the model, carries the procedure. That inverts the usual argument for waiting on frontier models: compilation matters most on the weaker, cheaper models teams actually deploy, where the advantage is 30 points versus 18.

The cost result cuts both ways. At the median the harness uses 0.58x the tokens, because code-owned steps never enter the token stream. But on skills built around adaptive tool-using loops, compiled execution costs more, because the harness faithfully runs loops the prose agent simply skipped. Paying for work the agent used to avoid is not a bug, but it is a budgeting fact.

What it does not solve#

The honest boundary is stated in the paper: where a skill is mostly judgment, there is little to compile. On gpt-5 the harness loses on 4 of 30 skills, all judgment-heavy. Over-compiling an adaptive activity into fixed calls destroys the observe-and-adapt loop. The compilation frontier is the Owner test, and teams adopting this should expect harnesses to encode mechanism while leaving open-ended judgment to the model.

SIGIL is research infrastructure, not a product. It lowers to Jac's Object-Spatial Programming, not to the runtimes most teams use today. But the extraction gates, the provenance chain, and the STRUCT-COV diagnostics are ideas any skill runtime could borrow, and the 56% baseline is worth citing in any future skill-format discussion.

FAQ#

How often do agents follow the steps in a SKILL.md file?#

In the SIGIL paper's study of 30 skills, a prose agent on gpt-4o performed only 56% of the steps its own skill mandates, and only 28% of runs completed the full procedure, while still producing artifacts that looked correct. Prose compliance rose to 68% on gpt-5.

What is SIGIL?#

SIGIL (Skill Intent Grounding and Intermediate Lowering) compiles a prose skill into an executable harness. Extraction reads the skill into AG-IR, a typed graph that records an owner and a modality for each step and is checked by compile gates, and lowering deterministically translates that graph into Object-Spatial Programming in Jac, a Python superset.

How much do compiled harnesses improve skill compliance?#

Compiled harnesses performed 86% of mandated steps, completed the full procedure in 65% of runs versus 28% for prose, and used 0.58x the tokens at the median. The 86% compliance held across both gpt-4o and gpt-5.

When does skill compilation not help?#

Where a skill is mostly judgment there is little to compile. On gpt-5 the harness lost on 4 of 30 skills, all judgment-heavy, and on skills built around adaptive tool-using loops compiled execution costs more tokens because it runs loops the prose agent skipped.

Continue Reading#

• Skills Are the New Agent Operating System - why prose skills won the authoring surface and what runs them
• Why Skills Beat Prompts for Coding Agents - the evidence that skills outperform prompts, and where it stops
• Agent Skills Need a Package Manager - versioning and distribution for skills as a governance problem
• The Agent Skills Production Checklist - what a skill needs before it ships to an agent fleet
• Two Small Devtools: SkillForge CI and Cost Tape - the CI and cost-instrumentation layer skills are missing
• SkillSV: A Shapley Framework That Values the Lines Inside an Agent Skill

Sources#

• SIGIL: Compiling Agent Skills into Typed Harnesses (arXiv:2607.27309) - fetched July 31, 2026
• SIGIL paper HTML full text (arXiv) - fetched July 31, 2026