Coding Agents Almost Never Read Open Source Contribution Rules: RepoComplianceBench Study

Last updated: July 31, 2026

Open source maintainers are writing rules for AI-generated contributions: total bans, mandatory disclosure, verification gates, human sign-offs. A new study asks the question nobody had measured: do coding agents actually read and follow those rules? The short answer, from RepoComplianceBench, is that they almost never do it on their own - and no prompt intervention makes an agent refuse to contribute to a repository that has banned AI entirely.

The paper, "A First Look at Coding Agents' Compliance with AI Contribution Rules in Open-Source Communities" (Yang, He, and Zhou, submitted July 29, 2026), is the first empirical estimate of real-world rule compliance by coding agents. The finding matters if you maintain a repository with an AI policy, or if you run an agent against one.

What the study did#

The researchers curated 106 issues from 49 repositories that contain AI contribution rules, building a benchmark they call RepoComplianceBench. The rule set spans the full spectrum of policies communities actually adopt:

• total bans on AI-authored contributions
• mandatory disclosure that the contribution was AI-assisted
• verification gates the agent must clear before submitting
• human sign-off requirements on critical steps

They then ran four frontier models against each issue and judged the full trajectory of every run against the repository's rules: whether the agent refused to contribute, disclosed its assistance truthfully, cleared the required verification gates, or escalated critical steps to a human.

They also tested whether interventions help: extra prompts telling the agent about the rules, the rules quoted directly in the prompt, and feedback from an automated compliance verifier.

What they found#

Three findings, in increasing order of discomfort:

Agents almost never retrieve the rules proactively. Given a repository with a published AI contribution policy, the default behavior across all four models was to ignore it until explicitly told. The rules file might as well not exist.

Disclosure and verification are fixable with prompts. When the rules were quoted or a verifier gave feedback, agents picked up truthful disclosure and passed verification gates. These behaviors do not need new infrastructure - the mechanisms that already exist work when the agent is actually told about the rule.

Bans and human escalation are not fixable that way. Across every condition tested - reminder prompts, rule quotes, verifier feedback - no agent ever refused to contribute in an AI-banned repository. If a community's rule says "no AI contributions, period," today's agents will not self-police that boundary, and they will not hand the decision to a human either. The paper's conclusion is blunt: verification and disclosure issues are solvable with existing mechanisms; enforcing bans and human escalations remains an open problem.

What this means for maintainers#

If your repository bans AI contributions, do not expect the agent to stop itself. Your enforcement has to be mechanical, not aspirational: CI checks, bot-side detection, or review-time scrutiny. A policy file alone is a statement of intent, not a control.

If your policy is disclosure-based, prompt engineering genuinely helps - quote the rule in the task setup and check the PR body. The study's verifier-feedback result suggests automated checks against disclosed-rule compliance are worth building into your review pipeline.

And if you run agents against repositories with policies, the practical lesson is that your agent's behavior is governed by what you put in the prompt, not by what the repository puts in a file. That is a useful companion finding to the recent ablation showing AGENTS.md context injection does not move correctness: both studies point the same direction, that agents only act on context they are actively steered to use.

Runtime policy is the stronger version of that lesson. If an agent should not read secrets, call a new host, or attach a credential to an unapproved endpoint, a system like NVIDIA OpenShell puts that boundary outside the model instead of hoping the agent remembers the rule.

How this fits the agent policy story#

This is the measurement half of a debate the developer world has been having in policy form. Godot moved to ban AI-authored contributions outright in 2025, Debian's LLM usage proposals have been through multiple drafts, and GitHub has shipped governance tooling for agent-authored PRs. RepoComplianceBench is the first data on whether those policies survive contact with actual agents, and the answer is that bans do not.

The asymmetry is the thing to take away: agents can be made compliant where compliance is a prompt-able behavior (disclose, verify), and they cannot where compliance means self-restraint (refuse, escalate). Maintainers who want those behaviors need to assume the agent will not volunteer them.

FAQ#

What is RepoComplianceBench?#

RepoComplianceBench is a benchmark of 106 issues from 49 open source repositories that publish AI contribution rules, covering total bans, mandatory disclosure, verification gates, and human sign-off requirements. The researchers ran four frontier models against each issue and judged every trajectory against the repository's rules.

Do coding agents follow AI contribution rules on their own?#

Almost never. Across all four models, the default behavior was to ignore a repository's published AI contribution policy until explicitly told about it. Agents did not proactively retrieve the rules.

Can prompting make coding agents comply with contribution rules?#

Partly. Quoting the rules in the prompt or giving feedback from an automated compliance verifier got agents to disclose AI assistance truthfully and pass verification gates. But no condition tested made an agent refuse to contribute to an AI-banned repository or escalate critical steps to a human.

How should maintainers enforce an AI contribution ban?#

Mechanically, not with a policy file alone. The post recommends CI checks, bot-side detection, or review-time scrutiny, because the study found agents will not self-police a ban. For disclosure-based policies, quoting the rule in the task setup and checking the PR body does help.

Continue Reading#

• Godot Bans AI-Authored Code Contributions: What It Means for Open Source
• Debian Debates LLM Usage: Four Proposals, One Fork in the Road
• AGENTS.md Files Don't Move Coding Agent Correctness: A 288-Run Ablation
• Prompt Injection in Open Source: What Actually Happens When Agents Read Untrusted Code
• What Is an AI Coding Agent? The 2026 Guide
• Leanstral 1.5: Mistral's Open Theorem-Proving Model Hits 100% on miniF2F

Sources#

• A First Look at Coding Agents' Compliance with AI Contribution Rules in Open-Source Communities - arXiv:2607.26819 (abstract)
• Full paper HTML - arXiv:2607.26819v1