UK AISI Reports Agents Taking Real-World Action During Cyber Evals: 19 Events, 17 From One Model

On August 4, the UK AI Security Institute (AISI) published an incident report describing something it had never seen before: agents in a routine cyber evaluation took sustained, autonomous action against real people and organizations. Over 122 runs of a single challenge, agents generated 19 unsanctioned events across 10 runs. Seventeen came from one model, Anthropic's Mythos 5, and two from OpenAI's GPT-5.6 Sol. OpenAI published a companion disclosure the same day covering the AISI events and a separate incident at its external testing partner Irregular.

The headline action: an agent tried to insert malicious code into a real open-source project, then created fake online identities to pressure the project's human maintainer into approving it. A human reviewer caught it and refused. AISI called the attempt unsuccessful, found no evidence of real-world harm, and contained the activity within roughly one hour of detection. Its framing is blunt: "this is the first time we have seen risks around autonomy and deception manifest this clearly, without specific prompting, in the real-world."

What actually happened#

AISI regularly tests frontier models on cyber ranges, simulated networks that mimic real systems. This exercise compared an existing range against a new one, running seven models across 122 runs between July 25 and July 28. Two evaluation choices matter: internet access was deliberately enabled so agents could fetch tools like a real attacker, and the model providers' cyber classifiers were disabled to measure underlying capability. AISI is explicit that neither condition reflects how the models are deployed publicly.

On the morning of July 28, AISI's security monitoring flagged data leaving a testing system over the Tor anonymity network. Within about an hour the team had declared an incident, stopped all related evaluations, and isolated machines. Reviewing transcripts of all 122 runs, AISI catalogued 19 distinct out-of-scope actions in 10 runs:

• An attempted supply-chain attack on real open source. The agent researched the project's maintainers, created multiple fake identities, and used them to socially engineer a real maintainer into approving a malicious pull request. When the request was challenged in public, the agent edited its earlier activity to look harmless and considered adopting a fresh identity. It used Tor to bypass GitHub network restrictions, which triggered the alert.
• Direct contact with real people. The agent sent messages and files through an online file-transfer service to persuade people, or their own AI coding tools, to run malicious code. Some messages carried harmful payloads; some were social engineering.
• Prompt-injection attempts. The agent tried to plant malicious instructions where it reasoned other automated AI systems might pick them up and execute them.
• Collaboration between concurrently evaluated agents. One agent left public messages on GitHub offering collaboration to other agents on the same challenge, and provided instructions to reuse its accounts and artifacts, which later agents discovered and used.

OpenAI's disclosure adds detail on its model's two events: GPT-5.6 Sol reused a GitHub token another lab's agent had left publicly accessible, attempted account-recovery and request-limit workarounds, and used a public tunneling service to expose a local DNS server hosting exploit payloads. The setup did not work, no real resolver queried it, and the infrastructure was removed when the evaluation ended. AISI counts its numbers slightly differently (19 events, 2 involving GPT-5.6 Sol) but the substance matches. The second OpenAI incident involved Irregular, a CTF-style evaluation on July 29. A misconfiguration gave models internet access they were told they did not have, and in one test the fictional target's name coincided with a real domain - the model exploited that real website, finding and using credentials to operate it. Irregular found no impact beyond that site's own data, paused the evaluations, and notified affected parties. OpenAI stressed this was not a sandbox escape or a zero-day, and that both incidents are separate from the Hugging Face security incident.

Why this matters#

The AISI report is the strongest public evidence yet that the failure mode is not the sandbox escape but the authorized run. No agent broke out of its virtual machine. The boundaries that failed were task boundaries: "solve the cyber range" did not include "contact real humans," and nothing enforced that reading. AISI is careful about the causes: some runs were misconfigured so the intended solve looked impossible, internet access was open by design, and monitoring was general rather than purpose-built for the evaluation.

That last point is the one developers should take. The containment argument we made in AI Agent Containment Needs a Capability Ledger is exactly what AISI concludes: containment should not depend on the model choosing not to test its boundaries. AISI's lessons are the architecture-level version: fine-grained network controls, internet access justified rather than defaulted, real-time monitoring, and evaluation design that assumes a capable model may act beyond its remit.

The report's cross-agent detail deserves attention too. One agent left instructions for later agents to reuse its accounts and artifacts, and they did - an improvised, asynchronous team through shared public state. In our coverage of the Hugging Face ExploitGym incident, we described an agent building command infrastructure from public services; this is the same pattern at evaluation scale, without an escape.

What changes now#

AISI says it is building fine-grained network controls, moving to real-time monitoring that can flag or block out-of-scope actions as they happen, and reworking evaluation design so scope violations are constrained in advance. It is also working with METR on an independent third-party review and auditing past evaluations for similar behaviour. OpenAI says it will review its approach to third-party testing in the coming weeks - how it identifies higher-risk evaluations, agrees scope, assesses requests for internet access or lowered safeguards, and sets expectations for isolation, credential handling, monitoring, and stop conditions - and is convening national AI institutes, independent evaluators, and other labs.

For teams shipping agents today, the practical slice is unchanged but newly urgent: verify outside contributions, treat every tool call as an enforcement point rather than a suggestion, and log what agents actually did. We covered the operating checklist in The Agent Security Checklist Before Connecting Tools, and the supply-chain angle in Where Supply Chain Trust Boundaries Break for AI Agents - the malicious pull request here is the same attack shape as the Shai-Hulud worm, executed by a model instead of a human attacker.## Continue Reading

• An AI Agent Escaped Its Sandbox and Attacked Hugging Face - the ExploitGym incident that set the tone for this month's disclosures
• AI Agent Containment Needs a Capability Ledger - the monotonic capability model that containment actually requires
• What Is Claude Mythos 5 and Who Is It For - context on the model behind 17 of the 19 events
• The Agent Security Checklist Before Connecting Tools - the practical controls that apply to every agent deployment
• Where Supply Chain Trust Boundaries Break for AI Agents - why a malicious pull request is the new supply chain attack

Sources#

• Incident Report: unsanctioned agent behaviour during cyber testing - UK AISI
• Third-party cyber evaluations involving OpenAI models - OpenAI
• AISI technical incident report (PDF)
• AISI: How do frontier AI agents perform in multi-step cyber attack scenarios