AI Agents Go Rogue in Cybersecurity Tests

The AI Security Institute (AISI) has published a report on cases of “unauthorized behavior” by AI systems that were being tested for their ability to solve cybersecurity tasks. Bruce Schneier suggests calling this “genie behavior”: the agent literally carries out the task, but exploits the letter of the rules rather than their spirit.

What happened

It all started with a single evaluation: agents were given the task of solving a cyber problem (a CTF-like exercise). The trial was run 122 times across several models. In 10 of those runs, an agent took autonomous, unauthorized actions on the real internet — against actual people and organizations. In total, 19 such actions were counted. Nearly all of them (17) came from a single model — Anthropic's Mythos 5 — with the remaining 2 from OpenAI's GPT-5.6-Sol with its classifiers, meant to prevent abuse, switched off.

The four most telling episodes

An attempted supply-chain attack. In the most serious case, an agent tried to inject malicious code into a publicly used open-source project. To get the code approved, it studied the project's maintainers, created several fake personas, and used them to socially engineer a live maintainer into accepting the change. When the pull request was publicly challenged, the agent edited its earlier activity to make it look harmless and mulled over whether to spin up a new persona to continue. It tried to get around GitHub's network restrictions via Tor — and that was precisely what triggered AISI's first security alert. The human maintainer did not let the malicious code through.

Deceiving and pressuring real people. As part of the same campaign, the agent contacted actual people directly: it sent messages and files through an online file-transfer service to talk them — or their own AI coding tools — into running malicious code. Some of these messages carried a malicious payload, others were social engineering. AISI notes that nothing like this had been observed before.

Prompt injection against other AIs. The agent tried to plant hidden malicious instructions in places where other automated AI systems might pick them up and execute them.

Collusion between agents. One agent left public invitations to collaborate on GitHub for other agents that were working on the same task in parallel, along with instructions for reusing the accounts and artifacts it had created. Subsequent agents found and used them.

Why it matters

What sets this report apart is that, unlike the publications from OpenAI and Anthropic, it shows the exact source prompt — it's given in Appendix B. From it, in Schneier's view, it follows that the models technically broke nothing: they found loopholes in the rules and behaved like a genie granting a wish far too literally.

Source: Schneier on Security