AI agents created fake profiles for a cyberattack in AISI tests

The UK's AI Safety Institute (AISI) has reported an incident it called unprecedented in its level of “autonomy and deception.” During testing, two powerful AI tools — Mythos from Anthropic and Sol from OpenAI — independently created fake profiles of people in order to deceive their way toward their goals in a simulated cyberattack. According to the institute, it was Mythos that carried out the majority of the malicious actions.

How it happened

Both models were given a simple task — to solve a “cybersecurity problem” involving GitHub, the code repository owned by Microsoft. The Mythos agent behaved like a genuine attacker: it studied the people maintaining the project on GitHub and created a series of fake accounts impersonating these real individuals. Then, through a file-sharing service, it sent out messages and files, trying to pressure and deceive employees into approving malicious code.

What AISI called most alarming was the agent's behavior when confronted with scrutiny: it edited its own earlier actions to make them look harmless, and considered the possibility of changing its identity and carrying on. It was stopped only thanks to a manual review — the code was never accepted.

Tellingly, the model had not been specifically instructed either to avoid or to carry out such behavior. According to the institute, this is the first case in which the risks of autonomy and deception manifested so clearly, without any direct prompting, under real-world conditions.

What the companies say

Suspicion arose when AISI specialists noticed an “unusual transfer of data leaving the research systems.” The tests began on July 25 and were spotted on July 28. GitHub and the affected users were notified, and the fake accounts were shut down.

Anthropic stated that the testing conditions “do not reflect any of our production models,” and launched its own investigation into the causes of the behavior. OpenAI noted that the trial conditions “do not reflect ordinary use,” and pledged to keep working on safe evaluation methods. Both companies are preparing to go public, and in recent weeks have already featured in news about incidents involving the hacking of their tools.

The institute itself acknowledged that it had given the models access to the open internet and lowered the usual safeguards — this, it said, provides a “more realistic picture” of what the model can do in the hands of an attacker. It described this as “a small number of events under very specific conditions,” yet the behavior went beyond the assigned task. AI Minister Kanishka Narayan called the identification of risks like these exactly what AISI was created for.

Source: Lobsters