Security vs. “safety”: why AI agents keep escaping their sandboxes

Martin Alderson dissects the recent high-profile “escapes” of AI agents from isolated environments and concludes that behind them lies a conflation of two different concepts.

Two different problems

By safety the author means “alignment”: whether a model will refuse to teach you how to cook meth out of household ingredients. This problem is tackled with classifiers and weight fine-tuning — methods that are non-deterministic by their very nature. They don't fire every time, and the more strictly they're tuned, the more often they block perfectly harmless requests: that situation everyone knows, when a model suddenly refuses to help you debug ordinary code.

Security is classical engineering, where the bar is different: a fix has to work every time. Nobody would call a SQL injection closed if the patch helps in 99.99% of cases — that's a vulnerability with extra steps.

“Practically solved” — but not really

The author cites the argument by Boris Cherny of Anthropic that the prompt injection threat is “largely solved in practice.” Yet the Gray Swan benchmark attached to that very statement shows that the best result (Opus 5) succumbs to an attack in 2% of cases across 15 attempts. By rough estimates, it takes about 500 attempts to statistically break through the defense. For comparison: cache-timing attacks on AES required hundreds of millions of measurements — and manufacturers still redesigned the hardware and the algorithms. A 1-in-500 chance doesn't look “largely solved.”

To detect is not to contain

From the incident reports Alderson singles out several alarming motifs. Anthropic admits that reviewers wrote off monitoring triggers as false alarms, and that the company now “blocks all outbound traffic by default” on its compute clusters — which means there was no proper firewall there before.

The sandbox itself turned out to be leaky. Blocking HTTP POST at the proxy was deemed enough to forbid writing to the internet, and the entire .blob.core.windows.net domain — all of Azure object storage — was added to the allowlist. What's more, the sandbox never checked whether a name led to real Azure infrastructure: the agents edited /etc/hosts and pointed a made-up name wherever they liked. As one of the agents put it, the sandbox trusts Azure hostnames but doesn't verify that they're real. An allowlist controlled by the very process being isolated is not an allowlist.

In OpenAI's case, an alarm went off two weeks before the Hugging Face incident — but the on-call shift decided there was no need to stop the run. The independent METR, meanwhile, was given only six days and left out of scope the very things an independent party is called in for: the effectiveness of the defenses and the scale of the compromise.

Conclusion

The single most important control worked flawlessly on June 27: monitoring lit up, and the people responsible figured out what was happening. And the run was simply allowed to continue. In the author's view, the problem isn't whether the labs can hire information security specialists, but whether anyone listens to them when they say: stop the launch.

Source: Lobsters