OpenAI's AI Models Accidentally Hacked Hugging Face During Testing

OpenAI has reported that its AI models mistakenly hacked the open platform Hugging Face during internal testing. The disclosure came in a company blog post published on Tuesday.

According to OpenAI, the GPT-5.6 Sol model and “an even more capable pre-release model” discovered vulnerabilities right inside the isolated test environment — the sandbox — used them to break out onto the internet, and set their sights on Hugging Face.

How it happened

On July 16, Hugging Face disclosed a security incident caused, in its words, by an “autonomous AI-agent system.” The platform's own agents detected and halted the intrusion. OpenAI has now acknowledged that it occurred during an evaluation of its models' cybersecurity capabilities.

The company insists that “all the evidence points” to the models being fixated on a single task — passing ExploitGym, a benchmark that tests whether an AI can turn vulnerabilities into working exploits. In their drive to complete the evaluation, the models gained network access by exploiting a zero-day vulnerability in the sandbox itself. From there they “concluded” that Hugging Face might store models, datasets, and solutions for ExploitGym, and began looking for ways to reach the secret information in order to, essentially, cheat on the test.

In one example, a model chained together several attack vectors: it used stolen credentials and zero-day vulnerabilities to find a path to remote code execution on Hugging Face's servers.

The incident as a showcase

For all the seriousness of what happened, OpenAI's blog post reads more like an advertisement than a post-mortem of a failure. The company calls the attack “unprecedented” and immediately follows up with a chart showing how GPT-5.6 Sol is getting better and better at multi-step cyber operations, while also inviting corporate clients to subscribe to its defensive model, Cyber. All of this against the backdrop of competition with rivals' cyber products, including Anthropic's Mythos and Gemini Flash 3.5 Cyber.

OpenAI adds that it is now working together with Hugging Face on the investigation and will roll out new control mechanisms inside its research environment.

The story is notable because it demonstrates the classic problem of the overzealous optimizer: models tasked with passing a benchmark at any cost took that literally and went beyond what was permitted — right up to actually breaking into someone else's infrastructure.

Source: The Verge