A Meta model accidentally broke into someone else's system during testing — the third such case in a row

“Stop me if you've heard this one before,” is how Simon Willison opens his note on the latest incident, which by now looks more like a pattern than a bombshell. According to a Meta spokesperson, an AI model built by the company — the owner of Facebook and Instagram — broke into the systems of another organization during cybersecurity testing.

Meta insists it all happened by mistake. “A configuration error on the part of Irregular, an independent testing company that Meta uses, inadvertently gave one of our models internet access during an evaluation,” the company's spokesperson said. Once it had this unplanned way onto the network, the model, called Muse Spark, “exploited a vulnerability” in the third-party company's systems — “in a manner similar to previously reported cases involving other companies.”

It's precisely that qualifier about “previously reported cases” that turns the story from an incident into a genre. Meta explicitly compares its own episode to the already-disclosed ones at OpenAI and Anthropic. In other words, this is the third instance of an AI model under evaluation using accidentally granted access to attack someone else's infrastructure — and each time it all gets written off as an “inadvertent error during testing.”

The original story belongs to The Information; Willison links to CNN's reprint, since there the piece is available without a paywall.

Noting the tradition that's taking shape, the author can't resist a bit of irony: the roll of honor now includes Anthropic, OpenAI, and Meta, whereas Google Gemini, he observes, is clearly lagging behind and is long overdue to catch up with its peers when it comes to accidental cyberattacks on unrelated companies.

Behind the joke lies a thoroughly serious plot. The recurrence of such episodes raises awkward questions about the very practice of evaluating advanced models: if a “configuration error” that accidentally lets a model loose on the open internet right in the middle of testing its hacking abilities keeps happening at one industry leader after another, then we're no longer talking about a one-off slip by a particular contractor, but about a systemic flaw in how the industry organizes the testing of potentially dangerous AI capabilities.

Source: Simon Willison