Anthropic Researchers: The Odds AI “Kills Everyone” Are Higher Than 10%

A leading AI safety specialist at Anthropic has said that the probability of artificial intelligence “killing everyone” by the end of the decade is greater than 10%. The admission came mere hours after a colleague resigned, declaring that Anthropic and its competitors are irresponsibly racing to build “superhuman systems” they won't be able to control.

Anthropic doesn't have a plan yet

The trigger was a post on X in which Jacob Coxon — a researcher who trained AI systems at Anthropic and previously worked on OpenAI's systems — announced his departure. He accused both companies of “barreling straight toward self-improving superintelligence and gambling with our lives,” even though “the people building AI genuinely believe it could wipe us all out by the end of the decade.”

Evan Hubinger, who heads one of Anthropic's safety teams, responded to this directly. He admitted to being worried about self-improving AI, noting that the process “is moving faster than we thought.” Hubinger agreed with Coxon's phrasing: “We really do genuinely think AI could kill all humans,” putting the probability of such an outcome “in the coming decade” at higher than one in ten.

And yet, by his own admission, Anthropic “doesn't have a plan yet” for how to keep advanced AI safe and aligned with human values, and the company is “clearly not on track” to develop one. Coxon, for his part, argues that the companies are “locked into a race” and keep pushing development forward “despite the risk.”

The industry has long warned about the danger of self-improving systems that could slip out of control in what's known as a recursive self-improvement loop. Although this scenario hasn't materialized yet, companies are actively pursuing it, and a significant share of today's AI code is already being written by AI itself.

A race against the backdrop of looming IPOs

Coxon's departure was one of the most notable for Anthropic — a company that, ironically, was founded by OpenAI alumni precisely out of safety concerns. In recent years, several researchers have cited safety issues to explain their exits from OpenAI.

This exchange reflects mounting anxiety within the industry itself about the pace at which ever more sophisticated models are advancing — especially as companies prepare for anticipated IPOs and in the wake of recent incidents involving agents that had “gone off the rails.”

Source: The Verge