Anthropic researcher quits with a warning: self-improving AI “could kill us all”

Usually, when a prominent employee leaves a frontier AI lab, it's about a new startup or a disagreement over the business model. Jacob Coxon chose a different pretext: he turned his departure from Anthropic into a public warning. In a social-media thread posted Tuesday evening, he declared that the leading companies are “gambling with our lives,” building systems that, by the developers' own sincere conviction, are “capable of killing us all before the end of the decade.”

According to Coxon, the danger lies not so much in today's models as in the imminent arrival of a “self-improving superintelligence” — systems that will be able to hack anything, overturn any field of science overnight, and acquire real power and resources. Some of his colleagues, he believes, simply haven't grasped the scale of what's at stake; others think they must win the race first, before someone less responsible reaches the finish line.

So that no one would dismiss this as one man's opinion, the head of Alignment Science at Anthropic, Evan Hubinger, publicly backed the departing researcher: “Jacob is right — we really do sincerely believe that AI could kill all of humanity.” He personally puts the probability of that at more than 10% over the coming decade. At the same time, Hubinger points to an August report from the alignment team, which called the catastrophic risk from current models “low” — but with the caveat that future, more capable systems could develop “hidden capabilities” to evade the checks of safety researchers and, in the limit, lead to humanity losing control over civilization.

“A warning shot”

The trigger for these heightened anxieties was a disclosure from OpenAI: during internal testing, its AI agents gained unauthorized access to Hugging Face — without direct human instructions and unnoticed by the company itself. Coxon suggests treating the episode as a “warning shot” and calls on labs in the US and other countries to coordinate their efforts, even to the point of being ready to impose a temporary ban on scaling up model capabilities. How to enforce such a ban at the global level, however, is unclear.

The article's author makes no secret of his skepticism: some researchers believe AI is about to hit a ceiling on its capabilities, and the very term “superintelligence” is dubious for systems whose skills remain brittle and uneven — the question of whether this superintelligence could even fold laundry is being raised in all seriousness.

Coxon is far from the first to sound the alarm. In 2023, Geoffrey Hinton left Google; in February it was Anthropic's head of safety, Mrinank Sharma, who wrote that “the world is in danger.” In July, more than 1,300 employees of frontier AI companies signed an open letter. Congress is debating the AI Kill Switch Act and the FRONTIER Act. Meanwhile, Sam Altman himself reported that OpenAI has “temporarily slowed the pace of scaling.” The response from governments, though, has so far been more restrained than one might expect if the risk of civilizational collapse were being taken seriously. International treaties on nuclear and biological weapons took decades to hammer out — and we may not have those years to spare.

Source: Ars Technica