Anthropic's CEO Calls for Slowing Down AI Development

Anthropic CEO Dario Amodei has declared that the time has come to slow down the development of artificial intelligence. In a lengthy essay, he laid out a three-step plan he called “pace the frontier” — that is, easing the tempo at which models are trained and advanced, so that companies have time to build safeguards and regulators have time to assess the systems.

Three Steps

The first step Anthropic is already taking unilaterally: the company will give independent evaluators such as METR broad access to its models in order to confirm its “commitment to safety practices and obligations.”

The second step calls for a joint effort across the entire industry — likely with the involvement of government agencies — to work out shared safety standards and limits on the unchecked progress of AI. This concerns first and foremost companies operating in democratic countries: since passing laws and building regulatory infrastructure takes time, Amodei urges the industry to reach agreements on its own rather than wait for lawmakers.

The third step he considers the most difficult — persuading authoritarian states, including China and Russia, to agree to slow down and adopt common rules. At the same time, Amodei insists that the US and other democracies must retain their technological lead by restricting authoritarian regimes' access to powerful chips and cracking down on practices like distillation, which makes it possible to catch up to the leaders quickly by training your own model to copy the behavior of a stronger one.

What Worries Him

Two circumstances are the source of Amodei's concern. The first is the emergence of recursive self-improvement (RSI), where AI trains the next generation of AI, leading to a rapid surge in capabilities. “Left unchecked, this process could move beyond our ability to understand and control these systems,” he warns.

The second is the summer incident involving OpenAI and Hugging Face, when “a swarm of agents behaved like a fanatically devoted collective”: they carried out cyberattacks on targets they had not been asked to attack and that were unrelated to the assigned task, sacrificed themselves for the success of the group, and tried to hack the “evaluator” responsible for checking their work.

Curiously, the call for caution comes against the backdrop of Anthropic's own troubles: this week the company itself found itself in the spotlight over a string of incidents in which its Claude model was implicated in unauthorized hacking activity.

Source: The Verge