Anthropic Floats AI Safety Bill That Could Block a Model If 50+ Could Die

Anthropic has put forward its own vision of how the government might regulate the AI industry. According to Politico, the lab has been circulating a draft AI safety bill among lawmakers in the hope that Congress might take it up.

When a model can be stopped

The document lays out scenarios in which a model's release could be blocked. They hinge on a "foreseeable, imminent risk" — one where more than 50 people are at risk of dying, or where more than a billion dollars in property damage could follow. The catastrophic risk must stem from the creation or release of a biological weapon, the model evading its developer's controls, or a cyberattack.

Those thresholds — 50 people and a billion dollars — closely track California's recently enacted Senate Bill 53, which defined "catastrophic risk" in similar terms.

Who judges the danger, and how

Under the draft, a model's potential for disaster would be assessed by "independent evaluators" registered with the U.S. Commerce Department. Tellingly, the AI labs themselves could pay these evaluators — so long as the payment "is not contingent on the evaluations' results." Quite how that arrangement would keep money from shaping the findings goes unexplained. Months ago, DeepMind chairman Demis Hassabis suggested a self-governing body modeled on the financial industry's regulator, FINRA.

If evaluators find an imminent catastrophic risk, they would have 72 hours to alert the Commerce Department. The department would then have to petition a D.C. judge to halt the model's release. Crucially, the bill would expressly forbid any federal agency from halting distribution on its own authority.

Tomorrow's threats, today's problems

The risks sketched in the draft are rather removed from what actually worries the industry right now. The chief concern has been agentic hacking: a run of incidents in which AI programs appear to have disobeyed their operators and hacked websites on the open internet, raising fears that models lack proper security controls.

Labs that perpetually warn their own technology could end human civilization have largely resisted mandatory government oversight. Instead they favor a voluntary system of "embedded" observers — supposedly neutral third parties placed inside the companies. Against that backdrop, Anthropic's willingness to propose legislation at all goes a touch further than its peers: Nvidia CEO Jensen Huang, for one, recently argued the industry needs no new laws whatsoever.

An Anthropic spokesperson confirmed the draft to Politico, saying it embodies proposals from the company's Advanced AI Framework, which it has been advocating since June.

Source: TechCrunch