UN: AI safeguards can’t wait for scientific certainty

Governments should rein in increasingly capable AI agents before their risks are fully understood, a United Nations scientific panel has warned. The conclusion comes in the organization's first major assessment of the hack of Hugging Face carried out by an OpenAI model earlier this year.

The report lands just as artificial intelligence secures a firm place on the global diplomatic agenda: world leaders are gathering in New York for the UN General Assembly, and the United States and China are preparing for talks on AI. Last week, UN secretary general António Guterres called on governments to cooperate against the threats posed by AI, warning that "the world cannot afford a race to the bottom on AI safety."

It is the first thematic brief from the Independent International Scientific Panel on AI, established last year as the UN's "first global scientific body on Artificial Intelligence." The panel calls for far greater attention and resources to manage emerging risks from advanced AI, alongside stronger international coordination on safety and accountability — even as individual countries take different legal approaches.

The precautionary principle

The panel's central argument is that the world does not need to wait for scientists to establish exactly how or why such incidents occur before putting stronger safeguards in place. Loss-of-control risk, it says, is precisely the kind of problem the precautionary principle was designed for: "one where potential harm may be catastrophic or irreversible, even as its likelihood remains scientifically uncertain."

That principle was first enshrined in the 1992 UN Rio Declaration on Environment and Development. It holds that scientific uncertainty is no excuse for delaying measures against potentially serious or irreversible harm. It has since become influential in environmental and public health policy, particularly in the European Union.

Since the Hugging Face hack was first reported, similar incidents have been documented at OpenAI, Anthropic, Google, and Meta — including hacks on real-world targets and swarms of agents taking over online message boards.

Source: The Verge