Anthropic bans ‘cruel behavior’ toward Claude

For the first time in more than a year, Anthropic has overhauled the usage policy for its Claude chatbot. The trigger was a set of new, high-risk abuse scenarios: election interference, weapons development, surveillance, and uses in health and finance. But the clause that drew the most attention forbids “sustained and needless abusive or cruel behavior” toward the model itself.

Caring for ‘model welfare’

Back in August of last year, the company let Claude end conversations with “persistently harmful or abusive” users as part of its research into so-called “model welfare.” Terminating conversations remains the “primary enforcement mechanism.” Whether the company will go further — for instance, banning accounts — Anthropic declined to say.

The rule, the company stresses, is meant only for extreme cases, where users repeatedly act cruelly “with no discernible purpose.” Ordinary frustration, arguing with the bot, dark creative themes, and testing or research are not covered.

Propaganda, weapons and surveillance

The rest of the changes are more down to earth. Anthropic has gathered previously scattered restrictions into a single ban on deceptive commercial or political campaigns — a response to the rise of AI-generated content. It is now forbidden to obscure who is behind a message or to amplify content through fake accounts. A separate clause bars misinformation about candidates and how to vote, impersonating candidates or officials, and attempts to suppress turnout.

Weapons development was already off-limits, but the company ran into attempts to use Claude to craft guidance and control software. The ban now explicitly covers “software and components that make weapons work,” as well as arming drones and other autonomous vehicles.

Surveillance rules have been tightened too: Anthropic’s latest threat report found Claude increasingly used to identify and track “political dissidents.” Tracking people without consent is now prohibited, as is using Claude to decide whom to investigate or arrest, or to build surveillance tools. One caveat: the rules may be relaxed for certain government customers if Anthropic judges the safeguards adequate. The company has previously contracted with the US military.

A robot must obey its operator

There is also a clause on physical embodiment: when a model is connected to hardware that can act autonomously and might cause injury, a “qualified operator” must be able to watch the equipment and stop it. Whether that operator must be human the document does not say.

Over the past year Anthropic has repeatedly flirted with the idea that its models might possess some form of consciousness. The contrast with the rest of the industry is stark: Microsoft AI’s code of conduct flatly rejects the notion of “AI welfare” or granting models rights.

Source: The Verge