OpenAI, Google, Microsoft and Anthropic back a shared signal for opting out of AI scraping

OpenAI, Google, Microsoft and Anthropic say they will adopt a new standard that lets publishers opt out of having their content collected by AI crawlers. Search Engine Roundtable was first to report the news. The key change is that publishers will be able to set their AI training preferences once for several companies, rather than configuring them for each one individually.

How it works

The shared signal, called a "policy token," was proposed by Kevin Kelley, a technical lead for standards at Anthropic. The proposal was put forward within the Internet Engineering Task Force (IETF). The token will live in a site's robots.txt file, which has long been used to indicate which crawlers may access a site, whether for search indexing, archiving or AI training. At present, site owners have to communicate their preferences to each AI company separately.

All four companies say their crawlers will stay away from a site if they encounter the shared token.

Microsoft's finer controls

Microsoft will go a step further and offer more granular settings. Publishers will be able to opt out of AI training and of AI grounding separately. Grounding is the practice of pointing a model to a verified source of information to reduce hallucinations. Microsoft notes that sites will keep appearing in Bing search results regardless of the preferences they choose.

Timeline

Anthropic already supports the new signal. Google and Microsoft plan to add support next year, and OpenAI says it will implement it in the "coming months."

Context

The announcement comes as publishers continue to contend with a steady stream of AI crawlers that collect their content and use it to train models. There have been earlier attempts to push back against unauthorized scraping. The Really Simple Licensing (RSL) standard lets publishers charge crawlers that visit their sites, and Cloudflare blocks AI crawlers by default.

Source: The Verge