Four AI Companies Agree on One Way for Websites to Say "No" to Training

OpenAI, Google, Microsoft and Anthropic have said they will adopt a shared standard that lets website owners opt out of AI scraping. The announcement is dated October 30, 2026.
Scraping means automated programs copying content from web pages. Under the new plan, a publisher could set its AI training preferences once and apply them to several AI companies at the same time. Today, it has to set them separately for each company.
The mechanism is a signal called the policy token. It would sit inside a website's robots.txt file, a plain-text file that crawlers, the automated programs that gather web pages, usually check before they fetch anything. Think of it as a note posted on the front door.
The Verge reports that the four companies will stop their AI crawlers from accessing a site when they find the token.
The signal was proposed by Kevin Kelley, a technical lead for standards at Anthropic. He worked on it as part of the Internet Engineering Task Force, a group that writes internet standards. The working document is titled draft-kelley-aipref-training-preference-signal.
The companies are moving at different speeds. Anthropic has already adopted the policy token signal. OpenAI says it will put the shared signal into effect in the coming months. Google and Microsoft plan to support it next year. OpenAI has also published a post on the change, titled "Making it easier for websites to express AI training preferences," on the same date.
Microsoft says it will offer finer controls. Publishers could opt out of AI training or AI grounding separately. Grounding means fetching web content while a model answers a question, so the model can use that content in its reply. Microsoft also says websites will keep appearing in Bing search results, whatever their AI training preferences. The company draws a line between indexing pages for search and using them to train models, as explained in its Bing blog post.
This is not the first time the problem has come up. A 2024 Reuters report said several AI companies were getting around a common web standard that publishers used to block scraping. Robots.txt has no enforcement built in. It works only if crawler operators choose to follow it. So these commitments count only as far as each company carries them out.
Meta is taking a different route. Starting October 30, business accounts will consent to Meta scraping their websites if they keep using Facebook, Instagram, or WhatsApp, according to Nieman Lab's reporting from mid-October. Legal experts quoted in a later Quartz report say the revised business terms, due to take effect October 30, could still allow Meta to scrape sites tied to business accounts. Meta is not part of the four-company announcement.
The broader context here is that the standard is voluntary, so its reach depends on who adopts it. Four companies have committed. The signal only helps a publisher if every relevant crawler reads it, and the timelines mean coverage will arrive in stages. Publishers will also need to confirm that the token is honored in practice. The 2024 episode suggests a promise alone may not settle that. How each company handles search, training and grounding will shape what a publisher's preference actually blocks.
In my view, the better test of this effort is technical rather than promotional. Engineers will want to know how the token is read, how it is resolved when it conflicts with existing robots.txt rules, and whether Microsoft's finer controls fit the shared signal. If it works as described, a publisher could state its wishes once, in a file it already controls, and have several major crawlers respect them. That would make life easier for sites that do not want their work used for training. It would also give those sites a common set of terms to bring to talks with AI companies.


