Four AI Companies Agree on a Shared Opt-Out Signal for Web Scraping
OpenAI, Google, Microsoft and Anthropic have agreed to adopt a shared standard that lets publishers opt out of AI scraping across all four companies at once. The announcement, dated October 30, 2026, would let a publisher set its AI training preferences once, rather than configuring each company separately.
The mechanism is a signal called the policy token, which would sit inside a website's robots.txt file, the plain-text file that crawlers conventionally read before fetching pages. The Verge reports that the four companies will restrict their AI crawlers from accessing a site when they encounter the token. The signal was proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of work in the Internet Engineering Task Force (IETF), the body that develops internet protocols. The working document is titled draft-kelley-aipref-training-preference-signal.
Implementation timelines differ by company. Anthropic has already adopted the policy token signal. OpenAI says it will implement the shared signal in the coming months, while Google and Microsoft plan to support it next year. OpenAI published a post on the change, titled 'Making it easier for websites to express AI training preferences,' on the same date.
Microsoft says it will offer more granular controls, letting publishers opt out of AI training or AI grounding separately. Grounding means retrieving web content at inference time, the moment a model answers a query, so the model can draw on that content. Microsoft also says websites will continue to appear in Bing search results regardless of their AI training preferences. The company distinguishes indexing a page for search from using its content to train models, as set out in its Bing blog post.
The shared approach follows a long-running problem. A 2024 Reuters report stated that multiple AI companies were circumventing a common web standard that publishers used to block scraping of their content. Robots.txt has no enforcement mechanism of its own. It depends on crawler operators choosing to honor it, so the commitments in this announcement carry weight only to the extent that each company implements them.
Meta is on a different track. Starting October 30, business accounts will consent to Meta scraping their websites if they continue using Facebook, Instagram, or WhatsApp, according to Nieman Lab's reporting from mid-October. Legal experts quoted in a later Quartz report say the revised business terms, set to take effect October 30, could still allow Meta to scrape sites tied to business accounts. Meta is not a party to the four-company announcement.
The broader context here is that the standard is voluntary, and its reach depends on who adopts it. Four companies have committed, but the signal's value to a publisher depends on every relevant crawler reading it, and the stated timelines mean coverage will arrive in stages. Publishers will also need to verify that a token set in robots.txt is honored in practice, a question the 2024 episode suggests a commitment alone cannot settle. How each company handles search indexing versus model training, and how grounding is treated, will determine what a publisher's preference actually blocks.
In my view, the more useful measure of this effort will be technical rather than rhetorical. Engineers will want to see how the token is parsed, how conflicting directives are resolved between the policy token and existing robots.txt rules, and whether Microsoft's granular controls map cleanly onto the shared signal. If the mechanism works as described, a publisher could state its preference once, in a file it already controls, and have that preference respected by several major crawlers. That would reduce operational friction for sites that do not want their content used for training, and it would give them a common vocabulary for negotiating access terms with AI companies.
