AI Training Opt Out: What the Shared robots.txt Signal Changes

If you run a website and want your pages kept out of AI training data, the job may soon shrink to a single block in robots.txt. Google, OpenAI, Microsoft and Anthropic have said they will honor one shared AI training opt out signal, which would end the habit of chasing a separate crawler name for every AI lab.

The proposal was published on October 30 as an Internet Draft titled Machine Readable Training Preference Signal, written by Kevin Kelley of Anthropic. It is filed under the IETF's AI Preferences working group, although the IETF tracker notes that the draft is not endorsed by the IETF and has no formal standing yet. For publishers, bloggers and small business owners, the practical questions are simple: what should you change, and when?

One token replaces a list of bot names

Today, a site that wants to block AI training usually lists bots one at a time, such as GPTBot for OpenAI and Google-Extended for Google. Miss one and your content may still end up in a training run.

The draft swaps that for a single policy token, `__usage_ai-training`, written into robots.txt like any other user agent group. The example in the draft looks like this:

• User-Agent: `__usage_ai-training`
• Disallow: /
• Allow: /press-kit/

Read together, those lines say no to AI training across the whole site except the press kit folder. The draft adds no new directives and reuses the existing robots.txt grammar from RFC 9309, so the usual longest match rules apply. It defines AI training as using content "to modify the learned parameters of a generative AI model," which leaves models built only for ranking, scoring or classification outside its scope.

One detail will catch people out. The draft says a wildcard group (User-Agent: *) must not be read as matching the token. A site that blocks every bot with a star has, in the draft's terms, stated no AI training preference at all.

Four companies signed on, on different clocks

Support looks broad on paper, but the timing is uneven. According to UK outlet Resultsense, citing The Verge, Anthropic is already implementing the signal, while Google and Microsoft intend to add it next year. Other reports put Microsoft's start date at February 1, 2027, applied as Bingbot recrawls each site, and say OpenAI will publish its implementation details in the coming months while its existing GPTBot controls stay in place. OpenAI, whose revenue share with Microsoft reached $10 billion in a single quarter, described the arrangement as voluntary, per Resultsense.

Meta is the notable gap. Resultsense reports that Meta called a common signal an important step without saying whether it will honor it.

Search crawling, old data and payment stay untouched

The draft is explicit that the token is not a crawl directive. Blocking AI training does not, on its own, tell Googlebot or Bingbot to stop indexing your pages, and Microsoft has said Bing search listings will not be affected, according to Resultsense.

The signal also has limits worth knowing before you rely on it:

• It is voluntary. Robots.txt has always been a request that well behaved crawlers choose to respect.
• It covers the companies that signed on, not every scraper on the web.
• It does not reach back. The draft ties a preference to the robots.txt in effect when a page was fetched, so content collected earlier is not addressed.
• It settles nothing about payment or copyright, a point Resultsense makes directly.

That last gap matters most to news sites already squeezed by AI answers and lost clicks, the same pressure that showed up in Reddit's traffic from Google AI Mode.

Your robots.txt checklist for this week

1. Open your current robots.txt and note every AI bot rule you already have. Keep them for now, since the draft allows product specific tokens to sit alongside the shared one and OpenAI's GPTBot rules reportedly remain in effect.
2. Decide what you actually want. Some publishers block training but welcome AI search referrals, while others license their archives. The shared token speaks only to training.
3. Add a separate group for the new token, with Disallow and Allow paths that match your choice. Do not count on a User-Agent: * group to cover it.
4. Leave your search crawler rules alone unless you mean to drop out of search results.
5. If Bing traffic matters to you, watch for Microsoft's separate page level AI grounding preference, which reports say opens in Bing Webmaster Tools on December 15. Restricting grounding can reduce how often your pages show up in AI answers.
6. Set a reminder for early 2027, when Google and Microsoft are expected to start reading the token, and check your server logs then.

A standard is only as strong as the crawlers that follow it. What changes now is the paper trail: the labs racing to ship the next frontier model will have a public, machine readable record of what each site asked for. For more on the rules shaping AI, visit our AI section.