For years, a website that wanted to keep its content out of AI training data had to write a separate rule for every company: one for OpenAI’s GPTBot, another for Anthropic’s ClaudeBot, another for Google-Extended, and more. On October 30, the four largest Western AI labs announced they will honor a single shared signal instead — one line in a site’s robots.txt file that says no to training across all four at once.
It is the first coordinated attempt by major AI companies to standardize how publishers express training preferences, and it arrives in the middle of a bruising fight over who gets to use the web’s content to train models.
How the shared token works
The mechanism is a policy token, __usage_ai-training, placed in robots.txt, the plain-text file that tells crawlers which pages they may visit. A site owner writes “User-Agent: __usage_ai-training” followed by “Disallow: /” to refuse training on the whole site, and can carve out exceptions with “Allow:” lines for specific paths, such as a press kit.
The proposal comes from an individual Internet-Draft titled “Machine-Readable Training Preference Signal,” authored by Anthropic standards lead Kevin Kelley and dated October 30, 2026, submitted to the Internet Engineering Task Force’s AI Preferences working group. It is an individual draft, not a formal standard: the IETF has not endorsed it, and the draft expires on May 3, 2027, unless it is updated.
Who is on board — and when
The four companies announced their support together, but their timelines are uneven, and none of them reads the signal everywhere today.
| Company | What it has said | Timing |
| Anthropic | Has begun implementing support | Under way, no completion date |
| OpenAI | Aims to implement the signal | In the coming months, no date |
| Microsoft | Will honor the preferences as it crawls | From February 1, 2027 |
| Plans to support the token | In 2027, with crawler documentation pointing to early 2027 |
Google’s John Mueller confirmed that if a site uses the shared token without explicitly naming Google-Extended, Google will interpret the directive as if the publisher had disallowed Google-Extended. Microsoft went further than the shared baseline, adding a separate control for AI grounding — the retrieval step that pulls live web content into chat answers rather than training model weights — and a page-level training setting that overrides the site-wide rule. Microsoft also said its older NoArchive and NoCache tags will stop expressing training or grounding preferences after March 1, 2027, and that opting out of training does not affect eligibility for regular Bing search results.
What the token does not do
The limits matter as much as the promise. The token controls training permission, not crawling behavior: as the draft states, it “is not a directive for crawling behavior,” so opting out does not prevent a crawler from visiting or caching a page. It does not block search indexing, and it does not keep a page out of AI-written answers that quote it.
It also does nothing retroactive. Updating robots.txt does not remove content already collected into existing datasets or trained into model weights. And robots.txt itself has no enforcement mechanism — it works only because crawler operators choose to honor it, so these commitments carry weight only to the extent each company implements them.
Why publishers care
The announcement follows years of tension between publishers and AI labs, including lawsuits over training data and unilateral moves by platforms to expand their own content use. A single shared signal lowers the cost of saying no, especially for smaller publishers without the legal teams to track four separate opt-out systems.
But the uneven timelines mean the practical advice for now is unchanged: publishers that want protection today should keep their existing company-specific rules in place alongside the new token, and verify later that the shared signal is actually being honored in practice.
What to watch next
Whether the draft gains formal standing at the IETF, whether other labs — notably Meta, which is on a separate track for its own content terms — join the shared signal, and whether publishers test compliance once implementation deadlines arrive. The token is a coordination mechanism, not a law, and its real test will be in how faithfully it is followed when it starts to bite.
Sources: OpenAI, Google Search Central and Microsoft Bing announcements, October 30, 2026; IETF Internet-Draft draft-kelley-aipref-training-preference-signal-00 by Kevin Kelley, October 30, 2026; The Verge reporting, October 30, 2026; Search Engine Land.
One thought on “One Line in robots.txt: Four AI Giants Agree on a Shared Opt-Out for Training Data”