Explainer
The robots.txt AI training opt-out: who honors it, and when?
Only Anthropic says it has started honoring the robots.txt AI training opt-out. Microsoft starts Feb 1, 2027; Google and OpenAI later. Here's what to do.

Since October 30, 2026, one robots.txt group is meant to say “don’t train AI on this site”: User-Agent: __usage_ai-training. Google, Microsoft, OpenAI and Anthropic all backed it, but as of November 16 only Anthropic says it has started honoring it. IETF draft, PPC Land.
If you build datasets or run a crawler, the decision is whether to parse it now. If you run a site, it’s whether to add it and what to keep. My answer to both is yes, with details below. This is research, not legal advice.
Short on time?
Short on time? Here's my pick
- You run a crawler or build training datasets
- Parse and honor __usage_ai-training now
- One extra group lookup and no new grammar. Log the robots.txt text with each fetch. The draft is version -00 and may change.
- You run a site
- Add the group, keep your GPTBot, ClaudeBot and Google-Extended lines
- Only Anthropic says it has started. Microsoft starts February 1, 2027, Google says early 2027, and OpenAI gives no date.
- You worry about Bing grounding or Web IQ
- Page-level ai-grounding tags
- Microsoft offers no site-wide robots.txt grounding preference. The new token covers training only.
Prices verified . Sponsors cannot buy a verdict or a position.
What does the new robots.txt line look like? #
A normal robots.txt group with a reserved name. The draft, “Machine-Readable Training Preference Signal” by Kevin Kelley of Anthropic, gives this example. IETF draft.
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
That’s two leading underscores, a single underscore between “usage” and “ai”, then a hyphen. The draft “introduces no new directive names and no new grammar”, so a parser that already reads robots.txt groups can read it.
Three rules matter. A wildcard group “MUST NOT be interpreted as matching the policy token”, so User-agent: * says nothing about training, and no group means no preference. Specific expressions “SHOULD take precedence over general ones”. And “Private agreements MAY override preferences”.
“Training” takes its meaning from the AIPREF vocabulary draft: “Using an asset to modify the learned parameters of a generative AI model”.
Is it an IETF standard yet? #
No, not yet. The datatracker lists it as an individual draft, version -00, with the status “I-D Exists”, under related drafts rather than the working group’s own documents. Those are aipref-vocab-08 (September 13, 2026) and aipref-attach-05 (August 18, 2026), both on the Proposed Standard track. Datatracker.
The draft expires May 3, 2027 unless revised.
Does the training opt-out block crawling? #
No. The draft says “This policy token is not a directive for crawling behavior”, and calls usage and traffic signals “independent and orthogonal”. Search crawlers keep crawling.
Microsoft says the training opt-out leaves Bing search eligibility unchanged, and that “An Allow setting does not override a robots.txt rule blocking Bingbot”. Bing.
If you want a crawler off your pages, you still need a normal Disallow for its user agent. Whether any of this binds anyone is a separate question; see is robots.txt legally binding?.
Which AI companies honor it, and from when? #
Only Anthropic says it has started. Microsoft has a firm date, Google says early 2027, and OpenAI gives no date. Here’s where each stands:
| Company | Status | Start date | Company-specific control that still works | Source |
|---|---|---|---|---|
| Anthropic | Recognition began, rolling out 'over time' | October 30, 2026 | Rules naming ClaudeBot take priority | Anthropic help page |
| Microsoft (Bing) | Will honor it while crawling | February 1, 2027 | Page-level ai-training meta tag or X-Robots-Tag, which overrides robots.txt | Bing blog |
| Intends to support it | 'early 2027' | Google-Extended; an explicit rule wins | Google crawler docs and blog | |
| OpenAI | Committed, per two news reports | 'the coming months', no date | GPTBot and existing robots.txt controls | Let's Data Science, PPC Land; OpenAI's crawler docs don't mention the token yet |
Source: Anthropic, Microsoft, Google and OpenAI pages and two news reports, read 16 Nov 2026. OpenAI's own announcement returned 403. Scroll sideways for more columns.
Anthropic says “As of October 30, 2026” it began recognizing a machine-readable training-preference signal, rolling out “across all of our services over time and globally”. Its page lists ClaudeBot, Claude-User and Claude-SearchBot and says rules naming ClaudeBot take priority, though it doesn’t print the token string. Anthropic.
A November 1 explainer said the page didn’t mention the signal; that’s out of date. SSD Nodes.
Microsoft gives the firmest date: “Starting February 1, 2027, Microsoft will honor the new preferences as it crawls”. That’s 77 days after November 16. Bing.
Google says it “intends to support the AIPREF shared machine-readable training preference signal” “in early 2027”. Its October 30 post says it aims to support the control “along with other AI providers in 2027”. Google docs, Google blog.
OpenAI committed to honoring a shared training signal and aims to implement it in “the coming months”, per Let’s Data Science’s rewrite of its post. PPC Land quotes the post saying the signal “will not replace the controls we already provide through robots.txt”. Let’s Data Science, PPC Land.
On November 16, OpenAI’s crawler docs described GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot with no mention of the token. OpenAI.
PPC Land also quotes Kelley: “It’ll take us some time to add support across our services”. Plan for months of partial coverage.

How does it interact with Google-Extended and Bing? #
Named rules win where a company says so. Google treats a token opt-out without an explicit Google-Extended rule as a Google-Extended disallow, so a site that allows Google-Extended but disallows the token is still allowing Google. Google blog.
Google-Extended covers Gemini training and grounding, including Gemini Apps and Vertex AI grounding, and “does not impact a site’s inclusion in Google Search”. Google docs. For Anthropic, a ClaudeBot rule beats the shared token.
Bing splits training from grounding, meaning using your page to back an AI answer. Training can be set site-wide with the token, or per page with <meta name="Bingbot" content="ai-training:disallow"> or an X-Robots-Tag header; the page-level setting overrides robots.txt. Grounding is page-level only, and Microsoft says there’s “no site-wide robots.txt grounding preference”. Bing.
The ai-grounding:disallowextended value keeps your pages in Microsoft’s own experiences but excludes third-party grounding such as Web IQ. Temporary grounding preferences in Bing Webmaster Tools open on December 15, 2026 for sites that used NoArchive or NoCache, which the new controls replace. March 1, 2027 is the deadline to publish grounding preferences, and the temporary ones expire September 30, 2027. Both training and grounding default to Allow.
What should a crawler or dataset builder do now? #
Parse it now. It costs one more group lookup, and the models you train next year will use pages you collect this year.
- Match the token exactly.
__usage_ai-training, underscores as written. Never letUser-agent: *stand in for it. - Let named groups win for companies that say so, and record both the token group and the group for your own crawler name.
- Store the robots.txt text with each fetch date. The preference is about use, so you’ll want to show what it said when you collected the page.
- Test your parser. SSD Nodes warns that stock robots.txt parsers may misread the token. I didn’t check that, but feeding the draft’s example through your parser takes a minute, whether it’s your own or one of the open-source LLM crawlers.
- Check your contracts. “Private agreements MAY override preferences”, so a licensed feed can say something different.
If you buy crawled data rather than crawl yourself, ask the vendor whether it reads the token. I found no statement from any crawl or scraping API vendor, though I didn’t search every one. My guides to web scraping for RAG and whether scraping for AI training is legal cover the wider picture.
How do you block AI training in robots.txt? #
Add the group and keep your named lines, since only Anthropic says it has started and company-specific rules keep working. Here’s a block that refuses training while leaving search crawlers alone:
# Shared AI training preference (IETF draft -00)
User-Agent: __usage_ai-training
Disallow: /
# Keep the company-specific lines you already have
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Search crawlers and everyone else
User-agent: *
Allow: /
One trade-off: Google-Extended also covers Gemini grounding, so that line keeps you out of Gemini Apps and Vertex AI grounding as well as training. If Bing grounding is your concern, use the page-level tags above. If your site sits behind Cloudflare, read Cloudflare’s AI crawler categories before you edit the file by hand.

What I would do #
If I operated a crawler that feeds model training, I’d ship token parsing this month: exact match, never folded into *, named crawler groups winning where the operator says so, and the robots.txt text stored with the crawl date.
If I owned a site, I’d add the group today, keep my GPTBot, ClaudeBot and Google-Extended lines, and recheck OpenAI’s and Google’s crawler docs in the first quarter of 2027. More on site-side data choices sits under the data pillar.
FAQ
Does __usage_ai-training block crawling?
No. The draft says the token is not a directive for crawling behavior. It states a preference about training use, so search crawlers such as Googlebot and Bingbot keep crawling as before.
Does User-agent: * count as an AI training opt-out?
No. The draft says a wildcard group must not be interpreted as matching the policy token. Without a __usage_ai-training group, a site has stated no preference through this signal.
When does Google start honoring the AI training token?
Google's crawler docs say early 2027. Until then, Google-Extended is the control Google documents. Once supported, a token opt-out with no explicit Google-Extended rule counts as a Google-Extended disallow.
Is __usage_ai-training an official IETF standard?
No. It is an individual Internet-Draft, version -00, posted October 30, 2026, with the datatracker status I-D Exists. It isn't a working-group document, and it expires May 3, 2027 unless revised.
Will the opt-out remove my content from models already trained?
Nothing in the sources I read says so. The draft defines a preference about using content for training, and Microsoft describes honoring it as it crawls from February 1, 2027.
Do I still need Google-Extended, GPTBot and ClaudeBot rules?
Yes, keep them. Only Anthropic says it has started honoring the shared token, company-specific controls keep working, and rules naming ClaudeBot or Google-Extended take priority over the shared token for those companies.
Sources
12 sources, accessed .
Official pricing, docs, changelogs and status pages 7
- IETF draft: Machine-Readable Training Preference Signal (-00, 2026-10-30)
- IETF datatracker: AIPREF working group documents
- Google Search Central blog: AI training control (2026-10-30)
- Google: common crawlers and Google-Extended
- Microsoft Bing: Clearer preferences for training and grounding (2026-10-30)
- Anthropic: robots.txt controls
- OpenAI: crawlers and user agents


