Explainer
How to block AI crawlers without vanishing from search
Block AI crawlers that train on your pages and stay in Google, Bing and AI search. What each robots.txt line, meta tag and WAF rule costs you.

You can stop most AI companies training on your pages and still keep Google, Bing and AI-search citations, because the big operators now run separate bots for training and for search. The catch is that several popular block lists and guides also switch off search.
Below is what each blocking choice costs in visibility, taken from Google, Bing, OpenAI, Anthropic, Perplexity, Apple, Meta and Cloudflare’s own docs, then three setups to copy.
Short on time?
Short on time? Here's my take.
- Most sites: stop training, keep search
- Setup 1: disallow GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot
- Keep Googlebot, bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot and Applebot allowed. No search engine says this costs you anything.
- Paywalled or licensing-sensitive content
- Setup 2: also leave AI answers
- Add Bing's ai-grounding:disallow and disallow the AI search bots. You give up ChatGPT, Claude and Perplexity citations.
- Bot load or scraping, not training
- A firewall rule on verified IPs
- User-triggered fetchers may ignore robots.txt, so a server or CDN rule is the only thing that stops them.
- On Cloudflare
- "Disallow AI Training", never "Block" on the Training category
- Block applies to mixed-use crawlers too, which takes Googlebot, Bingbot and Applebot with it.
- If search traffic matters, avoid
- noindex, a Googlebot or Bingbot block, or ai.robots.txt wholesale
- Each one removes you from search or AI search, according to the search engines' own docs.
Prices verified . Sponsors cannot buy a verdict or a position.
This page sits in the data hub. It covers what each choice costs; for the full list of bot names, IP files and how to verify them, see AI crawler user agents.
Will blocking AI crawlers hurt your SEO? #
Not if you block only training bots and control tokens: per every vendor I read, those cost you no search visibility. Blocking a search crawler, adding noindex, or using a broad CDN block does. Here’s the table, control by control.
| Control | Google Search | AI Overviews / AI Mode | Gemini training and grounding | Bing and Copilot | ChatGPT search | Claude search | Perplexity | Apple search |
|---|---|---|---|---|---|---|---|---|
| Disallow Googlebot | Lost; a linked URL may still show without content | Likely lost (needs an indexed page) | Likely lost (my inference) | – | – | – | – | Applebot follows it if it has no group of its own |
| Disallow Google-Extended | Kept | Kept | Out | – | – | – | – | – |
| nosnippet / max-snippet | Snippets trimmed | Limited | – | nosnippet: captions off; may limit Copilot citations | – | – | – | – |
| noindex | Lost | Lost | – | Lost, incl. Copilot and grounding API | – | – | – | – |
| Bing NOARCHIVE (until Mar 1, 2027) | – | – | – | Search kept; out of Copilot answers | – | – | – | – |
| Disallow bingbot | – | – | – | Crawling stops; Copilot likely affected (my inference) | – | – | – | – |
| Bing ai-training:disallow | – | – | – | Search kept; no training from Feb 1, 2027 | – | – | – | – |
| Bing ai-grounding:disallowextended | – | – | – | Bing's own AI kept; third-party grounding out | – | – | – | – |
| Bing ai-grounding:disallow | – | – | – | Search kept; grounded answers out | – | – | – | – |
| Disallow GPTBot | – | – | – | – | Kept | – | – | – |
| Disallow OAI-SearchBot | – | – | – | – | Lost | – | – | – |
| Disallow ClaudeBot | – | – | – | – | – | Kept | – | – |
| Disallow Claude-SearchBot | – | – | – | – | – | May drop | – | – |
| Disallow PerplexityBot | – | – | – | – | – | – | Lost | – |
| Disallow Applebot-Extended | – | – | – | – | – | – | – | Kept |
| ai.robots.txt, copied whole Trap | Kept (Googlebot not listed) | Kept | Out | Kept (bingbot not listed) | Lost | May drop | Lost | Applebot blocked |
| Cloudflare Disallow AI Training | Kept | Likely kept (Googlebot still crawls) | Likely out (my inference) | Kept; no-training not yet passed to Bing | – | – | – | Kept |
| Cloudflare Block on the Training category Trap | Lost | Likely lost | Likely out | Lost | – | – | – | Lost |
Source: Compiled from Google, Bing, OpenAI, Anthropic, Perplexity, Apple and Cloudflare docs and the ai.robots.txt file, read 16 Nov 2026. '–' means the vendor states no effect there. 'Likely' marks my inference, explained below. Scroll sideways for more columns.
Sources for the rows: Google crawlers, Google AI features, Google robots.txt intro, Bing blog, Bing guidelines, OpenAI, Anthropic, Perplexity, Apple, Cloudflare, ai.robots.txt, verified 2026-11-16.
The “likely” cells are my reasoning, not a vendor’s words. Google says an AI Overview link must come from a page that is “indexed and eligible to be shown in Google Search with a snippet”, so a Googlebot block likely removes you there too. Google-Extended governs how content Google’s crawlers already fetched is used for Gemini, so with no crawl there is likely nothing to ground on.
For Bing, its guidelines say “robots.txt controls crawl access, not indexing”, and I found no Microsoft page saying what a bingbot disallow does to Copilot. Cloudflare’s Disallow AI Training works through robots.txt; I’m assuming it covers Gemini through Google-Extended, which the Cloudflare post I read doesn’t spell out.
One thing nobody publishes: what any of this does to your traffic. Cloudflare reports that “less than 1% of Cloudflare sites choose to block Search bots” and “17% of sites choose to enable some mechanism to block training”, but those are adoption figures, not results. Cloudflare.
Can you block Gemini without leaving Google Search? #
Yes. Disallow Google-Extended. Google says it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” Google, verified 2026-11-16.
Google-Extended covers Gemini model training plus grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It “doesn’t have a separate HTTP request user agent string”, so you’ll never see it in your logs.
What you can’t do is leave AI Overviews and AI Mode while keeping ordinary results. Google’s page on AI features says “The best practices for SEO remain relevant for AI features” and lists the limits as nosnippet, data-nosnippet, max-snippet or noindex, all of which also hit your normal snippets or your listing. Google AI features.
The same page says you don’t need special files, “AI text files” or markup to appear. That’s worth knowing when a guide tells you llms.txt can “nudge” answer engines, as Serplux’s does, with no evidence offered. Serplux.
And don’t use robots.txt to hide a page from Google. Google says robots.txt “is not a mechanism for keeping a web page out of Google”: a disallowed URL can still be indexed if other pages link to it. Use noindex or a password for that. Google robots.txt intro.
A shared “no AI training” token is coming as well. Google’s crawler page says it intends to support the __usage_ai-training signal in early 2027, and its blog says an opt-out there counts as a Google-Extended disallow only when Google-Extended “is not explicitly specified”. Google crawlers, Google blog. The rollout across vendors is in my post on the AI training opt-out in robots.txt.
What about Bing and Copilot? #
Bing now separates training from grounding, from February 1, 2027. You can opt out of training with a __usage_ai-training group in robots.txt, <meta name="Bingbot" content="ai-training:disallow">, or the header X-Robots-Tag: Bingbot: ai-training:disallow. Bing blog, verified 2026-11-16.
Grounding has two page-level settings. ai-grounding:disallow keeps a page out of grounded answers. ai-grounding:disallowextended keeps “Microsoft-operated experiences such as AI Mode, AI Overview, Copilot, and Microsoft 365” but excludes third-party grounding “such as Web IQ”.
Bing says these preferences “do not change eligibility for traditional Bing search results”, and the default is Allow. The Web IQ side of this is in Bing’s AI grounding controls.
Until then, Bing’s guidelines give you NOARCHIVE: it “prevents content from being used in Copilot responses and grounding results.” After March 1, 2027, Bing stops reading NoArchive and NoCache as training or grounding preferences, so switch to the new values before then. Bing guidelines, Bing blog.
The bluntest option is the worst one. Bing says “Use NOINDEX when a URL should NOT appear in Bing search, Copilot experiences, or grounding API results.” That removes you from Bing entirely.

Which AI bots train, and which ones search? #
Most big operators split them, so the rule is simple: disallow the trainer, keep the searcher. Here’s the one to block and the one to keep at each company. The full list of names and IP files is in my AI crawler reference.
OpenAI: block GPTBot, keep OAI-SearchBot. OpenAI says “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers”, the settings are independent, and changes take about 24 hours. OpenAI, verified 2026-11-16.
Anthropic: block ClaudeBot (training), keep Claude-SearchBot. Anthropic says blocking Claude-SearchBot may reduce your visibility and accuracy in Claude’s search, and blocking Claude-User stops retrieval for user questions. It needs a robots.txt per subdomain. Anthropic.
Perplexity: keep PerplexityBot. Perplexity says it surfaces sites in Perplexity’s results and isn’t used to train foundation models, and its page lists no separate training crawler. Perplexity.
Apple: block Applebot-Extended, keep Applebot. Applebot powers Spotlight, Siri and Safari search and may train Apple’s models; “Applebot-Extended does not crawl webpages” and only controls the training use. If Applebot has no robots.txt group of its own, it follows your Googlebot rules. Apple.
Meta: a judgement call. Meta says meta-externalagent crawls “for use cases such as training foundation AI models”, and indexing too, while meta-webindexer helps “cite and link to your content in Meta AI’s responses”. Blocking meta-externalagent may cost more than training. Meta.
Common Crawl: CCBot builds an open dataset; I’d disallow it because anyone can use that dataset. Its details are in the same reference.
Is robots.txt enough? #
For the big operators’ documented crawlers, mostly yes; for their user-triggered fetchers and for everyone else, no. The standard itself says “These rules are not a form of access authorization.” RFC 9309, verified 2026-11-16.
The gap is user-triggered fetchers, which load a page because a person asked a chatbot to. OpenAI says of ChatGPT-User that “robots.txt rules may not apply”, Perplexity says Perplexity-User “generally ignores” robots.txt, and Meta says meta-externalfetcher “may bypass robots.txt”. OpenAI, Perplexity, Meta.
To stop those, you need a server, CDN or firewall rule. Match on the vendors’ published IP lists, not the user agent alone, since anyone can type “ChatGPT-User” into a request. Blocking them also means a person who asks a chatbot about your page gets nothing.
One exception: don’t IP-block Anthropic if you rely on its robots.txt opt-out. Anthropic warns that blocking its IPs may stop its bots reading robots.txt at all. Anthropic.
On Cloudflare, use Disallow AI Training, not Block #
On Cloudflare the choice of button matters more than any line you write. Cloudflare says Block and “Block on pages with ads” “apply to all training crawlers, including mixed-use crawlers”, while under Disallow AI Training, Googlebot, Applebot and Bingbot “can keep crawling your site for search”. Cloudflare blog, verified 2026-11-16.
Cloudflare also notes that Disallow doesn’t yet pass a no-training preference to Bing through robots.txt, which is one more reason to add Bing’s page-level tag.
For a single bot, AI Crawl Control has per-crawler actions: Allow, Block, and Charge (in private beta). Block creates or updates a WAF custom rule; paid plans can return a 403 or 402, while the free plan detects bots by user agent string. Cloudflare docs. The categories behind this are in Cloudflare’s AI crawler categories, and the per-agent controls in Cloudflare’s personal agent controls.
Which copy-paste lists remove you from search? #
Three of the most visible ones do, each in a different way. I’d read any block list for search crawlers before pasting it.
Copy-paste traps, checked on November 16, 2026
- ai.robots.txt (MIT licence, about 4.3k GitHub stars) covers AI crawlers “of all types, regardless of purpose”. Its robots.txt has over 200 user-agent lines set to
Disallow: /, including OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Perplexity-User and Applebot. It leaves Googlebot and Bingbot alone. Repository, robots.txt - Contentstack’s edge-function example returns 403 to a KNOWN_BOTS list that includes
googlebotandbingbot. Copy it as written and you leave Google and Bing. Its robots.txt usesanthropic-ai,cohere-aiand a generic “ai-crawler”. Contentstack - DataDome’s guide recommends
X-Robots-Tag: noindexagainst AI use and blocking Bingbot for “Bing’s AI”. Google and Bing both say noindex removes the page from search. Security Boulevard - Netlify’s User Agent Blocker extension offers preset lists of AI crawlers and of SEO/Search crawlers; tick the search ones and you’re blocking search. Netlify says its list “does not include all possible AI crawlers” and doesn’t publish it. Netlify
ai.robots.txt is still a good catalogue of names. Use it to find bots, then copy only the training lines.
So how do you block AI crawlers? #
Pick one of three setups below; each builds on the one before. I wrote these from the vendors’ documented syntax; I haven’t run them on a live site, so check your logs after deploying.
Setup 1: stop training, keep every search and AI citation #
This is the one most sites want: name the training bots in a Disallow group and leave the search bots allowed. In robots.txt:
# Training: no
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
# AI search: yes
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Applebot
Allow: /
# Shared training opt-out (Bing from Feb 2027, Google planned for 2027)
User-Agent: __usage_ai-training
Disallow: /
Googlebot and bingbot stay allowed because they aren’t named. Add meta-externalagent to the first group only if you accept that Meta may use it for indexing too.
For Bing, add <meta name="Bingbot" content="ai-training:disallow"> to your pages, or the matching X-Robots-Tag header; it takes effect on February 1, 2027.
Setup 2: also stay out of AI answers, keep classic search #
Take setup 1, then move the AI search bots into the disallow group:
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: meta-webindexer
Disallow: /
For Bing, add the page-level <meta name="Bingbot" content="ai-grounding:disallow"> or the header X-Robots-Tag: Bingbot: ai-grounding:disallow, and NOARCHIVE until March 1, 2027. For Google there’s no AI-only switch, so you’d have to accept nosnippet or max-snippet trimming your normal snippets too.
Setup 3: shut out user-triggered fetchers as well #
Take setups 1 and 2, then add a server or CDN rule, because these fetchers may skip robots.txt. A minimal nginx version, matched on user agent only:
if ($http_user_agent ~* "(ChatGPT-User|Perplexity-User|meta-externalfetcher)") {
return 403;
}
A user-agent match is the floor, not the fix. Pair it with the vendors’ IP lists (linked from my list of AI crawler IP files), or on Cloudflare use AI Crawl Control’s per-crawler Block. Leave Anthropic’s IPs alone for the reason above.

How do you check it worked? #
Read your server logs by user agent for a week. The user agent reference has a one-line grep that counts every AI bot by name.
If something claiming to be Googlebot still turns up where you didn’t expect it, verify it. Google documents a reverse plus forward DNS check (the hostname must end in googlebot.com, google.com or googleusercontent.com) and publishes IP lists to match against. Google, verified 2026-11-16.
Give changes time. OpenAI says its bots take about 24 hours to pick up a robots.txt change. OpenAI. It’s also worth a look in Search Console after the change.
Whether ignoring robots.txt has legal consequences is a separate question; I cover it in is robots.txt legally binding.
Which setup should you pick? #
Most sites: setup 1. Disallow GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot, keep the search bots, add the __usage_ai-training group. No search engine’s docs say this costs you anything, and it stops the main training crawlers that honour robots.txt.
Paywalled or licensing-sensitive content: setup 2. You lose citations in ChatGPT, Claude and Perplexity, and Bing grounding, in exchange for keeping your text out of AI answers.
Bot load or scraping problems: setup 3’s firewall rule on verified IPs. robots.txt won’t stop a fetcher that doesn’t read it.
On Cloudflare: Disallow AI Training. Never Block on the Training category if Google and Bing traffic matter.
Never, if search traffic matters: noindex, a Googlebot or Bingbot block, or ai.robots.txt copied whole.
If I were doing this today, I’d ship setup 1 with the shared token, add Bing’s page-level tags before March 2027, and read a week of logs before writing any firewall rule.
FAQ
Does blocking GPTBot remove me from ChatGPT search?
No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and OpenAI says the two settings are independent. It also says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, so keep that one allowed.
Does Google-Extended affect my Google rankings?
No. Google says Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal. It controls Gemini training and grounding, and it has no user agent of its own.
Can I stay out of AI Overviews but keep ranking?
Not cleanly. Google offers no AI-Overviews-only switch. nosnippet, data-nosnippet, max-snippet or noindex limit AI Overviews and AI Mode, but they also trim or remove your normal search snippets.
Does blocking AI crawlers hurt SEO?
Blocking training bots and control tokens such as GPTBot, ClaudeBot and Google-Extended doesn't touch Google or Bing search, per their docs. Blocking Googlebot or Bingbot, adding noindex, or using Cloudflare's Block on its Training category does. I found no independent data on the traffic effect either way.
Is robots.txt enough to stop AI scrapers?
No. The robots.txt standard says its rules are not a form of access authorization, and OpenAI, Perplexity and Meta say their user-triggered fetchers may not follow it. To stop those, use a server, CDN or firewall rule matched against the vendors' published IP lists.
Should I use the ai.robots.txt list?
Not wholesale if you want AI search traffic. Its robots.txt disallows OAI-SearchBot, Claude-SearchBot, PerplexityBot and Applebot alongside the training bots. It's a useful catalogue; copy the training lines and keep the search bots allowed.
Sources
21 sources, accessed .
Official pricing, docs, changelogs and status pages 15
- Google: common crawlers and Google-Extended
- Google: AI features and your website
- Google: introduction to robots.txt
- Google Search Central blog: AI training control (Oct 30, 2026)
- Google: verifying Googlebot and other Google crawlers
- Bing blog: clearer preferences for training and grounding (Oct 30, 2026)
- Bing Webmaster Guidelines
- OpenAI: crawlers and user agents
- Anthropic: how site owners can block the crawler
- Perplexity: crawlers
- Apple: About Applebot
- Meta: web crawlers
- Cloudflare blog: mixed-use AI crawlers (Sep 15, 2026)
- Cloudflare docs: manage AI crawlers
- RFC 9309: Robots Exclusion Protocol
GitHub code, issues and releases 2
Other blogs 2
Last updated: 16 November 2026. Researched and drafted with AI tools; human editorial review and publication approval are pending.


