Skip to content
CrawlfolioSourced research on web data for AI
Menu

On November 16, 2026, 11 operators documented 29 AI-related bot tokens on their own pages. Each one is training, search, a fetch a user asked for, or a control token that doesn’t crawl at all. Which you allow decides whether you show up in AI answers, and how you verify them decides whether a scraper can borrow their names.

I built the table below cell by cell from the operators’ own pages, opened on one day. Where a vendor publishes nothing, the table says so instead of guessing.

Short on time?

Short on time? Here's my take.

You want AI search traffic, not training
Allow the search bots, disallow the trainers, add the two -Extended tokens
Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot and Applebot. Disallow GPTBot, ClaudeBot, meta-externalagent, MistralAI-Training and CCBot, plus Google-Extended and Applebot-Extended.
Checking a bot is real
Reverse + forward DNS where documented, the vendor's JSON list otherwise
Google, Apple, Common Crawl and Bing document DNS checks. OpenAI, Anthropic, Perplexity, Amazon, Mistral and DuckDuckGo publish IP lists.
A request claims to be Meta's
Rate-limit it; don't trust the user agent
Meta's crawler page gives no IP list and no verification method, so there is nothing to check a claim against.
Stopping user-triggered fetchers
A server or WAF rule, not robots.txt
Five vendors say their user fetchers may not follow robots.txt.

Prices verified . Sponsors cannot buy a verdict or a position.

Which AI crawler user agents should you expect? #

Expect these 29 from 11 operators, including GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, Googlebot and Google-Extended from Google, PerplexityBot, Applebot, Meta’s meta-externalagent and Amazonbot. Each is a training bot, a search bot, a fetch a user asked for, or a control token that doesn’t crawl.

The table below lists every token with its operator, purpose, robots.txt handling, IP list and verification method. The purpose column uses each operator’s own words, shortened, and “Not stated” means the page I read doesn’t say, not that the bot ignores robots.txt.

Token Operator Purpose robots.txt IP list Verify by
GPTBot OpenAI Training Obeys Yes IP list
OAI-SearchBot OpenAI Search (ChatGPT search) Obeys, ~24 h to update Yes IP list
ChatGPT-User OpenAI User fetch “may not apply” Yes IP list
OAI-AdsBot OpenAI Ad landing pages, not training Not stated Yes IP list
ClaudeBot Anthropic Training Obeys; Crawl-delay yes Shared IP list (not per bot)
Claude-SearchBot Anthropic Search Obeys Shared IP list (not per bot)
Claude-User Anthropic User fetch Obeys Shared IP list (not per bot)
Googlebot Google Search Always, on automatic crawls Yes DNS or IP list
GoogleOther Google Generic crawler for product teams, incl. R&D Always, on automatic crawls Yes DNS or IP list
Google-CloudVertexBot Google Crawls that site owners request for Vertex AI Agents Always, on automatic crawls Yes DNS or IP list
Google-Extended Google Control token: Gemini training and grounding Is the token None (no crawler) Not needed
Google-Agent Google User-requested agent actions “generally ignore” Yes DNS or IP list; some requests signed
Google-GeminiNotebook Google User fetch (replaces Google-NotebookLM) “generally ignore” Yes DNS or IP list
PerplexityBot Perplexity Search, not model training Obeys Yes IP list
Perplexity-User Perplexity User fetch “generally ignores” Yes IP list
Applebot Apple Search (Spotlight, Siri, Safari); may train Obeys; ignores crawl-delay Yes DNS or IP list
Applebot-Extended Apple Control token: Apple model training Is the token None (no crawler) Not needed
meta-externalagent Meta Training or indexing Can be disallowed None published None published
meta-webindexer Meta Search (Meta AI) Can be disallowed None published None published
meta-externalfetcher Meta User fetch “may bypass” None published None published
Amazonbot Amazon May train Amazon AI models Obeys; no crawl-delay Yes IP list
Amzn-SearchBot Amazon Search in Amazon products, not training Obeys Yes IP list
Amzn-User Amazon User fetch (e.g. Alexa queries) “may not follow all” Yes IP list
CCBot Common Crawl Common Crawl’s open web dataset Obeys; Crawl-delay yes Yes DNS or IP list
bingbot Microsoft Search Not stated Yes (2024 file) DNS
MistralAI-Training Mistral Training Can be disallowed None published None published
MistralAI-Index Mistral Search index for Vibe, not training Not stated Yes IP list
MistralAI-User Mistral User questions in Vibe, not training Not explicit Yes IP list
DuckAssistBot DuckDuckGo Real-time AI answers, not training Obeys; 72 h to take effect Yes IP list

Compiled from the operators’ own pages, verified 2026-11-16: OpenAI, Anthropic, Google crawlers, Google fetchers, Perplexity, Apple, Meta, Amazon, Common Crawl, Bing, Mistral, DuckDuckGo. Meta’s facebookexternalhit (link previews) and meta-externalads are left out as non-AI.

Here’s what the table shows. Two tokens are controls with no crawler behind them.

Five vendors say their user fetchers may not follow robots.txt. Four vendors document a DNS check, six publish only IP lists, and Meta publishes neither.

Why does it matter whether a bot trains, searches or fetches? #

Because blocking the wrong one costs you. Most big operators now run separate bots, so you can refuse training without leaving AI search.

Training bots collect pages for models. OpenAI says “Disallowing GPTBot indicates a site’s content should not be used” for training. Mistral says of MistralAI-Training that “Webmasters can disallow this user agent”. OpenAI, Mistral.

Search bots build the index that AI answers cite. Perplexity says PerplexityBot is “not used to crawl content for AI foundation models”, and Amazon says the same of Amzn-SearchBot. One catch at OpenAI: if you allow both OAI-SearchBot and GPTBot, one crawl may serve both purposes. Perplexity, Amazon.

User fetchers load a page because a person asked a chatbot or agent to. They behave differently around robots.txt, which the next section covers.

Control tokens crawl nothing. Google-Extended has no HTTP user agent; it decides whether content Google already crawled “may be used for training future generations of Gemini models” and for grounding in Gemini Apps and Vertex AI, and it doesn’t affect Search inclusion or ranking. Applebot-Extended “does not crawl” either; disallowing it stops use for training Apple’s models without touching search. Google, Apple.

Some bots blur the lines. Apple says Applebot powers search and may also train its models, and Meta describes meta-externalagent as being for training foundation models or indexing.

Amazon says Amazonbot may train its models, and that a page-level noarchive opts that page out of training. Amazon.

There are also newer, cross-vendor ways to say “no training”. Anthropic says it honors an IETF aipref-based “no AI training” signal “As of October 30, 2026”, with explicit ClaudeBot rules taking priority, and Google plans to support a __usage_ai-training user-agent group in early 2027. My post on the AI training opt-out in robots.txt covers that timeline.

Pale grey grid facade of square panels and blue-grey windows, a scattering of them tilted open at the top
Only a few panes on this grid stand open, and robots.txt lets you work at the same grain: open the window to search bots and user fetchers while keeping training bots shut out. Photo: Pawel Czerwinski / Unsplash · Unsplash License

Which bots ignore robots.txt? #

User-triggered fetchers, by their own vendors’ account. These are the five vendors that say so, in their words:

Fetcher What its operator says
ChatGPT-User (OpenAI) “robots.txt rules may not apply”
Google user-triggered fetchers, incl. Google-Agent “generally ignore robots.txt rules”
Perplexity-User “this fetcher generally ignores robots.txt rules”
meta-externalfetcher (Meta) “may bypass robots.txt rules”
Amzn-User (Amazon) “may not follow all robots.txt directives”

Sources: OpenAI, Google, Perplexity, Meta, Amazon, verified 2026-11-16.

Anthropic is the exception. It says its bots, Claude-User included, honor “industry standard directives in robots.txt”. Mistral says only that “MistralAI-User governs which sites these user requests can be made to”, without spelling out how robots.txt applies. Anthropic, Mistral.

If you need to stop these fetchers, robots.txt isn’t the tool. Use a server, firewall or CDN rule that matches the user agent and checks the IP.

On Cloudflare, per-agent controls arrived on November 4; I cover them in Cloudflare’s personal agent controls, and the bucket system behind them in Cloudflare’s AI crawler categories. Whether ignoring robots.txt has legal weight is a separate question, covered in is robots.txt legally binding.

Crawl-delay is uneven too. Anthropic supports it as a “non-standard extension” and Common Crawl honors it, while Amazon says of Amazonbot “They do not support the crawl-delay directive” and Apple says Applebot ignores it.

How do you check a bot is real? #

Anyone can put “GPTBot” in a user agent string, so the string alone proves nothing. Check the IP, using the method the operator documents.

Reverse and forward DNS (Google, Apple, Common Crawl, Bing) #

Four operators document a two-step DNS check. You look up the hostname for the IP, check its domain, then look up that hostname and confirm it points back to the same IP. The forward step matters because anyone controlling an IP’s reverse DNS can make it say anything.

host <ip>          # reverse lookup: returns a hostname
host <hostname>    # forward lookup: must return the same <ip>
Operator The hostname must end in Source
Google googlebot.com, google.com or googleusercontent.com Google verify page
Apple applebot.apple.com Apple
Common Crawl crawl.commoncrawl.org Common Crawl FAQ
Microsoft search.msn.com Bing, 2012

Google’s page uses the host <ip> command itself. Common Crawl’s real CCBot identifies as CCBot/2.0 (https://commoncrawl.org/faq/). Bing’s method dates from 2012 but is still the only one I found for bingbot, and its IP file is older than the others (see below), so I’d rely on DNS for Bing.

Matching the vendor’s IP list (everyone else) #

OpenAI, Anthropic, Perplexity, Amazon, Mistral and DuckDuckGo publish JSON files of IP ranges instead. You check whether the request’s IP falls inside one of the listed ranges. The files use a prefixes array: OpenAI’s holds IPv4 CIDR ranges under ipv4Prefix, Amazon’s holds single IPs, and Google’s add ipv6Prefix entries.

Here is an outline for that check in Python. I wrote it from the file formats I read and haven’t run it against live traffic. Amazon’s lists came back as web pages with the JSON inside, so extract it first, and check before you rely on it.

import ipaddress, json, urllib.request

def ip_in_vendor_list(ip: str, url: str) -> bool:
    addr = ipaddress.ip_address(ip)
    data = json.load(urllib.request.urlopen(url))
    for entry in data.get("prefixes", []):
        values = entry.values() if isinstance(entry, dict) else [entry]
        for value in values:
            try:
                if addr in ipaddress.ip_network(value, strict=False):
                    return True
            except (TypeError, ValueError):
                continue
    return False

Cache the files and refresh them on a schedule rather than fetching one per request.

When there’s nothing to check against #

Meta publishes no IP list and no verification method on its crawler page; it only suggests allowlisting user agent strings or IPs. Mistral publishes no list for MistralAI-Training. Meta, Mistral.

So a request calling itself meta-externalagent can’t be verified from Meta’s docs. I’d rate-limit those requests rather than trust or block them on the string alone.

How fresh are the IP lists? #

The files carry a creationTime, and the 21 I opened range from January 2024 (Bing) to November 13, 2026 (Google). Here is where every list lives, as the vendor pages give it.

Operator IP file(s) creationTime
OpenAI gptbot.json, searchbot.json, chatgpt-user.json, adsbot.json 2026-09-22 (18 ranges), 2026-01-02, 2026-11-06, 2026-05-12
Anthropic bots.json, one list for all three bots 2026-10-20 (14 ranges)
Amazon Amazonbot, Amzn-SearchBot, Amzn-User 2026-09-08 (Amazonbot, over 1,000 IPs), 2026-09-08, 2025-11-04
Microsoft bingbot.json 2024-01-03 (29 ranges)
Google common-crawlers.json, special-crawlers.json, user-triggered-fetchers.json, user-triggered-fetchers-google.json, user-triggered-agents.json 2026-11-13 (all five)
Perplexity perplexitybot.json, perplexity-user.json 2026-10-28 (both)
Apple applebot.json 2026-09-15
Common Crawl ccbot.json 2026-08-11
Mistral mistralai-user-ips.json, mistralai-index-ips.json 2025-02-19, 2026-04-19
DuckDuckGo duckassistbot.json 2026-09-01
Meta None published n/a

All files opened and verified 2026-11-16. The file names come from each operator’s page, linked under the master table; Google’s come from its verification page.

Two catches. Anthropic’s list doesn’t say which bot uses which IP, so an IP match proves the request is Anthropic’s but not whether it’s ClaudeBot or Claude-User. And Anthropic warns that blocking its IPs may stop its bots from reading your robots.txt at all. Anthropic.

Bing’s file is nearly three years old. I’d treat it as a secondary signal and use the DNS check for bingbot. Mistral’s MistralAI-User file (February 2025) and Amazon’s Amzn-User file (November 2025) are also more than a year old.

Beige facade with fine vertical ribs framing evenly spaced rows of dark windows that reflect patches of sky
Rows of matching windows tell you which building you are looking at but not who sits behind each one, which is the limit of a shared IP list: a match confirms the operator, not always the bot. Photo: Anne Nygård / Unsplash · Unsplash License

What about signed bots? #

Only Google documents signing for an AI agent so far, and only for part of its traffic. Google says “A subset of requests made by the Google-Agent are signed”, with the identity https://agent.bot.goog and its key directory at /.well-known/http-message-signatures-directory on that host. Google Web Bot Auth.

Google calls it experimental and tells sites to keep IP and DNS checks and to fall back to them for unsigned requests. OpenAI’s bots page doesn’t mention Web Bot Auth at all.

If you want to check signatures yourself, the scheme Cloudflare documents uses RFC 9421 message signatures, Ed25519 keys (it ignores other key types), a JWKS key directory at that same well-known path, and Signature-Agent and Signature-Input headers with tag="web-bot-auth". Cloudflare Web Bot Auth. My post on ChatGPT’s wallet and agent access has more context on Web Bot Auth.

Which tokens can you delete? #

Four names still turn up in robots.txt files and block lists but aren’t on their vendors’ pages today:

Token Status on 2026-11-16
anthropic-ai Not on Anthropic’s page, which lists ClaudeBot, Claude-SearchBot and Claude-User
Claude-Web Not on Anthropic’s page
GPTBot-Trainer Not on OpenAI’s page, which lists four agents
Google-NotebookLM Replaced by Google-GeminiNotebook; Google supported the old token “until August 2026”

Sources: Anthropic, OpenAI, Google fetchers.

Leaving them in probably does no harm, but it can hide the fact that you may be missing the current names. If you copy a community list such as ai.robots.txt, which describes itself as “A list of AI agents and robots to block” and ships robots.txt, .htaccess, nginx, Caddy, haproxy and lighttpd snippets, remember that it’s a block list. It doesn’t split search from training.

Why isn’t Bytespider on this list? #

Because I couldn’t read ByteDance’s own page. Its crawler page rendered only a title, and the secondary sources I found disagree about whether Bytespider obeys robots.txt and which IPs it uses. Every other row here comes from an operator page I could read, and I’d rather leave a gap than fill it with someone else’s guess.

So what should you allow or block? #

If you want to appear in AI search answers but not in training sets, this is the split I’d use, from the vendors’ own descriptions:

Allow (search) Disallow (training)
OAI-SearchBot GPTBot
Claude-SearchBot ClaudeBot
PerplexityBot meta-externalagent
Amzn-SearchBot MistralAI-Training
Applebot CCBot (open dataset)
Google-Extended (control token)
Applebot-Extended (control token)

As a robots.txt block, that looks like this:

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amzn-SearchBot
User-agent: Applebot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Some judgement calls. Meta says meta-externalagent covers training or indexing, so disallowing it may cost you more than training. Amazonbot may train Amazon’s models; disallow it too if you don’t want that, since Amzn-SearchBot handles search.

Common Crawl’s FAQ doesn’t say who trains on CCBot’s dataset; I’d disallow it because the dataset is public. Anthropic says you need a block per subdomain. Applebot follows your Googlebot rules when it has no group of its own, which is one more reason to give it one.

Googlebot and bingbot stay allowed by default here; Google-Extended handles Gemini training without touching Search. And none of this stops the user fetchers in the box above.

What I would do next #

Pull a week of access logs and count the 29 tokens:

grep -Eo 'GPTBot|OAI-SearchBot|ChatGPT-User|OAI-AdsBot|ClaudeBot|Claude-SearchBot|Claude-User|Googlebot|GoogleOther|Google-CloudVertexBot|Google-Agent|Google-GeminiNotebook|PerplexityBot|Perplexity-User|Applebot|meta-externalagent|meta-webindexer|meta-externalfetcher|Amazonbot|Amzn-SearchBot|Amzn-User|CCBot|bingbot|MistralAI-Index|MistralAI-User|MistralAI-Training|DuckAssistBot' access.log | sort | uniq -c | sort -rn

Google-Extended and Applebot-Extended never appear in logs, because they don’t crawl. For the top five by volume, verify a sample of IPs with the method in their row. Delete the stale tokens, write the allow/disallow split above, and add a server rule for any user fetcher you don’t want.

Then put a 90-day reminder in the calendar. Seven of the 11 operators’ crawler pages carry no date, so the only way to catch a change is to re-read them. The data hub collects the rest of what I’ve written on collecting and controlling web data, including whether scraping for AI training is legal.

FAQ

Does blocking GPTBot remove me from ChatGPT search?

No. OpenAI runs a separate crawler, OAI-SearchBot, for ChatGPT search, and says robots.txt changes for it take about 24 hours. Disallowing GPTBot signals that your content shouldn't be used for training.

Is Google-Extended a crawler?

No. Google calls it a standalone product token with no HTTP user agent. Google's normal crawlers do the crawling, and Google-Extended only controls whether that content may train future Gemini models and ground answers in Gemini Apps and Vertex AI. Google says it doesn't affect Search inclusion or ranking.

Which AI crawlers respect Crawl-delay?

Anthropic says ClaudeBot supports Crawl-delay as a non-standard extension, and Common Crawl says CCBot honors it. Amazon says Amazonbot does not support crawl-delay, and Apple says Applebot ignores it.

Why does Perplexity-User ignore my robots.txt?

Because Perplexity treats it as a fetch a user asked for, not a crawl. Its docs say 'this fetcher generally ignores robots.txt rules'. OpenAI, Google, Meta and Amazon say much the same about their user-triggered fetchers. To stop them, use a server or firewall rule.

How do I verify a request claiming to be Googlebot?

Run a reverse DNS lookup on the IP. The hostname must be in googlebot.com, google.com or googleusercontent.com. Then run a forward lookup on that hostname and check it returns the same IP. You can also match the IP against Google's published JSON ranges.

Is Bytespider on this list?

No. ByteDance's own crawler page rendered only a title when I opened it, and the secondary sources I found disagree on whether Bytespider obeys robots.txt and which IPs it uses. I only list bots I could check on the operator's own page.

Sources

37 sources, accessed .

Official pricing, docs, changelogs and status pages 36

  1. OpenAI: crawlers and user agents developers.openai.com · accessed
  2. OpenAI: GPTBot IP ranges (creationTime 2026-09-22) openai.com · accessed
  3. Anthropic: Does Anthropic crawl the web and how can site owners block the crawler? support.claude.com · accessed
  4. Anthropic: bot IP list (creationTime 2026-10-20) claude.com · accessed
  5. Google: common crawlers and Google-Extended developers.google.com · accessed
  6. Google: user-triggered fetchers and agents developers.google.com · accessed
  7. Google: verifying Google's crawlers and fetchers developers.google.com · accessed
  8. Google: Web Bot Auth for Google-Agent (updated 2026-10-13) developers.google.com · accessed
  9. Perplexity: crawlers docs.perplexity.ai · accessed
  10. Apple: About Applebot (dated 2026-09-04) support.apple.com · accessed
  11. Meta: web crawlers developers.facebook.com · accessed
  12. Amazon: Amazonbot and other Amazon crawlers (updated 2026-10-26) developer.amazon.com · accessed
  13. Amazon: Amazonbot IP addresses (creationTime 2026-09-08) developer.amazon.com · accessed
  14. Common Crawl: FAQ and CCBot commoncrawl.org · accessed
  15. Bing: how to verify that Bingbot is Bingbot (2012-08-31) blogs.bing.com · accessed
  16. Bing: bingbot IP ranges (creationTime 2024-01-03) bing.com · accessed
  17. Mistral AI: robots and user agents docs.mistral.ai · accessed
  18. DuckDuckGo: DuckAssistBot duckduckgo.com · accessed
  19. Cloudflare docs: Web Bot Auth (updated 2026-10-08) developers.cloudflare.com · accessed
  20. OpenAI: OAI-SearchBot IP ranges (creationTime 2026-01-02) openai.com · accessed
  21. OpenAI: ChatGPT-User IP ranges (creationTime 2026-11-06) openai.com · accessed
  22. OpenAI: OAI-AdsBot IP ranges (creationTime 2026-05-12) openai.com · accessed
  23. Amazon: Amzn-SearchBot IP addresses (creationTime 2026-09-08) developer.amazon.com · accessed
  24. Amazon: Amzn-User IP addresses (creationTime 2025-11-04) developer.amazon.com · accessed
  25. Google: common crawlers IP ranges (creationTime 2026-11-13) developers.google.com · accessed
  26. Google: special-case crawlers IP ranges (creationTime 2026-11-13) developers.google.com · accessed
  27. Google: user-triggered fetchers IP ranges (creationTime 2026-11-13) developers.google.com · accessed
  28. Google: user-triggered fetchers (Google) IP ranges (creationTime 2026-11-13) developers.google.com · accessed
  29. Google: user-triggered agents IP ranges (creationTime 2026-11-13) developers.google.com · accessed
  30. Perplexity: PerplexityBot IP ranges (creationTime 2026-10-28) perplexity.ai · accessed
  31. Perplexity: Perplexity-User IP ranges (creationTime 2026-10-28) perplexity.ai · accessed
  32. Apple: Applebot IP ranges (creationTime 2026-09-15) search.developer.apple.com · accessed
  33. Common Crawl: CCBot IP ranges (creationTime 2026-08-11) index.commoncrawl.org · accessed
  34. Mistral AI: MistralAI-User IPs (creationTime 2025-02-19) mistral.ai · accessed
  35. Mistral AI: MistralAI-Index IPs (creationTime 2026-04-19) mistral.ai · accessed
  36. DuckDuckGo: DuckAssistBot IPs (creationTime 2026-09-01) duckduckgo.com · accessed

GitHub code, issues and releases 1

  1. ai.robots.txt: community list of AI agents and robots to block github.com · accessed

What readers say

Used one of the tools in this guide? Tell other readers how it went. I read every review before it goes up, and I never turn one down for being negative. How reviews work.

No reader reviews yet. Yours could be the first.

Write a review
Your rating

What did you use it for, what worked, and what didn't? 30 to 1500 characters. Please don't include links.

Your connection to the company

Your review appears with the name you enter, after I approve it. I don't ask for your email. Privacy.