Data study
AI crawler user agents, and how to check they're real
29 AI crawler user agents and tokens from 11 operators, checked on each vendor's page: purpose, robots.txt, IP list, how to verify and which to allow.

On November 16, 2026, 11 operators documented 29 AI-related bot tokens on their own pages. Each one is training, search, a fetch a user asked for, or a control token that doesn’t crawl at all. Which you allow decides whether you show up in AI answers, and how you verify them decides whether a scraper can borrow their names.
I built the table below cell by cell from the operators’ own pages, opened on one day. Where a vendor publishes nothing, the table says so instead of guessing.
Short on time?
Short on time? Here's my take.
- You want AI search traffic, not training
- Allow the search bots, disallow the trainers, add the two -Extended tokens
- Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot and Applebot. Disallow GPTBot, ClaudeBot, meta-externalagent, MistralAI-Training and CCBot, plus Google-Extended and Applebot-Extended.
- Checking a bot is real
- Reverse + forward DNS where documented, the vendor's JSON list otherwise
- Google, Apple, Common Crawl and Bing document DNS checks. OpenAI, Anthropic, Perplexity, Amazon, Mistral and DuckDuckGo publish IP lists.
- A request claims to be Meta's
- Rate-limit it; don't trust the user agent
- Meta's crawler page gives no IP list and no verification method, so there is nothing to check a claim against.
- Stopping user-triggered fetchers
- A server or WAF rule, not robots.txt
- Five vendors say their user fetchers may not follow robots.txt.
Prices verified . Sponsors cannot buy a verdict or a position.
Which AI crawler user agents should you expect? #
Expect these 29 from 11 operators, including GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, Googlebot and Google-Extended from Google, PerplexityBot, Applebot, Meta’s meta-externalagent and Amazonbot. Each is a training bot, a search bot, a fetch a user asked for, or a control token that doesn’t crawl.
The table below lists every token with its operator, purpose, robots.txt handling, IP list and verification method. The purpose column uses each operator’s own words, shortened, and “Not stated” means the page I read doesn’t say, not that the bot ignores robots.txt.
| Token | Operator | Purpose | robots.txt | IP list | Verify by |
|---|---|---|---|---|---|
| GPTBot | OpenAI | Training | Obeys | Yes | IP list |
| OAI-SearchBot | OpenAI | Search (ChatGPT search) | Obeys, ~24 h to update | Yes | IP list |
| ChatGPT-User | OpenAI | User fetch | “may not apply” | Yes | IP list |
| OAI-AdsBot | OpenAI | Ad landing pages, not training | Not stated | Yes | IP list |
| ClaudeBot | Anthropic | Training | Obeys; Crawl-delay yes | Shared | IP list (not per bot) |
| Claude-SearchBot | Anthropic | Search | Obeys | Shared | IP list (not per bot) |
| Claude-User | Anthropic | User fetch | Obeys | Shared | IP list (not per bot) |
| Googlebot | Search | Always, on automatic crawls | Yes | DNS or IP list | |
| GoogleOther | Generic crawler for product teams, incl. R&D | Always, on automatic crawls | Yes | DNS or IP list | |
| Google-CloudVertexBot | Crawls that site owners request for Vertex AI Agents | Always, on automatic crawls | Yes | DNS or IP list | |
| Google-Extended | Control token: Gemini training and grounding | Is the token | None (no crawler) | Not needed | |
| Google-Agent | User-requested agent actions | “generally ignore” | Yes | DNS or IP list; some requests signed | |
| Google-GeminiNotebook | User fetch (replaces Google-NotebookLM) | “generally ignore” | Yes | DNS or IP list | |
| PerplexityBot | Perplexity | Search, not model training | Obeys | Yes | IP list |
| Perplexity-User | Perplexity | User fetch | “generally ignores” | Yes | IP list |
| Applebot | Apple | Search (Spotlight, Siri, Safari); may train | Obeys; ignores crawl-delay | Yes | DNS or IP list |
| Applebot-Extended | Apple | Control token: Apple model training | Is the token | None (no crawler) | Not needed |
| meta-externalagent | Meta | Training or indexing | Can be disallowed | None published | None published |
| meta-webindexer | Meta | Search (Meta AI) | Can be disallowed | None published | None published |
| meta-externalfetcher | Meta | User fetch | “may bypass” | None published | None published |
| Amazonbot | Amazon | May train Amazon AI models | Obeys; no crawl-delay | Yes | IP list |
| Amzn-SearchBot | Amazon | Search in Amazon products, not training | Obeys | Yes | IP list |
| Amzn-User | Amazon | User fetch (e.g. Alexa queries) | “may not follow all” | Yes | IP list |
| CCBot | Common Crawl | Common Crawl’s open web dataset | Obeys; Crawl-delay yes | Yes | DNS or IP list |
| bingbot | Microsoft | Search | Not stated | Yes (2024 file) | DNS |
| MistralAI-Training | Mistral | Training | Can be disallowed | None published | None published |
| MistralAI-Index | Mistral | Search index for Vibe, not training | Not stated | Yes | IP list |
| MistralAI-User | Mistral | User questions in Vibe, not training | Not explicit | Yes | IP list |
| DuckAssistBot | DuckDuckGo | Real-time AI answers, not training | Obeys; 72 h to take effect | Yes | IP list |
Compiled from the operators’ own pages, verified 2026-11-16: OpenAI, Anthropic, Google crawlers, Google fetchers, Perplexity, Apple, Meta, Amazon, Common Crawl, Bing, Mistral, DuckDuckGo. Meta’s facebookexternalhit (link previews) and meta-externalads are left out as non-AI.
Here’s what the table shows. Two tokens are controls with no crawler behind them.
Five vendors say their user fetchers may not follow robots.txt. Four vendors document a DNS check, six publish only IP lists, and Meta publishes neither.
Why does it matter whether a bot trains, searches or fetches? #
Because blocking the wrong one costs you. Most big operators now run separate bots, so you can refuse training without leaving AI search.
Training bots collect pages for models. OpenAI says “Disallowing GPTBot indicates a site’s content should not be used” for training. Mistral says of MistralAI-Training that “Webmasters can disallow this user agent”. OpenAI, Mistral.
Search bots build the index that AI answers cite. Perplexity says PerplexityBot is “not used to crawl content for AI foundation models”, and Amazon says the same of Amzn-SearchBot. One catch at OpenAI: if you allow both OAI-SearchBot and GPTBot, one crawl may serve both purposes. Perplexity, Amazon.
User fetchers load a page because a person asked a chatbot or agent to. They behave differently around robots.txt, which the next section covers.
Control tokens crawl nothing. Google-Extended has no HTTP user agent; it decides whether content Google already crawled “may be used for training future generations of Gemini models” and for grounding in Gemini Apps and Vertex AI, and it doesn’t affect Search inclusion or ranking. Applebot-Extended “does not crawl” either; disallowing it stops use for training Apple’s models without touching search. Google, Apple.
Some bots blur the lines. Apple says Applebot powers search and may also train its models, and Meta describes meta-externalagent as being for training foundation models or indexing.
Amazon says Amazonbot may train its models, and that a page-level noarchive opts that page out of training. Amazon.
There are also newer, cross-vendor ways to say “no training”. Anthropic says it honors an IETF aipref-based “no AI training” signal “As of October 30, 2026”, with explicit ClaudeBot rules taking priority, and Google plans to support a __usage_ai-training user-agent group in early 2027. My post on the AI training opt-out in robots.txt covers that timeline.

Which bots ignore robots.txt? #
User-triggered fetchers, by their own vendors’ account. These are the five vendors that say so, in their words:
| Fetcher | What its operator says |
|---|---|
| ChatGPT-User (OpenAI) | “robots.txt rules may not apply” |
| Google user-triggered fetchers, incl. Google-Agent | “generally ignore robots.txt rules” |
| Perplexity-User | “this fetcher generally ignores robots.txt rules” |
| meta-externalfetcher (Meta) | “may bypass robots.txt rules” |
| Amzn-User (Amazon) | “may not follow all robots.txt directives” |
Sources: OpenAI, Google, Perplexity, Meta, Amazon, verified 2026-11-16.
Anthropic is the exception. It says its bots, Claude-User included, honor “industry standard directives in robots.txt”. Mistral says only that “MistralAI-User governs which sites these user requests can be made to”, without spelling out how robots.txt applies. Anthropic, Mistral.
If you need to stop these fetchers, robots.txt isn’t the tool. Use a server, firewall or CDN rule that matches the user agent and checks the IP.
On Cloudflare, per-agent controls arrived on November 4; I cover them in Cloudflare’s personal agent controls, and the bucket system behind them in Cloudflare’s AI crawler categories. Whether ignoring robots.txt has legal weight is a separate question, covered in is robots.txt legally binding.
Crawl-delay is uneven too. Anthropic supports it as a “non-standard extension” and Common Crawl honors it, while Amazon says of Amazonbot “They do not support the crawl-delay directive” and Apple says Applebot ignores it.
How do you check a bot is real? #
Anyone can put “GPTBot” in a user agent string, so the string alone proves nothing. Check the IP, using the method the operator documents.
Reverse and forward DNS (Google, Apple, Common Crawl, Bing) #
Four operators document a two-step DNS check. You look up the hostname for the IP, check its domain, then look up that hostname and confirm it points back to the same IP. The forward step matters because anyone controlling an IP’s reverse DNS can make it say anything.
host <ip> # reverse lookup: returns a hostname
host <hostname> # forward lookup: must return the same <ip>
| Operator | The hostname must end in | Source |
|---|---|---|
| googlebot.com, google.com or googleusercontent.com | Google verify page | |
| Apple | applebot.apple.com | Apple |
| Common Crawl | crawl.commoncrawl.org | Common Crawl FAQ |
| Microsoft | search.msn.com | Bing, 2012 |
Google’s page uses the host <ip> command itself. Common Crawl’s real CCBot identifies as CCBot/2.0 (https://commoncrawl.org/faq/). Bing’s method dates from 2012 but is still the only one I found for bingbot, and its IP file is older than the others (see below), so I’d rely on DNS for Bing.
Matching the vendor’s IP list (everyone else) #
OpenAI, Anthropic, Perplexity, Amazon, Mistral and DuckDuckGo publish JSON files of IP ranges instead. You check whether the request’s IP falls inside one of the listed ranges. The files use a prefixes array: OpenAI’s holds IPv4 CIDR ranges under ipv4Prefix, Amazon’s holds single IPs, and Google’s add ipv6Prefix entries.
Here is an outline for that check in Python. I wrote it from the file formats I read and haven’t run it against live traffic. Amazon’s lists came back as web pages with the JSON inside, so extract it first, and check before you rely on it.
import ipaddress, json, urllib.request
def ip_in_vendor_list(ip: str, url: str) -> bool:
addr = ipaddress.ip_address(ip)
data = json.load(urllib.request.urlopen(url))
for entry in data.get("prefixes", []):
values = entry.values() if isinstance(entry, dict) else [entry]
for value in values:
try:
if addr in ipaddress.ip_network(value, strict=False):
return True
except (TypeError, ValueError):
continue
return False
Cache the files and refresh them on a schedule rather than fetching one per request.
When there’s nothing to check against #
Meta publishes no IP list and no verification method on its crawler page; it only suggests allowlisting user agent strings or IPs. Mistral publishes no list for MistralAI-Training. Meta, Mistral.
So a request calling itself meta-externalagent can’t be verified from Meta’s docs. I’d rate-limit those requests rather than trust or block them on the string alone.
How fresh are the IP lists? #
The files carry a creationTime, and the 21 I opened range from January 2024 (Bing) to November 13, 2026 (Google). Here is where every list lives, as the vendor pages give it.
| Operator | IP file(s) | creationTime |
|---|---|---|
| OpenAI | gptbot.json, searchbot.json, chatgpt-user.json, adsbot.json | 2026-09-22 (18 ranges), 2026-01-02, 2026-11-06, 2026-05-12 |
| Anthropic | bots.json, one list for all three bots | 2026-10-20 (14 ranges) |
| Amazon | Amazonbot, Amzn-SearchBot, Amzn-User | 2026-09-08 (Amazonbot, over 1,000 IPs), 2026-09-08, 2025-11-04 |
| Microsoft | bingbot.json | 2024-01-03 (29 ranges) |
| common-crawlers.json, special-crawlers.json, user-triggered-fetchers.json, user-triggered-fetchers-google.json, user-triggered-agents.json | 2026-11-13 (all five) | |
| Perplexity | perplexitybot.json, perplexity-user.json | 2026-10-28 (both) |
| Apple | applebot.json | 2026-09-15 |
| Common Crawl | ccbot.json | 2026-08-11 |
| Mistral | mistralai-user-ips.json, mistralai-index-ips.json | 2025-02-19, 2026-04-19 |
| DuckDuckGo | duckassistbot.json | 2026-09-01 |
| Meta | None published | n/a |
All files opened and verified 2026-11-16. The file names come from each operator’s page, linked under the master table; Google’s come from its verification page.
Two catches. Anthropic’s list doesn’t say which bot uses which IP, so an IP match proves the request is Anthropic’s but not whether it’s ClaudeBot or Claude-User. And Anthropic warns that blocking its IPs may stop its bots from reading your robots.txt at all. Anthropic.
Bing’s file is nearly three years old. I’d treat it as a secondary signal and use the DNS check for bingbot. Mistral’s MistralAI-User file (February 2025) and Amazon’s Amzn-User file (November 2025) are also more than a year old.

What about signed bots? #
Only Google documents signing for an AI agent so far, and only for part of its traffic. Google says “A subset of requests made by the Google-Agent are signed”, with the identity https://agent.bot.goog and its key directory at /.well-known/http-message-signatures-directory on that host. Google Web Bot Auth.
Google calls it experimental and tells sites to keep IP and DNS checks and to fall back to them for unsigned requests. OpenAI’s bots page doesn’t mention Web Bot Auth at all.
If you want to check signatures yourself, the scheme Cloudflare documents uses RFC 9421 message signatures, Ed25519 keys (it ignores other key types), a JWKS key directory at that same well-known path, and Signature-Agent and Signature-Input headers with tag="web-bot-auth". Cloudflare Web Bot Auth. My post on ChatGPT’s wallet and agent access has more context on Web Bot Auth.
Which tokens can you delete? #
Four names still turn up in robots.txt files and block lists but aren’t on their vendors’ pages today:
| Token | Status on 2026-11-16 |
|---|---|
anthropic-ai |
Not on Anthropic’s page, which lists ClaudeBot, Claude-SearchBot and Claude-User |
Claude-Web |
Not on Anthropic’s page |
GPTBot-Trainer |
Not on OpenAI’s page, which lists four agents |
Google-NotebookLM |
Replaced by Google-GeminiNotebook; Google supported the old token “until August 2026” |
Sources: Anthropic, OpenAI, Google fetchers.
Leaving them in probably does no harm, but it can hide the fact that you may be missing the current names. If you copy a community list such as ai.robots.txt, which describes itself as “A list of AI agents and robots to block” and ships robots.txt, .htaccess, nginx, Caddy, haproxy and lighttpd snippets, remember that it’s a block list. It doesn’t split search from training.
Why isn’t Bytespider on this list? #
Because I couldn’t read ByteDance’s own page. Its crawler page rendered only a title, and the secondary sources I found disagree about whether Bytespider obeys robots.txt and which IPs it uses. Every other row here comes from an operator page I could read, and I’d rather leave a gap than fill it with someone else’s guess.
So what should you allow or block? #
If you want to appear in AI search answers but not in training sets, this is the split I’d use, from the vendors’ own descriptions:
| Allow (search) | Disallow (training) |
|---|---|
| OAI-SearchBot | GPTBot |
| Claude-SearchBot | ClaudeBot |
| PerplexityBot | meta-externalagent |
| Amzn-SearchBot | MistralAI-Training |
| Applebot | CCBot (open dataset) |
| Google-Extended (control token) | |
| Applebot-Extended (control token) |
As a robots.txt block, that looks like this:
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amzn-SearchBot
User-agent: Applebot
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
Some judgement calls. Meta says meta-externalagent covers training or indexing, so disallowing it may cost you more than training. Amazonbot may train Amazon’s models; disallow it too if you don’t want that, since Amzn-SearchBot handles search.
Common Crawl’s FAQ doesn’t say who trains on CCBot’s dataset; I’d disallow it because the dataset is public. Anthropic says you need a block per subdomain. Applebot follows your Googlebot rules when it has no group of its own, which is one more reason to give it one.
Googlebot and bingbot stay allowed by default here; Google-Extended handles Gemini training without touching Search. And none of this stops the user fetchers in the box above.
What I would do next #
Pull a week of access logs and count the 29 tokens:
grep -Eo 'GPTBot|OAI-SearchBot|ChatGPT-User|OAI-AdsBot|ClaudeBot|Claude-SearchBot|Claude-User|Googlebot|GoogleOther|Google-CloudVertexBot|Google-Agent|Google-GeminiNotebook|PerplexityBot|Perplexity-User|Applebot|meta-externalagent|meta-webindexer|meta-externalfetcher|Amazonbot|Amzn-SearchBot|Amzn-User|CCBot|bingbot|MistralAI-Index|MistralAI-User|MistralAI-Training|DuckAssistBot' access.log | sort | uniq -c | sort -rn
Google-Extended and Applebot-Extended never appear in logs, because they don’t crawl. For the top five by volume, verify a sample of IPs with the method in their row. Delete the stale tokens, write the allow/disallow split above, and add a server rule for any user fetcher you don’t want.
Then put a 90-day reminder in the calendar. Seven of the 11 operators’ crawler pages carry no date, so the only way to catch a change is to re-read them. The data hub collects the rest of what I’ve written on collecting and controlling web data, including whether scraping for AI training is legal.
FAQ
Does blocking GPTBot remove me from ChatGPT search?
No. OpenAI runs a separate crawler, OAI-SearchBot, for ChatGPT search, and says robots.txt changes for it take about 24 hours. Disallowing GPTBot signals that your content shouldn't be used for training.
Is Google-Extended a crawler?
No. Google calls it a standalone product token with no HTTP user agent. Google's normal crawlers do the crawling, and Google-Extended only controls whether that content may train future Gemini models and ground answers in Gemini Apps and Vertex AI. Google says it doesn't affect Search inclusion or ranking.
Which AI crawlers respect Crawl-delay?
Anthropic says ClaudeBot supports Crawl-delay as a non-standard extension, and Common Crawl says CCBot honors it. Amazon says Amazonbot does not support crawl-delay, and Apple says Applebot ignores it.
Why does Perplexity-User ignore my robots.txt?
Because Perplexity treats it as a fetch a user asked for, not a crawl. Its docs say 'this fetcher generally ignores robots.txt rules'. OpenAI, Google, Meta and Amazon say much the same about their user-triggered fetchers. To stop them, use a server or firewall rule.
How do I verify a request claiming to be Googlebot?
Run a reverse DNS lookup on the IP. The hostname must be in googlebot.com, google.com or googleusercontent.com. Then run a forward lookup on that hostname and check it returns the same IP. You can also match the IP against Google's published JSON ranges.
Is Bytespider on this list?
No. ByteDance's own crawler page rendered only a title when I opened it, and the secondary sources I found disagree on whether Bytespider obeys robots.txt and which IPs it uses. I only list bots I could check on the operator's own page.
Sources
37 sources, accessed .
Official pricing, docs, changelogs and status pages 36
- OpenAI: crawlers and user agents
- OpenAI: GPTBot IP ranges (creationTime 2026-09-22)
- Anthropic: Does Anthropic crawl the web and how can site owners block the crawler?
- Anthropic: bot IP list (creationTime 2026-10-20)
- Google: common crawlers and Google-Extended
- Google: user-triggered fetchers and agents
- Google: verifying Google's crawlers and fetchers
- Google: Web Bot Auth for Google-Agent (updated 2026-10-13)
- Perplexity: crawlers
- Apple: About Applebot (dated 2026-09-04)
- Meta: web crawlers
- Amazon: Amazonbot and other Amazon crawlers (updated 2026-10-26)
- Amazon: Amazonbot IP addresses (creationTime 2026-09-08)
- Common Crawl: FAQ and CCBot
- Bing: how to verify that Bingbot is Bingbot (2012-08-31)
- Bing: bingbot IP ranges (creationTime 2024-01-03)
- Mistral AI: robots and user agents
- DuckDuckGo: DuckAssistBot
- Cloudflare docs: Web Bot Auth (updated 2026-10-08)
- OpenAI: OAI-SearchBot IP ranges (creationTime 2026-01-02)
- OpenAI: ChatGPT-User IP ranges (creationTime 2026-11-06)
- OpenAI: OAI-AdsBot IP ranges (creationTime 2026-05-12)
- Amazon: Amzn-SearchBot IP addresses (creationTime 2026-09-08)
- Amazon: Amzn-User IP addresses (creationTime 2025-11-04)
- Google: common crawlers IP ranges (creationTime 2026-11-13)
- Google: special-case crawlers IP ranges (creationTime 2026-11-13)
- Google: user-triggered fetchers IP ranges (creationTime 2026-11-13)
- Google: user-triggered fetchers (Google) IP ranges (creationTime 2026-11-13)
- Google: user-triggered agents IP ranges (creationTime 2026-11-13)
- Perplexity: PerplexityBot IP ranges (creationTime 2026-10-28)
- Perplexity: Perplexity-User IP ranges (creationTime 2026-10-28)
- Apple: Applebot IP ranges (creationTime 2026-09-15)
- Common Crawl: CCBot IP ranges (creationTime 2026-08-11)
- Mistral AI: MistralAI-User IPs (creationTime 2025-02-19)
- Mistral AI: MistralAI-Index IPs (creationTime 2026-04-19)
- DuckDuckGo: DuckAssistBot IPs (creationTime 2026-09-01)


