Cloudflare's AI crawler rules: what changed on Sept 15, 2026

As of November 12, 2026, Cloudflare sorts AI bot traffic into three groups, Search, Agent and Training, and lets every customer allow or block each one. The setting most site owners want arrived on September 15: Disallow AI Training, which refuses training while Googlebot, Bingbot and Applebot stay allowed for search. Cloudflare.

That isn’t what Cloudflare announced in July, and much of what you’ll read still describes the July plan, under which blocking training would also have blocked Googlebot. Below I line up what was promised against what shipped, map the well-known crawlers to each group, and give separate next steps for site owners and for people who build crawlers and agents.

Short on time?

What I'd do now

Site owner who wants search traffic but no training

Disallow AI Training, with Bot Preference Sync on

Googlebot, Bingbot and Applebot stay allowed for search. Don't pick Block: since September 15 it stops them too.

Ad-funded site that doesn't want agents

Agent: Block on pages with ads

Cloudflare's own preset for ad sites. By Cloudflare's definition it also stops assistants fetching pages for a user in real time.

You build a crawler or AI agent

Split by purpose, then get Verified or sign with Web Bot Auth

Unverified bots stay blocked by default, and crawlers that don't explain their purpose don't get 'the benefit of the doubt'.

Your team scrapes sites behind Cloudflare

Treat a policy block as a no

These are the site owner's stated choices, not just bot detection. Ask the owner, use an official API, or skip the site.

You buy search results from an API

Change nothing

These settings govern traffic to Cloudflare customers' sites. Search-results APIs are affected by Google's and Brave's own changes instead.

Prices verified 12 Nov 2026. Sponsors cannot buy a verdict or a position.

What are Search, Agent and Training? #

They’re Cloudflare’s three buckets for AI traffic, defined by what the bot does with your content, not by who runs it. From Cloudflare’s July 1, 2026 post:

• Search: “any behavior that collects or indexes your content, so it can answer questions about it later”.
• Agent: “acting, usually in real time, on a person’s behalf, to get something done right now”.
• Training: “a crawler taking your content to train or fine-tune a model”.

The November 2 docs add a fourth label, Personal Agents, which inherit the Agents policy; Cloudflare recommends Allow for them. Each bucket takes one of four actions: Allow, Block, Block on pages with ads (“where Cloudflare detects ads”), and Disallow, which exists for Training only. Cloudflare docs.

The tricky part is bots that do more than one thing. Googlebot indexes for search, and Google can also use what it crawls for AI. How Cloudflare treats those mixed-use crawlers is what changed between July and September.

What did July promise, and what shipped? #

In July, Cloudflare planned default blocks on ad pages and would have judged Googlebot by everything it does. In September it shipped a softer design built around Disallow AI Training and “Accountable” crawlers. By November, the docs say new domains start on Allow.

Sources: July blog, July press release, July changelog, Bot Preference Sync, September blog, access policies docs, robots.txt docs, all read November 12, 2026.

The question nobody at Cloudflare has answered #

The July 1 press release said the new defaults would apply “For new customers and for new sites for existing customers” and to existing Free customers who hadn’t changed settings by September 15. The July changelog and blog mention only “new domains”. Press release, changelog.

The September 15 blog and press release don’t mention existing Free zones at all. They say “Your current settings carry over on their own”, and the November 2 docs talk only about new domains. So as of November 12, 2026, Cloudflare hasn’t stated what happened to an untouched Free zone on September 15. If you run one, the only reliable answer is your own dashboard.

Some explainers published days before the change describe the July plan as fact. One said it would hit “every existing customer on the free tier”; another’s headline said sites blocking AI training “will start blocking Googlebot”. iqsource.ai, webhosting.today. PPC Land later reported the reversal under the headline “Cloudflare drops planned Googlebot block for sites refusing AI training”. PPC Land.

How it unfolded #

Date	What Cloudflare (or an operator) said	Source
2025-07-01	Content Independence Day: Cloudflare says it is changing the default to block AI crawlers unless they pay creators	Cloudflare blog
2025-08-28	Signed agents introduced, using cryptographic signatures	Cloudflare blog
2026-04-23	Bot reference last updated, still with older labels	Cloudflare docs
2026-07-01	Search, Agent, Training announced; plan for ad-page blocks from September 15	Cloudflare blog, press release, changelog
2026-07-28	Docs: pay per crawl in closed beta; Charge action in private beta	Cloudflare docs
2026-08-21	Bot Preference Sync writes your choices into robots.txt	Cloudflare blog
2026-09-15	Disallow AI Training and Accountable crawlers ship; new-domain presets	Cloudflare blog and press release
2026-10-30	Microsoft says Bing will honor a robots.txt training opt-out from February 1, 2027	Bing blog
2026-11-02	Docs: new domains default to Allow for Search, Agents and Training	Cloudflare docs
2027-02-01	Bing starts honoring __usage_ai-training (announced)	Bing blog

Source: Cloudflare blog posts, press releases, changelog and docs; Microsoft Bing blog. All read 12 Nov 2026. The 2025-07-01 row paraphrases that post's own wording. Scroll sideways for more columns.

Does blocking AI training remove you from Google? #

Not if you pick Disallow AI Training. Cloudflare says “Accountable mixed-use crawlers remain allowed for search”, while “Every other training crawler is blocked, including the training-only crawlers run by Amazon, Anthropic, Meta, and OpenAI — blocking those does not affect search.” Cloudflare.

Yes if you pick Block. Since September 15, “Block and Block on pages with ads now apply to mixed-use crawlers, including Applebot, Bingbot, and Googlebot”, so choosing Block for Training stops them “— search included”. The July plan would have done that even for a plain training opt-out; that’s the change.

What makes a crawler “Accountable”? #

Cloudflare set four tests for mixed-use crawlers, published September 15, 2026. Cloudflare.

1. A training opt-out “through robots.txt or a similar standard”.
2. An opt-out from AI summaries, arranged with the operator.
3. “URL-level visibility into which pages were made available for training”, plus search metrics.
4. “Assurance that opting out of AI training will not affect traditional search results”.

Applebot, Bingbot and Googlebot are on the list. Amazon, Anthropic, Meta and OpenAI qualify differently: they “separate their Search and Training crawlers, so Cloudflare can block the Training crawler without affecting search”.

Cloudflare’s numbers explain why it bothered. Its September post says “less than 1% of Cloudflare sites choose to block Search bots” while “17% of sites choose to enable some mechanism to block training”, and its press release says mixed-use crawlers “now make up 36.6% of verified crawler traffic on Cloudflare’s network, the single largest category”. Those are Cloudflare’s figures. Press release.

Google, Bing and Apple each work differently #

Google splits the decision with a robots.txt token, Google-Extended. Google says it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”, and it has no separate user agent string; Google’s existing crawlers do the crawling. Google. Cloudflare’s September post also said Google would offer URL-level tools “in the weeks to come”; I found none launched as of November 12.

Bing had no training opt-out when Cloudflare shipped, so Cloudflare said Disallow wouldn’t reach Bing until support “targeted for early 2027”. Microsoft has since put a date on it. Its October 30 post introduces User-Agent: __usage_ai-training with Disallow: /, and says “Starting February 1, 2027, Microsoft will honor the new preferences” “without changing eligibility for Bing search results”. Bing.

Bing handles grounding, meaning using your page to back an AI answer, per page instead, with an ai-grounding:disallow meta tag or X-Robots-Tag. Whether Cloudflare’s Bot Preference Sync will write the new Bing token for you isn’t stated.

Apple uses Applebot-Extended, which “does not crawl webpages”; it only controls use. Pages disallowed for it “can still be included in search results”, per Apple. Apple.

Which crawler lands in which bucket? #

Cloudflare’s bot reference (updated April 23, 2026) still uses older labels. The “likely bucket” column is my inference from each operator’s own description; Cloudflare hasn’t published this mapping.

OpenAI says GPTBot is for its foundation models and OAI-SearchBot is “used to surface websites in search results in ChatGPT’s search features”. Anthropic says its bots “respect ‘do not crawl’ signals” in robots.txt. Google-Extended and Applebot-Extended aren’t in Cloudflare’s reference, since neither crawls on its own.

How do the settings work in practice? #

The controls live under Security > Settings > Bot traffic, and all customers, Free plan included, can use them. Docs, changelog.

Here is what each setting does to three well-known bots. The Googlebot and GPTBot cells come from Cloudflare’s text; the ChatGPT-User cells assume it falls in the Agent bucket, which is my inference.

Sources: September blog, access policies docs. Cloudflare doesn’t describe how it detects ads, only that “Cloudflare can detect which pages serve ads”.

Disallow only blocks at the edge with Sync on #

The November 2 docs are precise here: “When Bot Preference Sync is enabled, Disallow also blocks AI training crawlers at Cloudflare’s edge”, and “Mixed-purpose crawlers used for both Search and Training are excluded from this edge block.” Without Sync, Disallow is a stated preference that crawlers may or may not honor. Docs.

What Bot Preference Sync writes into robots.txt #

Sync adds Cloudflare’s rules to your robots.txt between BEGIN and END markers, “prepended to the existing material”, and creates the file if you don’t have one. Your own Disallow lines stay. Bot Preference Sync, robots.txt docs.

Rules are written only if one of your policies is Block, Block on pages with ads or Disallow; otherwise robots.txt is left alone. The docs’ example includes Content-signal: search=yes, ai-train=no, use=reference plus Disallow lines for crawlers such as GPTBot and CCBot. robots.txt itself is a request, not a lock, as RFC 9309 says; my post on whether robots.txt is legally binding covers what that means.

Free plan detection is simpler #

On the Free plan, Cloudflare identifies AI crawlers “based on their user agent strings”; paid plans use Bot Management detection IDs. Manage AI crawlers. A bot that lies about its user agent is harder to catch on Free.

What happened to old settings #

For sites that never used the new controls, Cloudflare mapped the legacy “Block AI Bots” switch like this: off became Allow for everything; Block became Search Allow, Training Disallow and Agent Block on pages with ads; and Block on pages with ads became the same as Block. Cloudflare.

Cloudflare’s September post says Block AI Bots and Managed robots.txt will be deprecated, but the November 2 robots.txt docs carry no deprecation note. I’d expect them to go, without a date to plan around.

If you run a crawler or agent, what gets you blocked? #

Cloudflare has said plainly what it rewards: honest identity, one purpose per crawler, and proof that you are who you say. Here’s what its pages say, oldest first.

• Unverified bots stay blocked by default. “non-verified bots are still default blocked” (July 1, 2026). Cloudflare.
• Full-content reproducers can’t be Verified. “bots that reproduce in full cannot have the Verified status” (same post).
• Split your crawlers. Operators are “strongly encourage[d]” to run separate Search, Agent and Training crawlers, and the September post treats split crawlers as the way to stay in search while training is refused.
• Mixed-use without transparency loses. “Crawlers that don’t provide Transparency will not get the benefit of the doubt.” Bot Preference Sync, as updated Sept 15.

How to prove who you are #

Cloudflare’s route is Web Bot Auth: sign each request with an Ed25519 key and publish your public keys at /.well-known/http-message-signatures-directory, which Cloudflare calls “a requirement”. Requests carry Signature-Agent, Signature-Input (with tag="web-bot-auth", alg="ed25519" and a short expiry) and Signature headers. Web Bot Auth docs.

You register through Cloudflare’s Bot Submission Form with “Request Signature”, and Cloudflare provides crawltest.com to check your setup. The same form covers signed agents, introduced August 28, 2025 for agents “generally directed by an end user instead of a single company”. Signed agents.

Cloudflare is also experimenting with the Forwarded header from RFC 7239 to say who an agent fetches for and how content will be used, for example Forwarded: for="openai";use="reference", with use levels of immediate, reference (the default) and full. That’s an experiment, not a requirement. July post.

If you’re choosing a crawler stack, my open-source LLM crawlers comparison covers which ones let you set user agents and respect robots.txt. For what an unnamed crawler can lead to, see the newspapers’ allegations in why the New York Times is suing Brave.

What does this mean for scraping APIs and proxies? #

These settings add a policy layer on top of Cloudflare’s bot detection. A site with Agent set to Block on pages with ads has said, in its settings, that it doesn’t want automated fetches of those pages, whoever sends them.

I won’t suggest ways around that, and I’d avoid any vendor that markets them for this case. If a site you need has blocked your bucket, ask the owner for access, look for an official API or feed, or drop the site. For the wider legal picture on collecting content for AI, see is scraping for AI training legal?.

Unblocking services still matter for sites that allow your traffic but trip generic bot defenses; my web unblocker comparison and scraping API vs proxies cover those choices. I found no public evidence on how any scraping vendor’s traffic is categorized under these settings.

Search-results APIs are a different case. SERP APIs return search engine results pages rather than fetching Cloudflare customers’ sites, so these settings aren’t the change to watch. Google’s and Brave’s own moves are: see Google’s goto links and poisoned SERPs and Brave’s encrypted result links.

What happened to Pay Per Crawl? #

It’s still not generally available. Cloudflare’s docs, updated July 28, 2026, say “Pay per crawl is currently in closed beta”, and the per-crawler Charge action is “private beta”. Pay per crawl, Manage AI crawlers.

The July 1, 2026 press release says Pay Per Crawl is “evolving” into Pay Per Use, with Ceramic.ai (paying per appearance in AI search results) and You.com (agents paying for premium content) as partners. It gives no availability date, and the docs don’t mention Pay Per Use. Press release.

The same release explains the push: Cloudflare says bots make up more than half of requests and that over half of AI crawler traffic re-fetches pages that haven’t changed. Matthew Prince: “Now that the majority of traffic on the Internet is non-human, we must go further and act faster.”

A five-minute check for site owners #

1. Open Security > Settings > Bot traffic and note what each bucket is set to. Don’t assume; Cloudflare hasn’t said what untouched Free zones got on September 15.
2. If you want search but not training, set Training to Disallow AI Training, not Block.
3. Turn on Bot Preference Sync if you want Disallow enforced at the edge, not just stated.
4. Load yoursite.com/robots.txt and check that Cloudflare’s block sits above your own rules and that your Disallow lines survived.
5. Decide on agents. If you sell ads and don’t want assistants reading those pages, use Block on pages with ads; otherwise leave Allow, which Cloudflare recommends for Personal Agents.

Set a reminder for February 1, 2027, when Bing’s training opt-out takes effect, and check whether your robots.txt carries it.

Which setting should you pick? #

You want Google, Bing and Apple traffic and no AI training: Disallow AI Training with Sync on. It’s the one setting that does both, and it’s enforced at the edge for training-only crawlers.

You’d rather lose search than be trained on: Block for Training. Know that since September 15 that stops Googlebot, Bingbot and Applebot too.

You run an ad-funded site and dislike agents: Agent on Block on pages with ads. Cloudflare’s preset for ad sites does this, and it leaves search alone.

You run a crawler or agent: one purpose per user agent, Verified or signed. Cloudflare’s direction since July has been consistent on this, even while its defaults changed twice.

For the broader set of crawl tools, the crawl API pillar maps them.

FAQ

Will blocking AI training on Cloudflare hurt my Google rankings?

Not if you choose Disallow AI Training. Since September 15, 2026, Cloudflare treats Googlebot, Bingbot and Applebot as Accountable mixed-use crawlers that stay allowed for search under that setting. If you choose Block for Training instead, Cloudflare says it now stops those crawlers too, search included.

What is Disallow AI Training?

A Training-only setting Cloudflare added on September 15, 2026. It publishes a no-training preference in your robots.txt through Bot Preference Sync, and with Sync on it also blocks training crawlers at Cloudflare's edge. Mixed-purpose crawlers used for both search and training, such as Googlebot, are excluded from that edge block.

What's the difference between Block and Disallow on Cloudflare?

Disallow states a no-training preference and blocks training-only crawlers while leaving Accountable search crawlers alone. Block stops every crawler in the category, and since September 15, 2026 that includes mixed-use crawlers like Googlebot, Bingbot and Applebot, so blocking Training that way also removes you from their search.

Did Cloudflare change my settings on September 15?

Cloudflare says most customers need to do 'Nothing, in almost every case' and that current settings carry over. Training selections of Block or Block on pages with ads migrated to Disallow AI Training. Cloudflare has not said what happened to existing Free zones that never touched these settings, so check Security, Settings, Bot traffic in your dashboard.

How do I get my crawler verified by Cloudflare?

Submit it through Cloudflare's Bot Submission Form. To sign requests, use Web Bot Auth: an Ed25519 key, a public key directory at /.well-known/http-message-signatures-directory, and Signature-Agent, Signature-Input and Signature headers. Cloudflare's July 2026 blog says bots that reproduce content in full cannot have Verified status.

Is Cloudflare Pay Per Crawl available?

Not generally. Cloudflare's docs (updated July 28, 2026) say pay per crawl is in closed beta and list the Charge action as private beta. Cloudflare's July 1, 2026 press release says it is evolving into Pay Per Use, with Ceramic.ai and You.com as partners, but gives no general-availability date.

Sources

24 sources, accessed 12 Nov 2026.

Official pricing, docs, changelogs and status pages 19

1. Cloudflare: Have it both ways (2026-09-15) blog.cloudflare.com · accessed 12 Nov 2026
2. Cloudflare press release: Search-or-AI-training tradeoff (2026-09-15) cloudflare.com · accessed 12 Nov 2026
3. Cloudflare: Your site, your rules (2026-07-01) blog.cloudflare.com · accessed 12 Nov 2026
4. Cloudflare press release: Your content, your rules (2026-07-01) cloudflare.com · accessed 12 Nov 2026
5. Cloudflare changelog: New options to manage AI traffic (2026-07-01) developers.cloudflare.com · accessed 12 Nov 2026
6. Cloudflare: Say it once, Bot Preference Sync (2026-08-21, updated 2026-09-15) blog.cloudflare.com · accessed 12 Nov 2026
7. Cloudflare docs: AI Crawler and Agent access policies (updated 2026-11-02) developers.cloudflare.com · accessed 12 Nov 2026
8. Cloudflare docs: robots.txt setting (updated 2026-11-02) developers.cloudflare.com · accessed 12 Nov 2026
9. Cloudflare docs: Manage AI crawlers (updated 2026-07-28) developers.cloudflare.com · accessed 12 Nov 2026
10. Cloudflare docs: What is pay per crawl (updated 2026-07-28) developers.cloudflare.com · accessed 12 Nov 2026
11. Cloudflare docs: bot reference (updated 2026-04-23) developers.cloudflare.com · accessed 12 Nov 2026
12. Cloudflare docs: Web Bot Auth developers.cloudflare.com · accessed 12 Nov 2026
13. Cloudflare: The age of agents (2025-08-28) blog.cloudflare.com · accessed 12 Nov 2026
14. Cloudflare: Content Independence Day (2025-07-01) blog.cloudflare.com · accessed 12 Nov 2026
15. Google: common crawlers and Google-Extended developers.google.com · accessed 12 Nov 2026
16. Apple: About Applebot support.apple.com · accessed 12 Nov 2026
17. OpenAI: crawlers and user agents developers.openai.com · accessed 12 Nov 2026
18. Anthropic: Does Anthropic crawl the web? support.claude.com · accessed 12 Nov 2026
19. Microsoft Bing: Clearer preferences for training and grounding (2026-10-30) blogs.bing.com · accessed 12 Nov 2026

News and press 1

1. PPC Land: Cloudflare drops planned Googlebot block (September 2026) ppc.land · accessed 12 Nov 2026

Other blogs 2

1. iqsource.ai explainer (2026-09-14, describes the July plan) iqsource.ai · accessed 12 Nov 2026
2. webhosting.today explainer (2026-09-11, describes the July plan) webhosting.today · accessed 12 Nov 2026

Other sources 2

1. RFC 9309: Robots Exclusion Protocol rfc-editor.org · accessed 12 Nov 2026
2. RFC 7239: Forwarded HTTP extension rfc-editor.org · accessed 12 Nov 2026