Skip to content
CrawlfolioSourced research on web data for AI
Menu

Explainer

Cloudflare can now block personal AI agents one by one

Cloudflare sites can now allow or block each personal AI agent, or block it only on ad pages. What changed on Nov 4 and what agent builders should do now.

Looking through a tunnel of hexagonal mirror panels that repeat a cloudy sky and a distant mountain ridge
Photo: drmakete lab / Unsplash · Unsplash License

Since November 4, 2026, sites on Cloudflare can set Allow, Block on pages with ads, or Block for each of the “top Personal Agents”, separately from the broader Agents policy. Cloudflare changelog. If you build an agent that browses or fetches pages for people, you need to decide what to change so yours isn’t the one blocked.

If you run a site, you need to pick a setting.

The catch is that Cloudflare doesn’t publish which agents are on that list, and its own docs recommend leaving personal agents on Allow. So for builders, the real lever is identity. For the Search, Agent and Training buckets underneath all this, see my explainer on Cloudflare’s AI crawler categories.

Short on time?

Short on time? Here's my take.

You build an AI agent
Sign with Web Bot Auth and submit the agent for verification now
Being recognized is the documented way to become an agent a site can allow on its own, and Web Bot Auth is Cloudflare's preferred route (IP validation is the other). Medium confidence: the drafts are still changing and the list criteria are unpublished.
Your agent reads ad-supported publisher pages
Route those reads through a search or content API or licensed source
Keep the browser for tasks that need a page session. 'Block on pages with ads' targets exactly that traffic. Medium confidence: adoption is unknown.
You run an ad-funded site
Start on Allow, watch the analytics, then block on ad pages per agent
Allow is Cloudflare's own recommendation for personal agents. Switch an agent to 'Block on pages with ads' only if it shows up heavily on ad pages.
You run a shop or a login-heavy site
Keep agents allowed and act per account
Account Abuse Protection and a WAF rule on the Hashed User ID stop one bad account without blocking an agent for everyone. Medium confidence: early access only.

Prices verified . Sponsors cannot buy a verdict or a position.

What changed for personal agents on November 4? #

Cloudflare added per-agent switches for personal agents inside the Agents policy. Cloudflare’s blog lists three options per agent: “Allow (do not block)”, “Block on pages with ads” and “Block”. Search and Training are unchanged.

The changelog defines personal agents as bots that “act on a person’s behalf, fetching information or browsing in response to a request”. The setting, under Security > Settings, was renamed from “Configure AI bot policies” to “AI Crawler and Agent access policies”.

There’s new visibility too. Bot Analysis gets a Personal Assistants category, a Top statistics widget lists each agent’s request count, and filtering on one agent shows its paths, status codes and networks. Onboarding now asks whether the site monetizes pages with ads and adjusts the recommended policies.

The quieter change is for logins. Since November 3, Account Abuse Protection “now shows when a recognized AI assistant makes a login or sign up request”. It covers “login and sign up events only”, and unrecognized assistants “appear as ordinary traffic”. Changelog.

It’s an early-access product with its own dashboard since November 2. Changelog index. Cloudflare’s blog says fraud teams can stop one account with a WAF rule on its Hashed User ID without blocking the agent for everyone.

Which agents can a site block? #

Cloudflare doesn’t say. The blog says Cloudflare “maintains the largest verified bot directory in the world” but names no agents and gives no traffic numbers. The access policy docs name no Personal Agents either.

The closest thing to a list is in the verified-bots docs, which define Agent as “User-directed agents visiting a page on behalf of a human” and give two legacy AI Assistant examples: Perplexity-User and DuckAssistBot. Groundy also found the supported list undefined, and Unite.AI names none.

What the docs do say matters more for builders: “Each policy applies to Verified bots with that behavior and matching unverified bots”. A recognized agent can be singled out; everything else falls under the broad rule.

Is anything blocked by default? #

Not on new domains. The docs, updated November 2, say “For new domains, Search, Agents, and Training use Allow (do not block) by default”, and “Cloudflare recommends Allow (do not block) for Personal Agents”. They don’t state defaults for existing zones.

How the personal agent settings behaveVerified
Setting or ruleWhat Cloudflare's docs say
Default for new domainsAllow for Search, Agents and Training
Personal AgentsInherit the Agents policy by default
Changing the Agents policyUpdates each Personal Agent, so per-agent picks must be set again
BlockBlocks the agent across the zone
Block on pages with adsBlocks where Cloudflare detects ads
Cloudflare's recommendationAllow (do not block) for Personal Agents
Existing zonesNot stated

Source: Cloudflare docs, AI Crawler and Agent access policies (updated 2 Nov 2026), read 16 Nov 2026. Scroll sideways for more columns.

Plan availability is also unclear. The Free plan page, last updated May 6, lists “Block AI bots” and says nothing about per-agent controls.

A dark metal door handle in sharp focus with a row of blurred doors along a coral-coloured corridor behind it
Every door in the corridor looks the same until you pick one: per-agent settings let a site owner decide door by door, and the defaults decide for everyone who doesn't. Photo: engin akyurt / Unsplash · Unsplash License

Why doesn’t robots.txt settle this? #

Because personal agents act for a person in real time, and their operators don’t always treat robots.txt as binding. OpenAI says of ChatGPT-User: “Because these actions are initiated by a user, robots.txt rules may not apply.” OpenAI.

On a Cloudflare site, the edge policy is what decides. That’s a practical difference, not just a legal one; my post on whether robots.txt is legally binding covers the law.

The ad-page option depends on Cloudflare’s ad detection. In July 2025 it described scanning HTML for ad-unit patterns with the 400 most common EasyList filters, plus CSP reports for dynamic ads and known ad hostnames, “available to all Cloudflare customers, including everyone on a Free plan”. Cloudflare, 2025.

That was for training crawlers; Cloudflare hasn’t confirmed the same method for agents.

What is Trusted Agent Delegation, really? #

A problem statement, so far. Cloudflare’s November 5 post lists 14 problems with agents acting for people and proposes a coalition “to bring a group of representatives across the ecosystem”. It names no members.

The paper sketches a user-identity spectrum (Nothing, Session, Account, Unique) and points to related standards: Web Bot Auth, MOLE, x402, OAuth, WIMSE and AIPREF. It calls Web Bot Auth Cloudflare’s “preferred method” for agent identity, and admits “We must still define a mechanism that supports multiple intermediaries”.

The coalition site says “Upcoming Collaboration”, with an email sign-up and no members, dates or spec. Eleven days after the paper, that’s still where it stands.

PPC Land quotes the paper’s conclusion, “In nearly every challenge above, the Site bears the primary cost”, and notes that Cloudflare sells bot management. Both are worth keeping in mind.

I found no Hacker News story on the topic above 5 points. HN search.

What should agent builders change? #

Make your agent recognizable. A site can only allow you by name if Cloudflare knows who you are, and the two documented routes are Web Bot Auth and IP validation. Signed agents docs.

The Web Bot Auth docs set out the steps:

  1. Generate an Ed25519 key pair.
  2. Publish your public keys as a JWKS at /.well-known/http-message-signatures-directory.
  3. Sign every request with Signature-Input (including tag="web-bot-auth"), Signature and Signature-Agent headers.
  4. Register the agent through Cloudflare’s Bot Submission Form.

The underlying IETF drafts are at draft-meunier-http-message-signatures-directory-03 and draft-meunier-web-bot-auth-architecture-02, so expect changes. Since July 1, 2026, signed agents are a metadata field inside Verified bots, with operators labelled “Direct” or “Intermediary”. Cloudflare is also “experimenting” with the Forwarded header to describe the end user.

Second, stop using a browser session for content you could get another way. “Block on pages with ads” is aimed squarely at agents reading ad-funded pages, so route those reads through a search or content API or a licensed source. My search vs fetch vs browser guide explains when each one fits, and the cloud browser roundup covers tasks that truly need a page.

A proxy doesn’t change a site owner’s stated choice. My proxies for AI agents guide is for sites that allow your traffic, not for getting around a block.

Cross-section of a cut tree trunk showing tight growth rings and deep radial cracks in warm brown wood
No two trunks share the same rings; a signed request plays the same role for your agent, giving a site something unique to recognize before it decides to let you in. Photo: Joel & Jasmin Førestbird / Unsplash · Unsplash License

What should site owners pick? #

Start on Allow, as Cloudflare recommends for personal agents, then act per agent or per account where the analytics show a problem. Here is my pick by site type:

Your site My pick Why
Ad-funded publisher Allow, then “Block on pages with ads” per agent Cloudflare recommends Allow; the analytics show which agents hit ad pages
Shop or login-heavy site Allow, plus Account Abuse Protection and per-account WAF rules Stops one bad account without turning away every customer’s agent
Docs, support or marketing site Allow Agents reading your pages for users are usually the point
Content you never want agents to fetch Block for that agent Remember it’s reset whenever you change the Agents policy

Whatever you choose, check it again after any change to the Agents policy, because the docs say that change “updates each” personal agent.

What I’d do, and what we still don’t know #

If I were building an agent, I’d publish a Web Bot Auth key directory, sign every request and submit the agent this week. I’d keep a path that doesn’t depend on a browser session for publisher content, and log 4xx and challenge responses per domain so a new block shows up in days, not months.

Here is what Cloudflare still hasn’t said:

Open question Status on 2026-11-16
Which agents are “top Personal Agents” Not published
Which plans get per-agent controls Not stated
What a blocked agent receives (403, challenge, 402) Not stated
Ad detection for agents 2025 method likely, not confirmed
Defaults for existing zones Not stated
Trusted Agent Delegation members and dates None announced

The agent browser hub maps the rest of the tooling for agents that browse.

FAQ

Which AI agents can Cloudflare block individually?

The 'top Personal Agents', in Cloudflare's words. Its blog, changelog and docs don't publish which agents those are. The verified-bots docs give only legacy AI Assistant examples, Perplexity-User and DuckAssistBot.

Are personal AI agents blocked by default on Cloudflare?

Not on new domains. Cloudflare's docs say Search, Agents and Training use Allow by default for new domains, and Cloudflare recommends Allow for Personal Agents. The docs don't state defaults for existing zones.

What does Block on pages with ads mean?

It blocks matching requests only where Cloudflare detects ads. In July 2025 Cloudflare described its ad detection as an HTML scan for ad-unit patterns using the 400 most common EasyList filters, CSP reports and known ad hostnames. It hasn't confirmed the same method for agents.

How does my agent get recognized by Cloudflare?

Through Web Bot Auth or IP validation. For Web Bot Auth you create an Ed25519 key, publish your keys at /.well-known/http-message-signatures-directory, sign requests with the Signature-Input, Signature and Signature-Agent headers, and register through Cloudflare's Bot Submission Form.

Does robots.txt control personal agents?

Not reliably. OpenAI says that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. On a Cloudflare site, the edge policy the owner sets is what decides.

What is Trusted Agent Delegation?

A Cloudflare-led coalition announced on November 5, 2026 with a paper listing 14 problems in how agents act for people. As of November 16 its site says Upcoming Collaboration and names no members, dates or spec.

Sources

17 sources, accessed .

Official pricing, docs, changelogs and status pages 13

  1. Cloudflare changelog: Granular access controls for the top Personal Agents (2026-11-04) developers.cloudflare.com · accessed
  2. Cloudflare blog: personal agents visibility and control (2026-11-04, modified 2026-11-09) blog.cloudflare.com · accessed
  3. Cloudflare changelog: Account Abuse Protection and AI assistants (2026-11-03) developers.cloudflare.com · accessed
  4. Cloudflare changelog: application security index developers.cloudflare.com · accessed
  5. Cloudflare docs: AI Crawler and Agent access policies (updated 2026-11-02) developers.cloudflare.com · accessed
  6. Cloudflare blog: control content use for AI training (2025-07-01) blog.cloudflare.com · accessed
  7. Cloudflare docs: Verified bots (updated 2026-11-12) developers.cloudflare.com · accessed
  8. Cloudflare docs: Signed agents (updated 2026-11-12) developers.cloudflare.com · accessed
  9. Cloudflare docs: Web Bot Auth (updated 2026-10-08) developers.cloudflare.com · accessed
  10. Cloudflare blog: Trusted Agent Delegation (2026-11-05, modified 2026-11-10) blog.cloudflare.com · accessed
  11. Trusted Agent Delegation site trustedagentdelegation.org · accessed
  12. OpenAI: crawlers and user agents developers.openai.com · accessed
  13. Cloudflare docs: Bots on the Free plan (updated 2026-05-06) developers.cloudflare.com · accessed

Community discussions 1

  1. Hacker News search: Cloudflare agents stories since early November 2026 hn.algolia.com · accessed

News and press 2

  1. PPC Land: Cloudflare says websites pay for AI agents (2026-11-09) ppc.land · accessed
  2. Unite.AI: per-agent access controls (2026-11-04) unite.ai · accessed

Other blogs 1

  1. Groundy: Cloudflare now lets websites gate AI agents (2026-11-04) groundy.com · accessed