Explainer
Ethical web scraping, as a checklist you can actually use
An ethical web scraping checklist of 14 sourced lines: robots.txt, bot identity, rate limits, AI-training signals, personal data, and when to stop.

Before your team runs a crawl, you need house rules everyone signs off on: what you may fetch, how the crawler behaves while it runs, and when it stops. This guide gives you those rules as a 14-line checklist and a go, ask or stop flow, with every line tied to a standard, a regulator’s own page or a clearly labelled house rule.
This is general research, not legal advice. It covers how you crawl; the legal questions live in my three legal guides, linked where they come up.
Short on time?
Short on time? Here's my take.
- Before any crawl
- Check robots.txt and terms, and log both
- Re-read robots.txt at least every 24 hours, and treat a 5xx on robots.txt as a full stop, as RFC 9309 says.
- Identifying your bot
- A named user agent with a URL and contact, plus a From header
- RFC 9110 asks robots to send From. Add Web Bot Auth if you crawl Cloudflare-fronted sites at scale.
- Collecting anything about people
- Minimise first, then ask counsel before AI-training use
- CNIL expects collection criteria set in advance and irrelevant data deleted.
- You hit a CAPTCHA, a block or an explicit opt-out
- Stop
- CNIL says to exclude sites that clearly oppose scraping, and Cloudflare de-verifies bots that evade site preferences.
Prices verified . Sponsors cannot buy a verdict or a position.
Here’s the ethical web scraping checklist #
Fourteen lines in three groups: before you crawl, while it runs, and after. In short, check robots.txt and the terms, identify your bot, back off when a site asks, minimise personal data, and stop at any block or opt-out.
Each line points to the source that backs it, and the sections below explain them.
| # | Line | Backed by |
|---|---|---|
| Before you crawl | ||
| 1 | Look for a licensed feed, API or dataset first | My house rule: cheaper and cleaner than a crawl |
| 2 | Read robots.txt for your bot’s token and for *; cache it no longer than 24 hours; a 5xx on robots.txt means stop |
RFC 9309 |
| 3 | Read the terms; anything behind a login, or terms you’ve accepted, means ask first | Joint statement, is web scraping legal |
| 4 | If the use is AI training or AI input, check Content Signals, TDMRep and vendor training tokens | Cloudflare, W3C TDMRep |
| 5 | If personal data is in scope, set collection criteria, excluded categories and retention before you start | CNIL |
| While it runs | ||
| 6 | Send a named user agent with a URL and contact, plus a From header | RFC 9110 |
| 7 | Publish what the bot does and its IPs; consider Web Bot Auth | Cloudflare policy, Web Bot Auth |
| 8 | Honor Crawl-delay where a site sets one; start at one connection per host (house rule) | Cloudflare policy, Anthropic |
| 9 | Back off on 429 and 503; wait as long as Retry-After says | RFC 6585, RFC 9110 |
| 10 | Never evade a block; a CAPTCHA or block page is a no | CNIL, Cloudflare policy |
| After | ||
| 11 | Delete any irrelevant data collected despite your criteria | CNIL |
| 12 | Log sources and timestamps per run, plus a robots.txt snapshot (house rule) | EDPB |
| 13 | Respect reuse limits: copyright and AI-training signals | AI training and the law |
| 14 | Give people a way to object to collection, and act on it | CNIL |
Sources verified 2026-11-16. Line 1, line 8’s connection limit and line 12’s robots.txt snapshot are my house rules, not standards.
Why another checklist? None of the four web scraping best-practice guides I read that rank for this topic covers AI-training opt-out signals. Bright Data’s and Grepsr’s don’t say how to identify your crawler, and Zyte’s checklist is seven legal checks you answer yes or no. This one is about how you crawl.
Before you crawl, what are you allowed to fetch? #
Start with robots.txt, then the terms, then anything behind a login. Each can turn a “go” into “ask” or “stop” before a single page is fetched.
robots.txt, read the way the standard says #
RFC 9309 is the robots.txt standard, and it opens with a warning: “These rules are not a form of access authorization.” A permissive file doesn’t hand you rights; it tells you what the site has asked crawlers to do. RFC 9309.
The RFC also sets the mechanics your crawler should follow. Match your product token case-insensitively; don’t use a cached copy “for more than 24 hours”; parse at least 500 kibibytes of the file. RFC 9309.
The two status rules matter most. If robots.txt returns a 4xx, the crawler may access any resource; if it returns a 5xx, the crawler “MUST assume complete disallow”. RFC 9309.
So a server error on robots.txt is a stop, not a free pass. Whether ignoring the file has legal consequences is a separate question, covered in is robots.txt legally binding.
Terms, logins and licensed alternatives #
Read the terms before you crawl, and treat anything behind a login as “ask first”. Privacy regulators have said “contractual terms cannot in and of themselves render data scraping lawful”, which cuts both ways: terms don’t make scraping fine, and they’re one part of the picture. Joint statement.
Some vendor guides define “Public web data” as “content accessible without logging in”. Bright Data docs. That’s a useful line for access, but it isn’t a privacy test, as the regulators’ section shows.
The legal side of terms, logins and access is in my guide to whether web scraping is legal, including what to write down before you start. And before any of this, ask whether the site offers a licensed API or dataset; I’d expect it to beat a crawl on both cost and risk.
How should your crawler introduce itself? #
With a stable name, a URL that explains it, and a way to reach you. A site owner who can see who you are can ask you to slow down instead of blocking you.
RFC 9110, the HTTP standard, says “A robotic user agent SHOULD send a valid From header field so that the person responsible for running the robot” can be contacted. It also says a user agent “SHOULD NOT generate a User-Agent header field containing needlessly fine-grained detail”. RFC 9110.
In practice, that looks like this. The bot name, URL and address are placeholders.
User-Agent: ExampleResearchBot/1.0 (+https://example.com/bot)
From: crawler-admin@example.com
The page at that URL should say what the bot collects, why, how to opt out, and which IP ranges it uses. That’s the same information the big AI crawlers publish about themselves, listed in my guide to AI crawler user agents.
Cloudflare’s verified-bot bar #
Cloudflare’s verified-bot policy, updated November 12, 2026, is the most concrete public yardstick I found. A bot must identify itself through a Web Bot Auth signature, “a published IP list with a stable user-agent”, or reverse DNS. Cloudflare policy.
It must also be a bot that “obeys robots.txt and crawl directives”, “maintains reasonable request rates” and isn’t seen “evading website owner preferences”. An AI crawler that doesn’t respect crawl-delay is a listed breach, the stated purpose must match the traffic, and Cloudflare can revoke verification across a whole organisation. Cloudflare policy.
You don’t need Cloudflare’s badge to crawl ethically. But if your bot couldn’t pass that bar, I’d ask why.
Web Bot Auth #
Web Bot Auth lets a bot sign its HTTP requests cryptographically, so a site can check the request really came from you. Cloudflare supports Ed25519 keys, published in a key directory at /.well-known/http-message-signatures-directory, with a Signature-Agent header on each request. Cloudflare Web Bot Auth.
It’s worth the setup if you crawl Cloudflare-fronted sites at scale. For a small, occasional crawl, a clear user agent, a From header and a published IP list cover the basics.

How fast is too fast? #
No standard says. RFC 9309 doesn’t define crawl-delay, and neither RFC 9110 nor RFC 6585 sets a rate, so any number you choose is a house rule. RFC 9309, RFC 6585.
That hasn’t stopped guides from printing numbers. DataCamp suggests “1 request every 3-5 seconds for smaller sites”, but gives no source for it. DataCamp. I wouldn’t treat that, or any figure, as a standard.
What you can follow are the signals sites actually send:
| Signal | What it means | What to do | Source |
|---|---|---|---|
Crawl-delay in robots.txt |
Not in the RFC, but expected by Cloudflare for AI crawlers and honored by ClaudeBot | Honor it | Cloudflare policy, Anthropic |
| 429 Too Many Requests | The client “has sent too many requests in a given amount of time” | Slow down; the response “MAY include a Retry-After header” | RFC 6585 |
| 503 with Retry-After | The service is temporarily unavailable | Wait as long as Retry-After says | RFC 9110 |
| No signal at all | The site hasn’t said | Start conservative, as a written house rule | None: no standard sets a rate |
Signals verified against the sources 2026-11-16.
Anthropic describes ClaudeBot as honoring “the non-standard Crawl-delay extension”. Anthropic. If a large AI crawler can honor it, a small one has no excuse not to.
My house rule, and it is only that: one connection per host to start, honor any Crawl-delay, and halve your pace on the first 429 or 503. Raise it only if the site stays healthy and nothing tells you to stop.
What about AI training and reuse signals? #
Check them whenever the data will train or feed an AI model, because robots.txt allow rules aren’t the whole answer. Three newer signals and the vendor training tokens can each say no.
| Signal | Who’s behind it | Status on 2026-11-16 | How it’s expressed |
|---|---|---|---|
| Content Signals | Cloudflare (Sept 24, 2025), released CC0 | A published policy, not a standard | A line such as Content-Signal: search=yes, ai-train=no; categories search, ai-input and ai-train |
| TDMRep | W3C Community Group final report (May 10, 2024) | “not a W3C Standard nor is it on the W3C Standards Track” | tdm-reservation 1 (reserved) or 0, in /.well-known/tdmrep.json, an HTTP header or an HTML meta tag, with an optional tdm-policy URL |
| AIPREF | IETF working group | Active; drafts only, no RFC yet | A vocabulary for AI-use preferences, attached via robots.txt, well-known URIs or HTTP headers |
| Vendor training tokens | Individual AI companies | Live | robots.txt rules for tokens such as GPTBot |
Sources: Cloudflare Content Signals, W3C TDMRep, IETF AIPREF, OpenAI, verified 2026-11-16.
With Content Signals, an omitted category means no preference. Cloudflare says the signals are “not technical countermeasures against scraping”, and its policy text frames the restrictions as express reservations of rights under Article 4 of the EU’s Directive 2019/790. Cloudflare. That legal effect is Cloudflare’s claim; I found no court ruling on it.
AIPREF’s milestone for sending its vocabulary and attachment drafts onward was August 31, 2026, and the group’s page still shows them as drafts. IETF AIPREF. Anthropic’s crawler page says that as of October 30, 2026 it has begun recognizing a machine-readable “no AI training” signal. Anthropic.
How sites write these opt-outs is covered in the AI training opt-out in robots.txt. Whether you may train on the content at all is a legal question: is scraping for AI training legal covers text-and-data-mining rules, Article 4 and fair use.
In the EU, the Commission describes its General-Purpose AI Code of Practice as “a voluntary tool”, with a copyright chapter meant to help show compliance with Article 53 of the AI Act. European Commission. I couldn’t open the chapter text itself, so I don’t describe its crawler commitments here.
What do privacy regulators expect? #
That you treat public personal data as personal data. The regulators I read all start from that point, then ask you to collect less and delete sooner.
| Regulator | Page date | What it asks |
|---|---|---|
| 16 privacy regulators, joint statement | October 2024 | “publicly accessible personal data is still subject to data protection and privacy laws in most jurisdictions” |
| CNIL (France) | January 5, 2026 | “define, in advance, specific collection criteria”; exclude unneeded data; delete irrelevant data; skip sites that oppose scraping |
| EDPB (EU) | July 8, 2026, consultation draft | Special-category data is “in principle prohibited” without a legal basis and an exception; reliable sources, timestamps, validation |
| ICO (UK) | Not shown | Personal data is “any information relating to an identified or identifiable natural person” |
| California Attorney General (CCPA) | Updated August 28, 2026 | “Personal information does not include publicly available information”, in defined senses; applies only above thresholds |
Sources: joint statement, CNIL, EDPB, ICO, California AG, verified 2026-11-16.
The joint statement #
Published in October 2024 on the site of Canada’s privacy commissioner, it is endorsed by “a total of 16 co-signatories”, including Australia, Canada, the UK, Switzerland, Norway, New Zealand and Hong Kong. It followed an initial statement of August 24, 2023. Its two lines worth pinning above your desk are the public-data line in the table and “contractual terms cannot in and of themselves render data scraping lawful”. Joint statement.
CNIL’s measures #
CNIL’s focus sheet is the most practical of the set. It asks controllers to set collection criteria in advance, exclude data categories they don’t need, delete irrelevant data, and automatically exclude irrelevant sensitive data. CNIL.
It also says to “exclude from the collection websites that clearly oppose the scraping of their content”, naming robots.txt and CAPTCHAs as the signs. As additional safeguards, it suggests publishing the list of sites you scrape and offering a prior right to object. It also mentions “do-not-train” registries that site publishers can join. CNIL.
The EDPB draft #
The EDPB adopted web-scraping guidelines for generative AI for public consultation “until 30 October 2026”. They say special-category data is “in principle prohibited” and needs both an Article 6 basis and an Article 9(2) exception, and they recommend reliable sources, timestamps and validation. EDPB.
I found no final version after the consultation closed, so treat the text as a draft that may change. The announcement I read doesn’t mention robots.txt.
The ICO and California #
The UK’s definition is broad: identifiers include “a name, an identification number, location data, an online identifier”. A scraped username can be personal data. ICO.
California is narrower. The CCPA excludes publicly available information, such as government records and information a business reasonably believes was lawfully made available to the general public. California AG.
It applies only to for-profit businesses in California that meet one threshold: revenue over $25 million, buying, selling or sharing the personal information of 100,000 or more California residents or households, or earning 50% or more of revenue from selling it. That’s per the AG’s FAQ; check the CPPA for inflation-adjusted figures. California AG.

Do user-triggered fetches play by the same rules? #
Some AI companies say no. OpenAI says that for ChatGPT-User “robots.txt rules may not apply”, and Perplexity says “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.” OpenAI, Perplexity.
Their reasoning is that a person asked for that one page, so it’s not a crawl. That’s a defensible line for a single fetch on a user’s behalf.
My rule is simpler: anything automated at scale follows robots.txt, whoever triggered it. If your tool fetches one page because a user pasted a link, the user-fetch argument may hold; if it fans out across a site, it’s a crawl, and the checklist applies.
Go, ask or stop? #
Run every new target through three questions in this order: is there a stop signal, is there an ask signal, and only then go.
| Decision | When | What happens next |
|---|---|---|
| Stop | robots.txt disallows you; robots.txt returns a 5xx; a CAPTCHA or block page; an explicit opt-out; sensitive personal data | Don’t crawl. Log why. Revisit only with written permission |
| Ask | Terms are unclear or restrict automated access; personal data at scale; AI-training use; anything behind a login | Get permission from the site, or review with counsel, before any request |
| Go | robots.txt allows you; no personal data, or it’s minimised; public pages, no login; reuse is permitted | Crawl, with the identification, pacing and logging lines on |
Built from the checklist’s sources: RFC 9309, CNIL, Cloudflare policy, joint statement.
Stop beats ask, and ask beats go. One stop signal ends the decision, however many go signals there are.
If your crawl goes through proxies, how those IPs were sourced is an ethics question of its own, covered in ethical residential proxies.
What I would do #
I’d write the 14 lines into the repository as a CRAWLING.md file that every crawler change gets reviewed against. Then I’d put the bot’s info URL and a contact address in the user agent and a From header from day one.
On every run, I’d snapshot robots.txt alongside the data, so the log shows what the site said on the day. And on anything ambiguous, I’d default to stop until someone with the authority to say yes has said it in writing.
For the legal review that sits behind this checklist, start with what US, EU and Indian law says about scraping. The data hub collects the rest of what I’ve written on collecting and controlling web data.
FAQ
Is web scraping ethical if the data is public?
Public isn't a free pass. Sixteen privacy regulators said in a 2024 joint statement that publicly accessible personal data is still subject to data protection and privacy laws in most jurisdictions. Check robots.txt, terms, personal data and how you'll reuse the content.
Do I have to follow robots.txt?
Ethically, I'd say yes, and Cloudflare won't verify a bot that ignores it. RFC 9309 itself says the rules are not a form of access authorization, so the legal question is separate; my post on whether robots.txt is legally binding covers it.
What should my scraper's user agent say?
A stable product token for your bot, a URL that explains what it does, and a contact. RFC 9110 says a robotic user agent should also send a From header so the person running it can be contacted, and should avoid needlessly fine-grained detail in the User-Agent.
How many requests per second is polite?
No standard sets a number. Honor Crawl-delay where a site sets one, slow down on 429 or 503 responses and wait as long as Retry-After says. Any starting rate you pick is a house rule, so write it down and keep it conservative.
Can I scrape for AI training if robots.txt allows it?
Not on robots.txt alone. Check AI-specific signals such as Content Signals, TDMRep and vendor training tokens, then the copyright and privacy rules for your use. My guide on whether scraping for AI training is legal covers the law.
Does GDPR apply to scraped public profiles?
In general, yes, according to EU and UK regulators. The ICO defines personal data as any information relating to an identified or identifiable person, and France's CNIL expects scrapers to set collection criteria in advance and delete irrelevant data. This is general research, not legal advice.
Sources
21 sources, accessed .
Official pricing, docs, changelogs and status pages 17
- IETF RFC 9309: Robots Exclusion Protocol (Sept 2022)
- IETF RFC 9110: HTTP Semantics (From, User-Agent, Retry-After)
- IETF RFC 6585: status code 429 Too Many Requests
- Cloudflare: verified bots policy (updated Nov 12, 2026)
- Cloudflare: Web Bot Auth (updated Oct 8, 2026)
- Anthropic: does Anthropic crawl data from the web?
- OpenAI: crawlers and user agents
- Perplexity: crawlers
- Cloudflare: Content Signals Policy (Sept 24, 2025)
- W3C Community Group report: TDM Reservation Protocol (May 10, 2024)
- IETF AIPREF working group
- CNIL: focus sheet on web scraping (Jan 5, 2026)
- Concluding joint statement on data scraping by privacy regulators (Oct 2024)
- EDPB: web-scraping guidelines for generative AI out for consultation (Jul 8, 2026)
- ICO: what is personal data?
- California Attorney General: CCPA FAQ (updated Aug 28, 2026)
- European Commission: General-Purpose AI Code of Practice


