Explainer
Is robots.txt Legally Binding? Why Ignoring It Is Risky
Is robots.txt legally binding? It's a crawler protocol, not a permission slip. What US cases, EU rules and draft GDPR guidance say. Not legal advice.
robots.txt is a technical crawler protocol, not a universal law or permission grant. I would still treat a restriction as a reason to stop and review the proposed collection: terms, access facts, EU rights reservations and personal-data expectations can make the surrounding signal relevant. RFC 9309, DSM Directive Article 4, EDPB consultation draft.
This is general research, not legal advice. A file alone cannot resolve the enforceability of a particular restriction or your right to reuse the content.
Short on time?
Here's my take
- Short answer
- Protocol plus legal context
- A parser result is not a universal legal permission or liability rule.
- Crawler operator
- Honor restrictions and resolve rights
- Check terms, reservations, privacy and intended reuse before collection.
- Site owner
- Use real controls for confidential data
- A public crawler file cannot secure private documents.
Prices verified . Sponsors cannot buy a verdict or a position.
What does robots.txt actually do? #
It communicates crawler rules. RFC 9309 expressly distinguishes the protocol from access authorization and from security; its normative requirement words specify protocol behavior rather than automatically create civil or criminal liability. RFC 9309, sections 1–3.
Google’s documentation also separates crawling from indexing. A disallowed URL can still appear in search results, and robots.txt cannot force an uncooperative bot to comply or protect confidential material. Google’s limitations explanation.
| Signal or control | What I would read it as | What I would not infer |
|---|---|---|
| Disallow rule | A crawler restriction to evaluate under the relevant protocol and context | One automatic legal outcome worldwide |
| Allow rule or no matching rule | A protocol result for that crawler and path | A copyright licence or privacy consent |
| Published terms | Separate statements requiring contract and notice review | A fact already decided by the robots parser |
| Authentication or effective restriction | A separate access and security boundary | Something a permissive robots rule overrides |
| Training-related crawler token | A product-specific documented control | Universal exclusion from every search index |
The protocol and product rows follow RFC 9309, Google’s interpretation and Google’s crawler documentation. The separate legal questions follow the limitations in Van Buren and the EU rights-reservation provisions.
What happens when Allow and Disallow overlap? #
For the same crawler, the more specific matching path takes precedence in the example below. This is hypothetical policy for a domain you own, not instructions to collect another site’s material. RFC 9309 matching rules.
User-agent: ExampleBot
Disallow: /drafts/
Allow: /drafts/public-summary/For ExampleBot, this permits /drafts/public-summary/ at the protocol level while the broader drafts path remains disallowed. Rules have host, protocol and port scope; this example does not decide content reuse rights. RFC 9309 matching rules, Google scope and rule precedence.
I would verify the actual crawler’s documentation before extrapolating behavior. Google documents different crawler tokens and purposes, including Google-Extended; a training-related control should not be treated as a universal search-index switch. Common crawler documentation, checked November 11, 2026.
Does ignoring robots.txt automatically break the CFAA? #
The sources I read do not establish that automatic rule. The CFAA’s authorization questions have statutory elements; Van Buren distinguished access to entitled information from improper purpose, while expressly leaving some questions about technical versus contractual limits open. 18 USC §1030, Van Buren, footnote 8.
The Ninth Circuit’s 2022 hiQ decision concerned a preliminary injunction involving public profiles. It left other claims available, and the later December 2022 consent judgment imposed $500,000 and a permanent injunction against hiQ. Neither step should be reduced to “robots is always enforceable” or “public data is unrestricted.” hiQ opinion, consent judgment. The US section of my scraping legality guide reads both cases in more detail.
An effective access control also raises a separate anti-circumvention question. I would seek specific review of the actual access method and restriction rather than treat a permissive or absent robots file as permission to bypass it. 17 USC §1201.
One court has addressed robots.txt under that statute directly. In a December 15, 2025 pleading decision in the OpenAI copyright litigation, Judge Stein held that Ziff Davis had not alleged that robots.txt files are a technological measure that effectively controls access under section 1201(a), or that disregarding them is circumvention; he compared them to a “keep off the grass” sign. That is one district court’s ruling on one complaint under one statute, not a holding on contract, copyright infringement, EU rights reservations or privacy. Opinion, pp. 6–7.
Does robots.txt matter for AI training in the EU? #
Yes, the surrounding legal and policy context can make crawler signals relevant. Article 4 of the DSM Directive conditions its broader text-and-data-mining exception on lawful access and the absence of an appropriate express rights reservation, including machine-readable means for publicly available online content. That does not establish that every Disallow rule conclusively reserves every right for every use. Directive 2019/790, Article 4.
The Commission’s GPAI Code is a voluntary tool for compliance with underlying AI Act duties. Its copyright chapter commits signatories to respect effective access restrictions and to use crawlers following RFC 9309 and other appropriate rights-reservation protocols. I would not confuse voluntary adherence to the Code with the status of the underlying law. Commission explanation, Copyright chapter, Measures 1.2–1.3.
My recommendation is to review the actual signal, work, claimant, use and national implementation with counsel. A general protocol explanation cannot certify whether one particular rights reservation meets all applicable conditions. For the rest of the training question (acquisition, retention, outputs and the Article 3 research route), see my guide to whether scraping for AI training is legal.
Is a robots.txt Allow the same as GDPR consent? #
No. The EDPB’s draft scraping guidance says missing restrictions do not equal GDPR consent; it also treats restrictions as potentially relevant to reasonable expectations. Legitimate interest requires the cumulative analysis described in that draft. Guidelines 03/2026 version 1, paragraphs 43–47 and 64–72.
The consultation’s feedback period closed on October 30, 2026, and I found no adopted final version on November 11. This is not a final new statute, and the EDPB’s earlier AI-model opinion already called for case-specific analysis of anonymity and legitimate interest. Consultation status, Opinion 28/2024 announcement.
For a collection involving people, I would record the personal-data purpose and basis separately from the crawler’s parser result. An empty file is not a signed consent form.
Do the Google v. SerpApi filings settle it? #
No. On November 9, 2026, the court granted SerpApi’s motion to dismiss Google’s amended DMCA anti-circumvention claims, with leave to amend within 21 days. The order turns on whether Google adequately alleged copyrighted content behind its SearchGuard measure and the copyright owners’ authority for it; it does not mention robots.txt or announce a universal robots rule. ECF 62, pp. 1, 5–7 and 19.
The court had vacated the October 13 hearing and decided the motion on the papers; Google’s October 12 statement of a recent decision was a party filing. ECF 60, ECF 61. I read these PDFs on November 11, not the complete docket, so a later amended complaint or ruling remains a release-day check. See the case-specific Google guide for its separately dated context.
I would keep protocol behavior, a litigant’s allegation and a court’s actual holding in different columns. Substituting one for another is the source of many overconfident answers to this question. The New York Times lawsuit against Brave is a fresh case to file that way: its complaints never mention robots.txt, even though Brave says its crawler doesn’t advertise a user agent of its own.
What should site owners and crawler operators do? #
For a site owner, I would write clear crawler rules and keep confidential content behind real access controls. Google’s documentation explains why robots.txt cannot provide confidentiality or reliably remove already known URLs from results. Google robots limitations. If your site sits behind Cloudflare, I explain its Search, Agent and Training bot categories and what changed on September 15.
For a crawler operator, I would honor restrictions and preserve the dated policy, applicable terms, intended use and permission record. If a restriction or a rights reservation conflicts with the intended collection, I would pause and resolve it rather than infer permission from public visibility. This is my operating recommendation, not a universal legal determination.
This guide does not report a test of any vendor’s crawler. Nothing here is an observed vendor result, a court finding or proof that a crawler obeyed a live target, so no first-hand vendor compliance claim follows from it.
The data hub, general scraping legal guide, AI-training guide and crawl API pillar cover the adjoining decisions. I would choose a crawler after resolving what it may collect and how its results may be used.
FAQ #
Is robots.txt a law? #
No. It is a technical protocol. Legal obligations depend on applicable law, terms and facts; the protocol does not itself grant access authorization. RFC 9309.
Can I ignore robots.txt since it’s voluntary? #
I would not treat that as a safe rule. EU rights reservations, code commitments and personal-data expectations can make a signal relevant; the actual facts need review. DSM Directive, GPAI Code chapter, draft EDPB guidance.
If robots.txt allows it, can I train a model on the content? #
No. A crawler rule does not itself grant copyright or personal-data rights. Review the applicable mining exception, licence, reservations and privacy basis separately. Directive Article 4, EDPB consultation draft.
Will Disallow keep a page out of Google? #
Not necessarily. Google says disallowed URLs can still appear in search results; crawl restriction and indexing control are separate. Google robots limitations.
Does Google-Extended block my site from Google Search? #
No. Google documents different tokens and purposes. Read the current product-specific control rather than assume a training token excludes every crawler or search index. Google common crawlers.
Can robots.txt protect private documents? #
No. Google warns that robots.txt cannot force bot compliance or protect confidential information. Use real access controls for private material. Google guidance.
Sources
19 sources, accessed to .
Official pricing, docs, changelogs and status pages 16
- RFC9309 Robots Exclusion Protocol
- Google robots introduction
- Google REP interpretation
- Google common crawlers
- Van Buren v United States, Supreme Court2021
- hiQ v LinkedIn, NinthCircuit2022
- hiQ consent judgment ECF406
- Google v SerpApi ECF60
- Google v SerpApi ECF61
- Google v SerpApi ECF62, order granting dismissal with leave to amend, 9 Nov 2026
- EU DSM Directive2019/790
- EDPB Opinion28/2024 AI-models announcement
- EDPB03/2026 consultation status
- EDPB03/2026 version1 PDF
- European Commission GPAI Code current page
- GPAI Code Copyright chapter
