Skip to content
CrawlfolioSourced research on web data for AI
Menu

llms.txt is a proposed markdown file at the root of a site that lists the pages an AI agent should read first. No major AI company says in its own docs that its crawler reads yours, and Google says Search ignores it.

So the decision is simple: publish one if developers and coding agents use your docs, and skip it if you’re a content or marketing site hoping for AI search traffic. Below is what the spec says, what each AI company says, what server logs show, and how to check your own.

Short on time?

Short on time? Here's my take.

Docs or API site used by developers and coding agents
Publish it, generated from your docs build
Cheap to add. In Ahrefs' log data the top AI fetchers were GPTBot and Claude-Code, a coding agent, which fits this use.
Blog, store or marketing site wanting AI search visibility
Skip it
Google says Search ignores the file, and SE Ranking found no link to AI citations. Spend the time on crawlable HTML and robots.txt rules.
You already have one
Keep it, and never let it return a server error
A server error is the only thing that fails Chrome's Lighthouse audit. Keep it in sync with your docs.
You want to know if it's used
Check your own logs first
Count /llms.txt requests by verified user agent before you believe anyone's adoption number.

Prices verified . Sponsors cannot buy a verdict or a position.

What’s actually in an llms.txt file? #

A short markdown document with one required line: an H1 with the name of the project or site. The proposal describes itself as “A proposal to standardise on using an /llms.txt file to provide information to help agents use a website”. llmstxt.org.

Jeremy Howard published it on September 3, 2024, and the page was last modified on August 10, 2026. It now says plainly, “This is v2 of the proposal”. llmstxt.org.

After the H1, everything is optional. The spec allows a blockquote summary, a few paragraphs of detail, then H2 “file list” sections where each line is a markdown link in the form [name](url): notes. An H2 called “Optional” holds links an agent can skip. llmstxt.org.

Here is a minimal example I wrote from the spec’s format. The site and URLs are made up.

# Example API

> Example API returns weather data as JSON. These links cover setup, auth and every endpoint.

Requests need an API key in the Authorization header.

## Docs

- [Quick start](https://example.com/docs/quickstart.md): install the SDK and make a first call
- [Authentication](https://example.com/docs/auth.md): keys, scopes and rotation
- [Endpoints](https://example.com/docs/endpoints.md): every route with parameters

## Optional

- [Changelog](https://example.com/changelog.md): release notes

What changed in v2? #

Four things matter if you’re writing one today. A file can now sit at a subpath such as /docs/llms.txt and covers the URLs under that path, and “where more than one file applies, agents should use the most specific one”. llmstxt.org.

Pages should offer a markdown copy at the same URL as page.html.md or, new in v2, page.md. Agents can discover those copies through rel="alternate" type="text/markdown" and rel="describedby", either in a <link> tag or an HTTP Link: header. llmstxt.org, v2 changes.

v2 also removed the llms_txt2ctx context-expansion tooling, and the “Optional” section lost its mechanical meaning. v2 changes.

Where does llms-full.txt fit? #

It doesn’t, as far as the spec goes. Neither the main proposal nor the v2 change log mentions llms-full.txt. llmstxt.org, v2 changes.

It’s a convention some docs platforms generate anyway. Mintlify, for example, creates /llms.txt, /llms-full.txt and .md page copies automatically for the docs it hosts. Mintlify.

Mintlify also says the format “was developed by Mintlify in collaboration with Anthropic and is now part of the official llms.txt proposal”. The v2 spec I read doesn’t mention it, so I treat it as a platform convention. Mintlify, llmstxt.org.

How is llms.txt different from robots.txt and a sitemap? #

robots.txt tells crawlers what they may fetch, a sitemap lists URLs for search engines to find, and llms.txt is a curated reading list. Only robots.txt carries crawl rules; llms.txt controls nothing.

llms.txt robots.txt XML sitemap
What it’s for A curated, markdown reading list of key pages for agents Crawl rules per user agent A list of URLs for search engines to discover
Who says they read it No AI crawler’s docs say so; Chrome’s Lighthouse audits it The crawlers that document following it Search engines
Does it allow or block anything? No It states rules; compliance is up to the crawler No
Format Markdown Plain-text directives XML

The llms.txt column follows the spec and the vendor pages in the next section, verified 2026-11-16.

If you want to stop AI training, llms.txt is the wrong file. The opt-out lives in robots.txt, covered in my post on the AI training opt-out in robots.txt, and whether those rules carry legal weight is in is robots.txt legally binding.

Two pale beige and off-white surfaces meeting along a soft, wavering diagonal line that runs corner to corner
Two plain surfaces share one edge without either fencing the other off; llms.txt can sit beside robots.txt at your site's root, but only robots.txt states rules a crawler is asked to follow. Photo: N N / Unsplash · Unsplash License

Who publishes an llms.txt file? #

Mostly developer docs. OpenAI, Anthropic, Google’s Gemini API and the Model Context Protocol all publish an llms.txt for their docs, and I opened each of them.

Publisher File What’s in it
OpenAI developers.openai.com/llms.txt H1 “OpenAI Developers”, a blockquote summary, about 41 links
Anthropic platform.claude.com/llms.txt (docs.claude.com/llms.txt redirects here) H1 “Anthropic Developer Documentation”, no blockquote, 850+ links, about 102,000 characters
Google (Gemini API) aistudio.google.com/llms.txt (ai.google.dev/api/llms.txt redirects here) H1 “Gemini API & AI Studio Documentation”, roughly 400 links
Model Context Protocol modelcontextprotocol.io/llms.txt H1 “Model Context Protocol”, no blockquote, about 360 links

All four files opened and verified 2026-11-16. Link counts are approximate.

Notice the spread. OpenAI’s file is a short index; Anthropic’s runs to 850+ links, which is closer to a full table of contents than a reading list. The spec doesn’t set a size limit, so both are valid.

The spec site claims “thousands of sites publish an llms.txt file” and lists Mintlify, GitBook, Yoast SEO, AIOSEO and Wix as integrations. That’s its own count, and it names no AI company as a reader. llmstxt.org.

Publishing is not the same as reading. Every company in the table above publishes a file for its own docs, and none says its crawler reads yours.

Do AI companies say they read llms.txt? #

None of them commits to reading other sites’ llms.txt files. I went through each company’s crawler or search documentation and recorded what it says, word for word where it says anything.

Company or product What its own docs say about llms.txt Source
Google Search Ignores it: creating one “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.” AI optimization guide
Google Search (AI features) “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.” AI features page
Chrome Lighthouse Audits it; flags a page only “if a server error occurs when attempting to retrieve the llms.txt file” Lighthouse audit
OpenAI Its bots page doesn’t say its crawlers read llms.txt. Its only mention: “For the complete documentation index, see llms.txt”, meaning its own OpenAI bots
Anthropic Its crawler article for ClaudeBot, Claude-User and Claude-SearchBot doesn’t mention llms.txt Anthropic
Perplexity Its bots page doesn’t say its bots read llms.txt; the header points only to its own docs index Perplexity bots
Microsoft (Bing, Copilot) No primary statement found n/a

All pages read and verified 2026-11-16.

The Microsoft row is a gap, not an answer. I found only secondary reports of a Bing representative advising against LLM-only pages, and no Bing page I could cite.

A missing statement isn’t proof that a company doesn’t read the file. It does mean nobody has promised to, so you can’t plan around it.

The bot names in that table, and how to tell a real one from a spoofed one, are in my list of AI crawler user agents.

Does anything actually fetch the file? #

Very little, going by the three log studies I read. The largest, from Ahrefs, found that 97% of sites with an llms.txt got zero requests for it in May 2026. Ahrefs study.

Study Who ran it Sample Window Finding Caveat
llms.txt study Ahrefs (Louise Linehan) 137,210 domains with traffic; 38,360 (28%) had a valid file May 2026 97% of files got zero requests; the other ~3% (about 1,100 domains) got all ~22,000 requests Ahrefs Web Analytics customers; sample skews technical
Crawler census dejan.ai (Dan Petrovic) One site 20 days from July 27, 2026 llms.txt fetches: Google 3, Anthropic 1 (ClaudeBot), OpenAI 0; robots.txt fetches: 1,150 One site only; its table counts 243 Google robots.txt fetches, its text 242 from Googlebot
llms.txt and AI citations SE Ranking (Yulia Deda) About 300,000 domains; 10.13% had the file Page dated Oct 15, 2026 A model of AI citations got better when the llms.txt variable was removed Doesn’t say where the citation data came from
Limy and Otterly data Via an Optimizely summary Limy: 500M+ AI bot visits; Otterly: one site, 62,100+ AI bot visits 90 days each Limy: “Only 408 requested llms.txt directly”; Otterly: 84 visits, about 0.1% Second-hand: I didn’t open the primaries

Studies read and verified 2026-11-16.

What “valid” means in the Ahrefs number #

Ahrefs’ study page defines it: the root /llms.txt returns a 200, the body is markdown rather than HTML, and soft-404 pages are screened out. It didn’t check whether files follow the spec, and Ahrefs says its technical-leaning sample makes 28% an upper bound. Ahrefs study.

That definition matters because Ahrefs’ own explainer repeats the 28% and 97% figures without defining “valid”. Ahrefs explainer.

Who made the requests that did happen? #

Mostly not AI. Ahrefs found 96% of requests came from bots, but only 19.5% of requests to fetched files came from AI bots, while SEO audit tools made 21.7%. Ahrefs study.

Among AI bots, GPTBot was the top fetcher at 4.51% of all requests, and Claude-Code, a coding agent, came second. Lighthouse made 22 requests, and Googlebot’s roughly 900 fetches looked like routine crawling. No AI bot requested /llms.txt on domains where it returned a 404. Ahrefs study.

Ahrefs calls every figure in its study “a ceiling on actual llms.txt consumption”: a request shows a bot downloaded the file, not that anything used it. That’s the right way to read every number in the table.

Does Google use llms.txt at all? #

Not for Search. Three different parts of Google touch the file: Search ignores it, Chrome’s Lighthouse audits it, and the Gemini API docs publish one.

Search ignores it. On June 15, 2026, Google’s Search Central update log “Added a note to the AI optimization guide clarifying Google Search’s usage of llms.txt files”. The note says the files “aren’t needed for Google Search” “(and won’t negatively or positively impact your visibility or rankings)”. Search Central updates.

The guide itself is blunter: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search”. AI optimization guide.

The same June note adds that “it’s fine if you want to maintain these files for other services or systems that use them”. That’s permission, not a recommendation. Search Central updates.

Lighthouse audits it. Chrome’s Lighthouse has an llms.txt audit under “agentic browsing”. It flags a page only when fetching the file causes a server error; a 404 makes the audit “Not Applicable (N/A)” because the file is optional for now. Lighthouse audit.

Google’s developer docs publish one. The Gemini API’s file at aistudio.google.com is in the publishers table above. A CMS change also put llms.txt on Google’s developer docs, and Search removed it from its own docs, as Search Engine Roundtable reported in January 2026. aistudio.google.com/llms.txt, Search Engine Roundtable.

Asked whether that was an endorsement, John Mueller answered, “to be direct, no.” Earlier he said of the files on Google sites, “It’s safe to assume that they’re there for other purposes.” Both quotes come via news coverage; I didn’t open his original posts. SER, endorsement, SER, other purposes.

Some ranking pages still lead with an older, undated Mueller line, “no Gen AI system currently uses llms.txt”. Similarweb, Dec 2025. It’s consistent with today’s position, but the June 2026 documentation is the primary source I’d quote.

Slanted panels of bright window light falling across a dusty pink, pitted concrete wall
Light from one window breaks into separate panels on the wall, much as Google splits on this file: Search ignores it, Lighthouse audits it and the Gemini API docs publish one. Photo: Bernard Hermant / Unsplash · Unsplash License

How can you see if bots read yours? #

Count requests for /llms.txt in your access logs, grouped by user agent. This is the only adoption number that applies to your site.

Assuming your server writes the common combined log format, where the user agent is the sixth quoted field, a one-liner like this does it. I wrote it from the log format, not from a run on real logs.

grep '"GET /llms.txt' access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

Three rules for reading the result:

  • Verify the bot. Anyone can type “GPTBot” into a user agent. Check the IP with the method the operator documents, listed in my guide to verifying AI crawler user agents.
  • A fetch is a ceiling. As Ahrefs puts it, a request proves a download, not use.
  • Compare against robots.txt. In dejan.ai’s single-site census, robots.txt drew 1,150 fetches over 20 days while llms.txt drew four. dejan.ai.

Give it at least a month. One quiet week tells you little.

So should you add one? #

Add one if your docs are read by developers and coding agents; skip it on a content or marketing site. The cost-benefit lands differently for each.

Your site My pick Why
Docs or API reference Publish, generated from the docs build, with .md page copies The labs publish their own for developer docs, and the top AI fetchers Ahrefs saw were GPTBot and Claude-Code
Blog, publisher or marketing site Skip it Google Search ignores it; SE Ranking found no link to AI citations
Store or product catalogue Skip it Same as above, and nobody documents reading it
You already publish one Keep it, fix any server errors, keep it in sync A server error is the only Lighthouse failure; a stale file points agents at dead links

Picks based on sources verified 2026-11-16.

On a docs site, generate the file in the same build that publishes your docs, so it never drifts. Mintlify does this for the docs it hosts, and the spec lists GitBook, Yoast SEO, AIOSEO and Wix as integrations too. Mintlify, llmstxt.org.

Keep the H1, add a one-line blockquote, and group links by task. If a whole section such as /docs/ is the useful part, v2 lets you place the file there instead of the root.

Serve .md copies of the linked pages too. If you also want to convert other people’s pages into markdown, that’s a different job, covered in website to markdown for LLMs, and for the retrieval side see web scraping for RAG.

On a content site, the June 2026 Google note settles the Search question, and SE Ranking found “There’s no correlation between AI citations and LLMs.txt.” SE Ranking. Mintlify, which sells generation, quotes the core criticism itself: “no major AI company officially supports the format”, while noting that Anthropic partnered with it on its own docs file. Mintlify.

What I would do #

If I managed a docs or API site, I’d generate llms.txt and .md copies in the docs build this week. Then I’d check the logs after 30 days for /llms.txt requests from verified agents, and keep the file only if it’s cheap to maintain, which it should be.

On a blog or marketing site, I wouldn’t bother. I’d spend the hour on crawlable HTML and clear robots.txt rules for the bots I care about.

I’d revisit that call the day any AI company documents reading third-party llms.txt files. Until then, the crawl APIs hub covers the tools on the other side of this file: the crawlers and converters that fetch your pages.

FAQ

Does Google use llms.txt?

Not for Search. Google's AI optimization guide says Google Search ignores llms.txt files, and a June 15, 2026 Search Central update added that note. Chrome's Lighthouse does audit the file, but it only flags a server error, and Google's own Gemini API docs publish one.

Does ChatGPT or Claude read my llms.txt?

Their crawler docs don't say so. OpenAI's and Anthropic's crawler pages never say their bots read other sites' llms.txt files. In Ahrefs' May 2026 log data, GPTBot and the Claude-Code agent were the top AI fetchers of the file, but only a small share of sites saw any request at all, and a fetch is not proof of use.

Is llms.txt an official standard?

No. It's a proposal by Jeremy Howard, first published September 3, 2024 and now at v2 (August 10, 2026). It isn't an IETF or W3C standard.

What's llms-full.txt?

A convention some docs platforms generate alongside llms.txt, usually one file with the full text of the docs. Mintlify, for example, creates it automatically and says it developed the format with Anthropic. The llms.txt v2 spec itself doesn't mention it.

Can llms.txt block AI training?

No. llms.txt is a reading list and controls nothing. To opt out of AI training, use robots.txt rules for the training crawlers and the newer training opt-out signals.

Will an llms.txt hurt my rankings?

Not in Google Search. Google says creating one will neither harm nor help your visibility or rankings, because Google Search ignores the file.

Sources

22 sources, accessed .

Official pricing, docs, changelogs and status pages 13

  1. llms.txt proposal, v2 (modified 2026-08-10) llmstxt.org · accessed
  2. llms.txt: changes in v2 llmstxt.org · accessed
  3. Google Search Central: documentation updates (June 15, 2026 llms.txt note) developers.google.com · accessed
  4. Google Search Central: AI optimization guide developers.google.com · accessed
  5. Google Search Central: AI features and your website developers.google.com · accessed
  6. Chrome for Developers: Lighthouse llms.txt audit developer.chrome.com · accessed
  7. OpenAI: crawlers and user agents developers.openai.com · accessed
  8. Anthropic: does Anthropic crawl data from the web? support.claude.com · accessed
  9. Perplexity: crawlers docs.perplexity.ai · accessed
  10. OpenAI Developers llms.txt developers.openai.com · accessed
  11. Anthropic Developer Documentation llms.txt platform.claude.com · accessed
  12. Gemini API and AI Studio llms.txt aistudio.google.com · accessed
  13. Model Context Protocol llms.txt modelcontextprotocol.io · accessed

Independent benchmarks and published studies 3

  1. Ahrefs: llms.txt study, 137,210 domains (June 15, 2026) ahrefs.com · accessed
  2. dejan.ai: crawler census, one site, 20 days (Aug 15, 2026) dejan.ai · accessed
  3. SE Ranking: llms.txt and AI citations, ~300,000 domains (page dated Oct 15, 2026) seranking.com · accessed

Other blogs 4

  1. Ahrefs: what is llms.txt (updated June 15, 2026) ahrefs.com · accessed
  2. Optimizely: does llms.txt work? (summary of Limy and Otterly data) optimizely.com · accessed
  3. Mintlify: what is llms.txt mintlify.com · accessed
  4. Similarweb: llms.txt, reality vs hype (Dec 10, 2025) aisearch.similarweb.com · accessed

What readers say

Used one of the tools in this guide? Tell other readers how it went. I read every review before it goes up, and I never turn one down for being negative. How reviews work.

No reader reviews yet. Yours could be the first.

Write a review
Your rating

What did you use it for, what worked, and what didn't? 30 to 1500 characters. Please don't include links.

Your connection to the company

Your review appears with the name you enter, after I approve it. I don't ask for your email. Privacy.

  1. Explainer

    How to crawl an entire website for your LLM

    Find every URL, keep the crawl in scope and price it first. Settings and costs for Firecrawl, Apify, Crawl4AI, Cloudflare and Spider, checked Nov 2026.

    14 min read