Explainer
What is llms.txt, and does any AI crawler read it?
llms.txt is a proposed markdown file for AI agents. Google Search ignores it, logs show few bot fetches, and I'd add one only on a docs or API site.

llms.txt is a proposed markdown file at the root of a site that lists the pages an AI agent should read first. No major AI company says in its own docs that its crawler reads yours, and Google says Search ignores it.
So the decision is simple: publish one if developers and coding agents use your docs, and skip it if you’re a content or marketing site hoping for AI search traffic. Below is what the spec says, what each AI company says, what server logs show, and how to check your own.
Short on time?
Short on time? Here's my take.
- Docs or API site used by developers and coding agents
- Publish it, generated from your docs build
- Cheap to add. In Ahrefs' log data the top AI fetchers were GPTBot and Claude-Code, a coding agent, which fits this use.
- Blog, store or marketing site wanting AI search visibility
- Skip it
- Google says Search ignores the file, and SE Ranking found no link to AI citations. Spend the time on crawlable HTML and robots.txt rules.
- You already have one
- Keep it, and never let it return a server error
- A server error is the only thing that fails Chrome's Lighthouse audit. Keep it in sync with your docs.
- You want to know if it's used
- Check your own logs first
- Count /llms.txt requests by verified user agent before you believe anyone's adoption number.
Prices verified . Sponsors cannot buy a verdict or a position.
What’s actually in an llms.txt file? #
A short markdown document with one required line: an H1 with the name of the project or site. The proposal describes itself as “A proposal to standardise on using an /llms.txt file to provide information to help agents use a website”. llmstxt.org.
Jeremy Howard published it on September 3, 2024, and the page was last modified on August 10, 2026. It now says plainly, “This is v2 of the proposal”. llmstxt.org.
After the H1, everything is optional. The spec allows a blockquote summary, a few paragraphs of detail, then H2 “file list” sections where each line is a markdown link in the form [name](url): notes. An H2 called “Optional” holds links an agent can skip. llmstxt.org.
Here is a minimal example I wrote from the spec’s format. The site and URLs are made up.
# Example API
> Example API returns weather data as JSON. These links cover setup, auth and every endpoint.
Requests need an API key in the Authorization header.
## Docs
- [Quick start](https://example.com/docs/quickstart.md): install the SDK and make a first call
- [Authentication](https://example.com/docs/auth.md): keys, scopes and rotation
- [Endpoints](https://example.com/docs/endpoints.md): every route with parameters
## Optional
- [Changelog](https://example.com/changelog.md): release notes
What changed in v2? #
Four things matter if you’re writing one today. A file can now sit at a subpath such as /docs/llms.txt and covers the URLs under that path, and “where more than one file applies, agents should use the most specific one”. llmstxt.org.
Pages should offer a markdown copy at the same URL as page.html.md or, new in v2, page.md. Agents can discover those copies through rel="alternate" type="text/markdown" and rel="describedby", either in a <link> tag or an HTTP Link: header. llmstxt.org, v2 changes.
v2 also removed the llms_txt2ctx context-expansion tooling, and the “Optional” section lost its mechanical meaning. v2 changes.
Where does llms-full.txt fit? #
It doesn’t, as far as the spec goes. Neither the main proposal nor the v2 change log mentions llms-full.txt. llmstxt.org, v2 changes.
It’s a convention some docs platforms generate anyway. Mintlify, for example, creates /llms.txt, /llms-full.txt and .md page copies automatically for the docs it hosts. Mintlify.
Mintlify also says the format “was developed by Mintlify in collaboration with Anthropic and is now part of the official llms.txt proposal”. The v2 spec I read doesn’t mention it, so I treat it as a platform convention. Mintlify, llmstxt.org.
How is llms.txt different from robots.txt and a sitemap? #
robots.txt tells crawlers what they may fetch, a sitemap lists URLs for search engines to find, and llms.txt is a curated reading list. Only robots.txt carries crawl rules; llms.txt controls nothing.
| llms.txt | robots.txt | XML sitemap | |
|---|---|---|---|
| What it’s for | A curated, markdown reading list of key pages for agents | Crawl rules per user agent | A list of URLs for search engines to discover |
| Who says they read it | No AI crawler’s docs say so; Chrome’s Lighthouse audits it | The crawlers that document following it | Search engines |
| Does it allow or block anything? | No | It states rules; compliance is up to the crawler | No |
| Format | Markdown | Plain-text directives | XML |
The llms.txt column follows the spec and the vendor pages in the next section, verified 2026-11-16.
If you want to stop AI training, llms.txt is the wrong file. The opt-out lives in robots.txt, covered in my post on the AI training opt-out in robots.txt, and whether those rules carry legal weight is in is robots.txt legally binding.

Who publishes an llms.txt file? #
Mostly developer docs. OpenAI, Anthropic, Google’s Gemini API and the Model Context Protocol all publish an llms.txt for their docs, and I opened each of them.
| Publisher | File | What’s in it |
|---|---|---|
| OpenAI | developers.openai.com/llms.txt | H1 “OpenAI Developers”, a blockquote summary, about 41 links |
| Anthropic | platform.claude.com/llms.txt (docs.claude.com/llms.txt redirects here) | H1 “Anthropic Developer Documentation”, no blockquote, 850+ links, about 102,000 characters |
| Google (Gemini API) | aistudio.google.com/llms.txt (ai.google.dev/api/llms.txt redirects here) | H1 “Gemini API & AI Studio Documentation”, roughly 400 links |
| Model Context Protocol | modelcontextprotocol.io/llms.txt | H1 “Model Context Protocol”, no blockquote, about 360 links |
All four files opened and verified 2026-11-16. Link counts are approximate.
Notice the spread. OpenAI’s file is a short index; Anthropic’s runs to 850+ links, which is closer to a full table of contents than a reading list. The spec doesn’t set a size limit, so both are valid.
The spec site claims “thousands of sites publish an llms.txt file” and lists Mintlify, GitBook, Yoast SEO, AIOSEO and Wix as integrations. That’s its own count, and it names no AI company as a reader. llmstxt.org.
Publishing is not the same as reading. Every company in the table above publishes a file for its own docs, and none says its crawler reads yours.
Do AI companies say they read llms.txt? #
None of them commits to reading other sites’ llms.txt files. I went through each company’s crawler or search documentation and recorded what it says, word for word where it says anything.
| Company or product | What its own docs say about llms.txt | Source |
|---|---|---|
| Google Search | Ignores it: creating one “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.” | AI optimization guide |
| Google Search (AI features) | “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.” | AI features page |
| Chrome Lighthouse | Audits it; flags a page only “if a server error occurs when attempting to retrieve the llms.txt file” |
Lighthouse audit |
| OpenAI | Its bots page doesn’t say its crawlers read llms.txt. Its only mention: “For the complete documentation index, see llms.txt”, meaning its own | OpenAI bots |
| Anthropic | Its crawler article for ClaudeBot, Claude-User and Claude-SearchBot doesn’t mention llms.txt | Anthropic |
| Perplexity | Its bots page doesn’t say its bots read llms.txt; the header points only to its own docs index | Perplexity bots |
| Microsoft (Bing, Copilot) | No primary statement found | n/a |
All pages read and verified 2026-11-16.
The Microsoft row is a gap, not an answer. I found only secondary reports of a Bing representative advising against LLM-only pages, and no Bing page I could cite.
A missing statement isn’t proof that a company doesn’t read the file. It does mean nobody has promised to, so you can’t plan around it.
The bot names in that table, and how to tell a real one from a spoofed one, are in my list of AI crawler user agents.
Does anything actually fetch the file? #
Very little, going by the three log studies I read. The largest, from Ahrefs, found that 97% of sites with an llms.txt got zero requests for it in May 2026. Ahrefs study.
| Study | Who ran it | Sample | Window | Finding | Caveat |
|---|---|---|---|---|---|
| llms.txt study | Ahrefs (Louise Linehan) | 137,210 domains with traffic; 38,360 (28%) had a valid file | May 2026 | 97% of files got zero requests; the other ~3% (about 1,100 domains) got all ~22,000 requests | Ahrefs Web Analytics customers; sample skews technical |
| Crawler census | dejan.ai (Dan Petrovic) | One site | 20 days from July 27, 2026 | llms.txt fetches: Google 3, Anthropic 1 (ClaudeBot), OpenAI 0; robots.txt fetches: 1,150 | One site only; its table counts 243 Google robots.txt fetches, its text 242 from Googlebot |
| llms.txt and AI citations | SE Ranking (Yulia Deda) | About 300,000 domains; 10.13% had the file | Page dated Oct 15, 2026 | A model of AI citations got better when the llms.txt variable was removed | Doesn’t say where the citation data came from |
| Limy and Otterly data | Via an Optimizely summary | Limy: 500M+ AI bot visits; Otterly: one site, 62,100+ AI bot visits | 90 days each | Limy: “Only 408 requested llms.txt directly”; Otterly: 84 visits, about 0.1% | Second-hand: I didn’t open the primaries |
Studies read and verified 2026-11-16.
What “valid” means in the Ahrefs number #
Ahrefs’ study page defines it: the root /llms.txt returns a 200, the body is markdown rather than HTML, and soft-404 pages are screened out. It didn’t check whether files follow the spec, and Ahrefs says its technical-leaning sample makes 28% an upper bound. Ahrefs study.
That definition matters because Ahrefs’ own explainer repeats the 28% and 97% figures without defining “valid”. Ahrefs explainer.
Who made the requests that did happen? #
Mostly not AI. Ahrefs found 96% of requests came from bots, but only 19.5% of requests to fetched files came from AI bots, while SEO audit tools made 21.7%. Ahrefs study.
Among AI bots, GPTBot was the top fetcher at 4.51% of all requests, and Claude-Code, a coding agent, came second. Lighthouse made 22 requests, and Googlebot’s roughly 900 fetches looked like routine crawling. No AI bot requested /llms.txt on domains where it returned a 404. Ahrefs study.
Ahrefs calls every figure in its study “a ceiling on actual llms.txt consumption”: a request shows a bot downloaded the file, not that anything used it. That’s the right way to read every number in the table.
Does Google use llms.txt at all? #
Not for Search. Three different parts of Google touch the file: Search ignores it, Chrome’s Lighthouse audits it, and the Gemini API docs publish one.
Search ignores it. On June 15, 2026, Google’s Search Central update log “Added a note to the AI optimization guide clarifying Google Search’s usage of llms.txt files”. The note says the files “aren’t needed for Google Search” “(and won’t negatively or positively impact your visibility or rankings)”. Search Central updates.
The guide itself is blunter: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search”. AI optimization guide.
The same June note adds that “it’s fine if you want to maintain these files for other services or systems that use them”. That’s permission, not a recommendation. Search Central updates.
Lighthouse audits it. Chrome’s Lighthouse has an llms.txt audit under “agentic browsing”. It flags a page only when fetching the file causes a server error; a 404 makes the audit “Not Applicable (N/A)” because the file is optional for now. Lighthouse audit.
Google’s developer docs publish one. The Gemini API’s file at aistudio.google.com is in the publishers table above. A CMS change also put llms.txt on Google’s developer docs, and Search removed it from its own docs, as Search Engine Roundtable reported in January 2026. aistudio.google.com/llms.txt, Search Engine Roundtable.
Asked whether that was an endorsement, John Mueller answered, “to be direct, no.” Earlier he said of the files on Google sites, “It’s safe to assume that they’re there for other purposes.” Both quotes come via news coverage; I didn’t open his original posts. SER, endorsement, SER, other purposes.
Some ranking pages still lead with an older, undated Mueller line, “no Gen AI system currently uses llms.txt”. Similarweb, Dec 2025. It’s consistent with today’s position, but the June 2026 documentation is the primary source I’d quote.

How can you see if bots read yours? #
Count requests for /llms.txt in your access logs, grouped by user agent. This is the only adoption number that applies to your site.
Assuming your server writes the common combined log format, where the user agent is the sixth quoted field, a one-liner like this does it. I wrote it from the log format, not from a run on real logs.
grep '"GET /llms.txt' access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn
Three rules for reading the result:
- Verify the bot. Anyone can type “GPTBot” into a user agent. Check the IP with the method the operator documents, listed in my guide to verifying AI crawler user agents.
- A fetch is a ceiling. As Ahrefs puts it, a request proves a download, not use.
- Compare against robots.txt. In dejan.ai’s single-site census, robots.txt drew 1,150 fetches over 20 days while llms.txt drew four. dejan.ai.
Give it at least a month. One quiet week tells you little.
So should you add one? #
Add one if your docs are read by developers and coding agents; skip it on a content or marketing site. The cost-benefit lands differently for each.
| Your site | My pick | Why |
|---|---|---|
| Docs or API reference | Publish, generated from the docs build, with .md page copies |
The labs publish their own for developer docs, and the top AI fetchers Ahrefs saw were GPTBot and Claude-Code |
| Blog, publisher or marketing site | Skip it | Google Search ignores it; SE Ranking found no link to AI citations |
| Store or product catalogue | Skip it | Same as above, and nobody documents reading it |
| You already publish one | Keep it, fix any server errors, keep it in sync | A server error is the only Lighthouse failure; a stale file points agents at dead links |
Picks based on sources verified 2026-11-16.
On a docs site, generate the file in the same build that publishes your docs, so it never drifts. Mintlify does this for the docs it hosts, and the spec lists GitBook, Yoast SEO, AIOSEO and Wix as integrations too. Mintlify, llmstxt.org.
Keep the H1, add a one-line blockquote, and group links by task. If a whole section such as /docs/ is the useful part, v2 lets you place the file there instead of the root.
Serve .md copies of the linked pages too. If you also want to convert other people’s pages into markdown, that’s a different job, covered in website to markdown for LLMs, and for the retrieval side see web scraping for RAG.
On a content site, the June 2026 Google note settles the Search question, and SE Ranking found “There’s no correlation between AI citations and LLMs.txt.” SE Ranking. Mintlify, which sells generation, quotes the core criticism itself: “no major AI company officially supports the format”, while noting that Anthropic partnered with it on its own docs file. Mintlify.
What I would do #
If I managed a docs or API site, I’d generate llms.txt and .md copies in the docs build this week. Then I’d check the logs after 30 days for /llms.txt requests from verified agents, and keep the file only if it’s cheap to maintain, which it should be.
On a blog or marketing site, I wouldn’t bother. I’d spend the hour on crawlable HTML and clear robots.txt rules for the bots I care about.
I’d revisit that call the day any AI company documents reading third-party llms.txt files. Until then, the crawl APIs hub covers the tools on the other side of this file: the crawlers and converters that fetch your pages.
FAQ
Does Google use llms.txt?
Not for Search. Google's AI optimization guide says Google Search ignores llms.txt files, and a June 15, 2026 Search Central update added that note. Chrome's Lighthouse does audit the file, but it only flags a server error, and Google's own Gemini API docs publish one.
Does ChatGPT or Claude read my llms.txt?
Their crawler docs don't say so. OpenAI's and Anthropic's crawler pages never say their bots read other sites' llms.txt files. In Ahrefs' May 2026 log data, GPTBot and the Claude-Code agent were the top AI fetchers of the file, but only a small share of sites saw any request at all, and a fetch is not proof of use.
Is llms.txt an official standard?
No. It's a proposal by Jeremy Howard, first published September 3, 2024 and now at v2 (August 10, 2026). It isn't an IETF or W3C standard.
What's llms-full.txt?
A convention some docs platforms generate alongside llms.txt, usually one file with the full text of the docs. Mintlify, for example, creates it automatically and says it developed the format with Anthropic. The llms.txt v2 spec itself doesn't mention it.
Can llms.txt block AI training?
No. llms.txt is a reading list and controls nothing. To opt out of AI training, use robots.txt rules for the training crawlers and the newer training opt-out signals.
Will an llms.txt hurt my rankings?
Not in Google Search. Google says creating one will neither harm nor help your visibility or rankings, because Google Search ignores the file.
Sources
22 sources, accessed .
Official pricing, docs, changelogs and status pages 13
- llms.txt proposal, v2 (modified 2026-08-10)
- llms.txt: changes in v2
- Google Search Central: documentation updates (June 15, 2026 llms.txt note)
- Google Search Central: AI optimization guide
- Google Search Central: AI features and your website
- Chrome for Developers: Lighthouse llms.txt audit
- OpenAI: crawlers and user agents
- Anthropic: does Anthropic crawl data from the web?
- Perplexity: crawlers
- OpenAI Developers llms.txt
- Anthropic Developer Documentation llms.txt
- Gemini API and AI Studio llms.txt
- Model Context Protocol llms.txt


