Explainer
What is Common Crawl, and can you build on it?
Common Crawl is a free monthly archive of about 2 billion web pages. What's in it, what the terms allow, how AI labs use it, and how to block CCBot.

Common Crawl is a free archive of the web run by a non-profit: about 2 billion pages a month, collected since 2008 and stored on Amazon S3 for anyone to use. It’s excellent for bulk text and for finding URLs, and weak when you need fresh or complete copies of particular sites. Common Crawl FAQ, October 2026 crawl notes, collinfo.json.
My take, as of November 17, 2026: if you need web text at scale, build on it, ideally through a cleaned derivative like FineWeb. If you run a site, two lines of robots.txt keep CCBot out of future crawls, but not out of the archives already published.
Short on time?
Short on time? Here's my take.
- Pretraining or large-scale text analysis
- Build on it, starting from FineWeb
- Start from a filtered derivative rather than raw WARC files, and only go back to the raw files for a filter it lacks.
- Finding URLs to crawl
- Use the CDXJ or URL Index
- Or Crawl4AI's cc seeding. Then fetch the live pages yourself so the content is current.
- RAG over sites that must be current or complete
- Crawl it yourself
- Common Crawl is monthly, takes a sample of each site and doesn't run JavaScript.
- Old versions of a page or domain
- CDXJ lookup plus a range request
- The cheapest route: one index query, then a few KB from one WARC file.
- Site owner who wants out
- Block CCBot in robots.txt
- That covers future crawls. Legal removal requests go by email, and old archive files stay as they are.
Prices verified . Sponsors cannot buy a verdict or a position.
What’s actually in a crawl? #
The October 2026 crawl, CC-MAIN-2026-43, holds 2.16 billion pages from 41.4 million hosts and 34.3 million registered domains, 371.19 TiB uncompressed. CCBot crawled from October 5 to 18, and Common Crawl published it on October 21. October 2026 crawl notes.
Of those, 830.1 million URLs had never appeared in an earlier crawl. By my arithmetic that’s about 38% new, so each month is part recrawl and part fresh ground.
Crawls arrive about monthly. The blog announced 12 between November 2025 and October 2026, ranging from 1.97 billion to 2.3 billion pages; the listing shows no post dates, so I matched months from the titles. Common Crawl blog.
Every crawl has an ID like CC-MAIN-2026-43, which is what you plug into the index and the file paths. The index server lists 129 crawl indexes, from that October crawl back to CC-MAIN-2008-2009, stored in the older ARC format. collinfo.json.
The organisation came first: Wikipedia says Gil Elbaz founded it in 2007, so “since 2007” and “since 2008” are both fair. Older size figures go stale fast, though. Wikipedia cites 3.1 billion pages by April 2023 and Massive’s glossary cites 2.42 billion for August 2025, so always date the number.
Which files do you need: WARC, WAT or WET? #
Usually the WET files: most text work never needs the raw WARC files. Each crawl ships in three main formats, described on the get-started page, so pick by the job, not by habit.
| Format | What’s inside | Who needs it | October 2026 size |
|---|---|---|---|
| WARC | Raw HTTP responses, the requests and crawl metadata | Anyone who needs the original HTML or headers | 86.19 TiB compressed |
| WAT | JSON metadata, including headers and each page’s links | Link-graph and metadata work | 14.06 TiB compressed |
| WET | Extracted plain text | Language-model and text-analysis work | 5.85 TiB compressed |
Sizes come from the October 2026 crawl notes, which also list path files for robots.txt responses (a 0.15 TiB dataset), non-200 responses and both indexes.
What isn’t in a crawl? #
A crawl is a sample, not the web. The FAQ says CCBot archives “a randomly selected subset” of a site, and a July 2026 post on coverage admits it’s hard to know how complete any one site is, because slow-changing pages get revisited less. Common Crawl FAQ, Measuring coverage.
That post gives no per-site percentage, and calls its CrawlDb “fetched” count an estimate. A better Host Index (v2) was still in testing when I read it.
CCBot doesn’t run JavaScript or keep cookies, so pages built in the browser are captured as their unrendered HTML. Responses are also truncated: at 5 MiB since the March 2025 crawl (CC-MAIN-2025-13), and at 1 MiB before that. FAQ, October 2026 crawl notes.

How do you use Common Crawl without downloading it all? #
Look up where a page lives first, then fetch only those bytes. A whole crawl is far bigger than most people expect: the October WARC files alone would take about 8.8 days to pull at a sustained 1 Gbit/s, or about 21 hours at 10 Gbit/s, by my arithmetic.
Here are the ways in, smallest first:
- Look up a URL or domain in the CDXJ index. Each crawl has its own endpoint, such as
https://index.commoncrawl.org/CC-MAIN-2026-43-index, which takes a URL or a pattern likeexample.com/*in the PyWB CDX Server syntax. Index server, collinfo.json. - Fetch one record with a range request. Each index hit gives a WARC filename, an offset and a length, so an HTTP Range request for just those bytes from data.commoncrawl.org returns one record of “a few KB” instead of a roughly 1 GB file. Browser access post, CASRAI’s walkthrough.
- Filter in bulk with the URL Index (formerly the Columnar Index). It was renamed on June 3, 2026, sits beside the CDXJ index as Parquet files at
s3://commoncrawl/cc-index/table/cc-main/warc/, and works with Athena, Spark, Pandas, Polars, Arrow and DuckDB. URL Index post. - Download whole files. Use Common Crawl’s cc-downloader, the AWS CLI with
--no-sign-requestagainsts3://commoncrawl/, or plain HTTPS from data.commoncrawl.org, which needs no AWS account. Get started. - Start from a mirror or a derivative. Selected crawls are mirrored in Hugging Face Storage Buckets, and cleaned datasets such as FineWeb have done the filtering for you. Get started, FineWeb card. If you need structured records rather than crawl text, I compare buying a web dataset with scraping it yourself.
Since May 6, 2026, the first three also work straight from a browser. Common Crawl opened CORS on data.commoncrawl.org and index.commoncrawl.org, so a web page can run CDX lookups, byte-range fetches and DuckDB-WASM SQL on the URL Index. Browser access post.
cc-downloader is a Rust command-line tool and library with Python bindings, licensed Apache-2.0 or MIT. It runs 10 downloads at once by default and warns that going higher triggers 403 errors it can’t retry; it retries each file up to 1,000 times. cc-downloader on GitHub.
Does the index have rate limits? #
Yes, but no number is published. The FAQ asks for HTTPS only, slowing down on a 503, sleeping between calls, no parallel threads per IP and no proxies, and says a blocked IP should wait 24 hours. Common Crawl FAQ.
The CDXJ index is shared and easy to overload. The index page just says “Please do not overload the URL index server”, and the browser post calls the limits “strict”. Index server, Browser access post.
For anything bigger than a few lookups, the FAQ points you to the URL Index or cc-downloader instead. The same back-off habits apply to any crawler you write; my rate-limit guide covers them.
As for cost, the data is free and the compute is yours. If you read from S3 outside AWS’s us-east-1 region, AWS charges inter-region transfer fees. Get started.
What do the terms let you do? #
Less than “free” suggests. Common Crawl’s Terms of Use, last updated March 7, 2024, say this in plain words:
- You get a limited licence. The grant is a “limited license to access and use the Service”.
- The pages keep their owners’ terms. Crawled content “may be subject to separate terms” set by whoever owns it.
- No harvesting personal information. The terms forbid collecting personal data from the archive.
- It comes as-is, with a small cap. Liability is capped at “$100 USD”.
- You carry the AI risk. You indemnify Common Crawl, including against claims tied to AI training and generated content.
- Disputes go to arbitration. California law applies, with individual JAMS arbitration in Los Angeles and a class-action waiver.
- No attribution rule. I found no clause requiring you to credit Common Crawl.
The second point matters most. Common Crawl can give you access to its files, but it can’t give you rights to the articles, photos or code inside them, and the indemnity puts claims about AI training on you rather than on Common Crawl.
This is my reading of the terms, not legal advice. My guides to whether scraping for AI training is legal and whether robots.txt is legally binding cover the law around it.
How did AI labs use it? #
Heavily, and the papers say so. GPT-3’s Table 2.2 lists filtered Common Crawl at 410 billion tokens and a 60% weight in the training mix, seen 0.44 times during training, beside WebText2 at 22%, two book sets at 8% each and Wikipedia at 3%. GPT-3 paper.
OpenAI built that slice from 41 monthly shards covering 2016 to 2019, cutting 45 TB of compressed text down to 570 GB after filtering. GPT-3 paper on arXiv.
Was it 60% of GPT-3 or 80%? #
Both numbers come from the same table; they measure different things. Common Crawl’s 410 billion tokens are about 82% of the 499 billion in the whole dataset, by my arithmetic, but the training mix sampled it at 60%.
Massive’s glossary says “80%+” and cites a Mozilla report that returned an error when I tried to open it. I’d quote 60% for “how much of the training”, and 82% for “how much of the available data”.
What about LLaMA and later datasets? #
Meta’s LLaMA paper puts CommonCrawl at 67.0% of its training mix (3.3 TB), plus 15.0% from C4. Meta processed five dumps from 2017 to 2020 through CCNet, with line-level deduplication, fastText language identification, an n-gram quality filter and a classifier for pages that Wikipedia references. LLaMA paper.
C4 is itself built from Common Crawl, from the April 2019 crawl per CASRAI. So by my count about 82% of LLaMA’s mix traces back to the archive.
Those are the two models I could confirm from primary papers. I’m not listing newer models, because I didn’t open a primary page that confirms them.
FineWeb is the cleaned route most builders should start from. Its card describes “more than 18.5T tokens (originally 15T)”, released under “ODC-By v1.0” with use “subject to CommonCrawl’s Terms of Use”, so Common Crawl’s terms still follow the data. FineWeb card.
Its pipeline filters URLs, extracts text with Trafilatura, keeps English scored at 0.65 or higher by fastText, applies Gopher, C4 and custom quality filters, deduplicates each crawl with MinHash and anonymises emails and IP addresses. It runs its own PII removal form, and version 1.3.0 removed domains after a cease-and-desist letter from Torstar. FineWeb card.
One conflict to know about: the card says 96 dumps from summer 2013 to April 2024, while the dataset viewer lists 114 subsets running to CC-MAIN-2025-26. FineWeb page. Check which version you download.
Can you use it for RAG or search? #
For finding URLs, yes; for the content itself, usually not. RAG needs pages that are current and complete, and Common Crawl is a monthly sample that never ran the page’s JavaScript.
| Job | Fit | Why |
|---|---|---|
| Pretraining or bulk text analysis | Good | About 2 billion pages a month, plain text already extracted in WET files, and cleaned derivatives such as FineWeb |
| Finding URLs to crawl | Good | Index lookups by domain or pattern; Crawl4AI's cc seeding finds 50 to 500 URLs a second |
| Old versions of a page | Good | 129 crawl indexes back to 2008, and a range request fetches a single record |
| Link-graph analysis | Good | WAT files carry each page's links |
| Fresh answers (news, prices, stock) | Poor | Monthly crawls of about two weeks, published a few days later |
| A complete copy of specific sites | Poor | A random subset of each site, no JavaScript, and a 5 MiB cap per response |
Source: Common Crawl FAQ, October 2026 crawl notes, collinfo.json, get-started page and Crawl4AI's URL-seeding docs. All read 17 Nov 2026. Scroll sideways for more columns.
URL discovery is the sweet spot. Crawl4AI can seed a crawl from cc, sitemap or sitemap+cc, and its docs quote 50 to 500 URLs a second from Common Crawl while warning that this “May miss very recent pages”. Crawl4AI URL seeding.
So I’d use the archive to build the list, then fetch and render the live pages yourself. My RAG scraping guide covers the rest of that pipeline, and my open-source LLM crawler comparison compares Crawl4AI with the other self-hosted options.
If you need live search results rather than pages, that’s a different job. A web search API answers today’s query; Common Crawl answers “what did the web look like last month”.
How do you block CCBot? #
Two lines in robots.txt stop future crawls:
User-agent: CCBot
Disallow: /
CCBot checks robots.txt before it crawls and rechecks it periodically. It also honours nofollow and reads sitemaps you announce in robots.txt. Common Crawl FAQ, CCBot page.
If you only want it slower, add a Crawl-delay under User-agent: CCBot; a value of 2 means one request every two seconds. By default it waits “a few seconds” between requests to a site and backs off on 429 and 5xx errors. Common Crawl FAQ.
How do you know it’s really CCBot? #
The real one identifies as CCBot/2.0 (https://commoncrawl.org/faq/). Some explainers, such as Search Engine World’s, still print an older form with +http://, so trust the FAQ.
Some crawlers fake the CCBot user agent. Common Crawl crawls from dedicated IP ranges whose reverse DNS sits under crawl.commoncrawl.org, so do a reverse lookup and then a forward lookup, or match against the ranges in index.commoncrawl.org/ccbot.json. Common Crawl FAQ, CCBot page.
The CCBot page notes that reverse DNS isn’t set up on IPv6, so use the IP file there. CCBot page. My AI crawler user agents guide shows the DNS check step by step; the Common Crawl pages I read document DNS and an IP file, not Web Bot Auth signatures.
Under the hood, CCBot is built on Apache Nutch and runs on Hadoop. It speaks HTTP/1.1 and HTTP/2 (TLS only) over IPv4 and IPv6, follows up to 4 redirects (5 for robots.txt), and uses conditional GETs with gzip, Brotli or zstd compression. Common Crawl FAQ.
Should you block it? #
I’d treat CCBot like the other training crawlers: block it if you block them, allow it if you don’t. My guide to blocking AI crawlers includes it in the standard setup, because anyone can reuse Common Crawl’s dataset.
The newer robots.txt AI-training opt-out is a separate signal with its own adoption story. A plain Disallow for CCBot is the control Common Crawl documents.
What blocking CCBot does, and what it doesn't
- It stops future crawls. CCBot honours robots.txt and rechecks it. FAQ
- It doesn’t edit old archives. Common Crawl calls its WARC files immutable; it says it filters removed URLs out of later crawls and hides them from its public tools and indices. Common Crawl, November 4, 2025
- Legal removal goes by email. Requests go to info@commoncrawl.org, and the Opt-Out Ledger, a public list rather than a self-serve form, publishes “the full opt-out list for every legal request we have received”. Opt-Out Ledger post
- Derived datasets are unclear. No page I read says whether a block affects FineWeb or C4, which were built from older crawls; FineWeb runs its own removal form. FineWeb card
The ledger post, from September 17, 2025, explains the rename from “registry” was made “to reflect the immutable nature of it”. It encourages robots.txt and says nothing about removing pages from past archives. Opt-Out Ledger post.
What did The Atlantic report, and what did Common Crawl say? #
In November 2025, The Atlantic published an investigation by Alex Reisner into Common Crawl. Its site refused my fetcher, so I’m relying on PPC Land’s November 5 summary and Wikipedia for what it found.
Per PPC Land, The Atlantic reported that paywalled articles were captured because CCBot doesn’t run the code that checks subscriptions. It also reported that no content files appeared modified since 2016, and that Common Crawl’s search tool showed “no captures” for nytimes.com in crawls from 2013 to 2022, and wrongly for more than 1,000 other domains.
PPC Land adds that The New York Times cited more than 16 million records in its lawsuit against OpenAI. Wikipedia notes that OpenAI and Anthropic each gave Common Crawl $250,000 in 2023.
Common Crawl’s reply, dated November 4, 2025, says CCBot doesn’t log in or get past paywalls and honours robots.txt. It explains the immutable WARC files and the filtering of later crawls, and says some AI companies have donated but no donor controls what’s collected. Common Crawl’s reply.
The two accounts conflict on what removal means, and I can’t settle it from the pages I read, but my read is that both can be true at once. Hiding a URL from the index isn’t the same as deleting it from the files, so anyone who already downloaded those files still has the pages.

So should you build on it or crawl it yourself? #
Build on it for scale and history; crawl it yourself for freshness and completeness. Here’s my pick for each case, on the documents above:
- Pretraining or large-scale text analysis: build on it, starting from FineWeb. The filtering, deduplication and PII work is done, and more than 18.5 trillion tokens is far more than most teams would gather on their own. Touch the raw WARC files only for a filter it lacks.
- Finding URLs to crawl: use the CDXJ or URL Index. Or Crawl4AI’s
ccseeding. Then fetch the live pages, because the archive’s copy may be weeks old. - RAG over specific sites that must be current or complete: crawl it yourself, or use a crawl API. Common Crawl is monthly, sampled and unrendered, which is the wrong shape for an index users trust today. My build-or-buy guide helps with that choice.
- Old versions of a page or domain: a CDXJ lookup plus a range request. One query and a few KB per record, and the data itself is free.
- Site owner who wants out: block CCBot in robots.txt. Email Common Crawl for a legal removal, and expect old archive files to stay.
If I needed web text at scale, I’d start from FineWeb and only open raw WARC files for a filter it lacks. For a RAG app over known sites, I’d use Common Crawl only to discover URLs and crawl the live pages myself, so the content is current and rendered.
As a site owner, I’d remember that a block only reaches forward. The data hub collects my other guides on crawler rules and training data.
FAQ #
Is Common Crawl free? #
Yes, the data is free; you pay for your own compute, and AWS charges inter-region transfer fees if you pull from S3 outside us-east-1. Plain HTTPS downloads need no AWS account. Get started page.
How often is Common Crawl updated? #
About monthly: the blog announced 12 crawls between November 2025 and October 2026. The October crawl ran from October 5 to 18 and came out on October 21. Common Crawl blog, October 2026 crawl notes.
Is my site in Common Crawl? #
Look it up in the CDXJ index for a recent crawl, such as CC-MAIN-2026-43. Each crawl takes a random subset of a site, so missing or partial pages are normal. Common Crawl FAQ, index server.
Does blocking CCBot remove my pages from old crawls? #
No. Common Crawl says its WARC files are immutable, so it filters blocked or removed URLs out of later crawls and hides them from its public tools and indices instead. Common Crawl’s November 2025 reply.
Can I use Common Crawl data commercially? #
The terms grant a limited licence and say the crawled pages may carry their owners’ own terms, so Common Crawl can give you access, not rights to the content inside. This is not legal advice; read my AI training legal guide. Terms of Use.
Does Common Crawl run JavaScript? #
No. CCBot doesn’t run JavaScript or use cookies, so pages built in the browser are captured as their unrendered HTML. Common Crawl FAQ.
Which AI models were trained on it? #
GPT-3 (60% of its training mix) and LLaMA (67%, plus 15% from C4) say so in their papers. FineWeb, a cleaned derivative, is built from Common Crawl dumps; I didn’t verify newer models. GPT-3 paper, LLaMA paper, FineWeb card.
Sources
25 sources, accessed .
Official pricing, docs, changelogs and status pages 16
- Common Crawl: FAQ (CCBot, opt-out, index rules)
- Common Crawl: CCBot
- Common Crawl: Terms of Use (updated March 7, 2024)
- Common Crawl: Get started (S3, HTTPS, WARC/WAT/WET)
- Common Crawl blog: October 2026 crawl archive (CC-MAIN-2026-43)
- Common Crawl blog index (monthly crawl announcements)
- Common Crawl index server: collinfo.json (129 crawl indexes)
- Common Crawl index server (CDXJ API)
- Common Crawl blog: The Columnar Index is now the URL Index (June 3, 2026)
- Common Crawl blog: build from the browser (CORS, May 6, 2026)
- Common Crawl blog: Opt-Out Registry, now the Opt-Out Ledger (September 17, 2025)
- Common Crawl blog: Setting the record straight (November 4, 2025)
- Common Crawl blog: Measuring crawled coverage of a website (July 20, 2026)
- Hugging Face: FineWeb dataset card (README)
- Hugging Face: FineWeb dataset page and viewer
- Crawl4AI docs: URL seeding


