Explainer
How to crawl an entire website for your LLM
Find every URL, keep the crawl in scope and price it first. Settings and costs for Firecrawl, Apify, Crawl4AI, Cloudflare and Spider, checked Nov 2026.

Getting a docs site, help centre or knowledge base into an LLM index takes four decisions: how you find every URL, how you keep the crawl inside the part you want, which tool does the whole site in one call, and what it costs at your page count. This guide answers each one with the settings and prices I found in the vendors’ docs on November 17, 2026.
The short version: for most docs sites I’d start with Firecrawl’s /crawl. It respects robots.txt by default, stops at 10,000 pages unless you raise the limit, and charges a flat credit per page. Firecrawl crawl docs.
Short on time?
Short on time? Here's my take.
- Most docs sites
- Firecrawl /crawl
- robots.txt on by default, a 10,000-page default cap, sitemap included, 1 credit a page. 100,000 pages fit the $99 Standard plan.
- Cheapest hosted, static pages
- Cloudflare Browser Run /crawl with render:false
- Not billed during the beta, honors crawl-delay, returns Markdown. Caveats: beta, 100,000-page max, a user agent you can't change.
- Self-hosted, no per-page bill
- Crawl4AI deep crawl, with check_robots_txt switched on
- Free and open source, but robots.txt checks are off by default and there's no page cap unless you set one.
- Keeping a big index fresh
- Apify Website Content Crawler
- Its modified-since recrawl uses lastmod dates and 304s and returns only what changed. Cloudflare's modifiedSince does a similar job.
Prices verified . Sponsors cannot buy a verdict or a position.
These picks come from reading docs and price pages. I didn’t crawl anything for this guide.
Where do you find every URL first? #
From the site’s own lists, before any link crawl. I’d work in this order: robots.txt, the sitemap index, llms.txt where it exists, then a map or seeding call, and finally a link crawl to fill gaps.
Start with robots.txt and the sitemap index #
A site can point crawlers to its sitemaps with a Sitemap: line in robots.txt. sitemaps.org. Read the rest of the file at the same time, because it also lists the paths you’re asked to skip.
One sitemap file holds at most 50,000 URLs and 50 MB (52,428,800 bytes) uncompressed, and a sitemap index can list up to 50,000 sitemaps. sitemaps.org. So a 100,000-page site needs at least two sitemap files plus an index, and /sitemap.xml may be the index rather than the pages.
Two rules in the protocol help you trust what you find. URLs in a sitemap must share its protocol and host and sit at or below its path, and lastmod is “the date the linked page was last modified, not when the sitemap is generated”. sitemaps.org.
That second rule is what makes cheap recrawls possible later, so keep the lastmod values with each URL.
Check for llms.txt on docs sites #
Some docs sites publish an llms.txt file: Markdown with a required H1, H2 sections of links, an “Optional” section, and often .md copies of the pages. It’s a proposal by Jeremy Howard, first published September 3, 2024 and last modified August 10, 2026, and it can sit at the root or under a subpath. llmstxt.org.
Where it exists, it’s the site owner’s own list of what an LLM should read. My llms.txt explainer covers the format; here it’s one more URL source.
Let a map or seeding call fill the gaps #
The hosted tools can do discovery for you, cheaply. Firecrawl’s /map costs 1 credit per call “regardless of the number of URLs returned”, reading the sitemap first and adding search results and earlier crawls. Firecrawl map docs.
Its docs also warn it “may not capture all website links”, so treat the map as a starting list, not the full site. Firecrawl map docs.
Crawl4AI calls this step URL seeding. It pulls candidates from the sitemap, Common Crawl (“cc”) or both, with a pattern filter, a max_urls cap, an optional live check and BM25 scoring against a query. Crawl4AI URL seeding.
The same page notes seeding may miss very new pages. That’s why I’d still let a link crawl run after it, inside the scope you set next.

How do you stop the crawl wandering? #
Set six things before you start: a page cap, a depth limit, include and exclude patterns, subdomains off, query strings ignored, and a plan for duplicates. The ranking tutorials I read skip several of these; the Hugging Face walkthrough sets an Apify include glob of https://help.obsidian.md/** and never mentions a page cap. Hugging Face tutorial.
| What to set | Firecrawl /crawl | Apify Website Content Crawler | Crawl4AI deep crawl | Cloudflare /crawl |
|---|---|---|---|---|
| Page cap | limit, default 10,000 |
maxCrawlPages, default 9,999,999 |
max_pages, no limit by default |
limit, default 10, max 100,000 |
| Depth | maxDiscoveryDepth |
maxCrawlDepth |
max_depth |
depth, default and max 100,000 |
| Patterns | includePaths / excludePaths: regex on the path, up to 1,000; regexOnFullURL for the whole URL |
includeUrlGlobs / excludeUrlGlobs: globs |
URLPatternFilter, DomainFilter, ContentTypeFilter |
includePatterns / excludePatterns: wildcards, and exclude wins |
| Leaving the site | crawlEntireDomain, allowSubdomains, allowExternalLinks all default to false |
No dedicated input; scope it with globs | include_external |
includeSubdomains, includeExternalLinks both default to false |
| Query strings | ignoreQueryParameters |
No dedicated input; use exclude globs | Not in the docs I read; use filters | Not in the docs I read; use patterns |
| robots.txt | Respected; ignoring it is Enterprise-only | respectRobotsTxtFile, on by default; no crawl-delay support |
Off unless check_robots_txt is True |
Respected, including crawl-delay |
Verified 2026-11-17: Firecrawl, Apify, Crawl4AI deep crawling, Crawl4AI parameters, Cloudflare.
The page cap is the setting I’d never leave at its default. Cloudflare’s 10 pages will stop you short, while Apify’s 9,999,999 and the no-limit defaults in Crawl4AI and Spider let a calendar or faceted search page run up the bill. I’d set it a little above the sitemap count, so a surprise shows up as a hit cap rather than a surprise invoice.
Watch the pattern syntax when you move rules between tools. Firecrawl takes regular expressions matched against the path, Apify takes URL globs and Cloudflare takes wildcards, so a rule written for one won’t mean the same thing in another.
What about duplicate pages? #
Let the site’s own canonical signals decide. Google treats permanent redirects and rel=canonical, in the HTML or an HTTP header, as strong canonical signals, and sitemap inclusion as a weak one. Google Search Central.
If I were building the pipeline, I’d key each document on its final URL after redirects, then collapse anything whose rel=canonical points elsewhere. ScrapingBee’s guide adds hash and MinHash checks for near-identical text, which makes a sensible second pass. ScrapingBee.
Which tools crawl a whole site in one call? #
Firecrawl, Apify’s Website Content Crawler, Crawl4AI, Cloudflare Browser Run and Spider all crawl a whole site from one start URL, and they differ more in defaults than in features. Here’s how they line up on the docs and price pages.
| Tool | Default page cap | Scope syntax | Sitemap use | robots.txt by default | Output | Recrawl option | Price unit |
|---|---|---|---|---|---|---|---|
| Firecrawl /crawl | 10,000 | Regex on path | Include (default), skip or only | Respected | Markdown, HTML, JSON | changeTracking format |
1 credit per page |
| Apify Website Content Crawler | 9,999,999 | URL globs | useSitemaps (off by default) |
Respected (respectRobotsTxtFile) |
Text, Markdown, HTML file, files | onlyCrawlPagesModifiedSince |
Compute units; Apify estimates $0.20-$5 per 1,000 pages |
| Crawl4AI deep crawl | None | Pattern and domain filters | Through URL seeding | Off | Markdown | Not covered in the docs I read | Free, self-hosted |
| Cloudflare Browser Run /crawl | 10 (max 100,000) | Wildcards | All, sitemaps or links | Respected, plus crawl-delay | HTML, Markdown, JSON | modifiedSince, maxAge |
Browser hours; render:false not billed in the beta |
| Spider | None (limit 0 crawls all pages) |
Regex whitelist / blacklist |
sitemap (off by default) or sitemap_only |
Respected (respect_robots) |
Raw by default; Markdown, text, XML and more | Not in the docs I read | $1 per GB plus $0.0001 per compute minute |
Verified 2026-11-17: Firecrawl, Firecrawl change tracking, Apify, Crawl4AI, Crawl4AI on GitHub, Cloudflare, Spider docs, Spider pricing.
Firecrawl /crawl #
It has the safest defaults of the five for a docs site: robots.txt respected, subdomains and external links off, the sitemap included and a 10,000-page cap. If your credits can’t cover the limit you set, it returns a 402 error. Firecrawl crawl docs.
Webhooks fire on crawl.started, each page and crawl.completed, signed with HMAC-SHA256. Results stay available through the API for 24 hours, so save them as they arrive. Firecrawl crawl docs.
My Crawl4AI vs Firecrawl comparison covers the head-to-head in more depth.
Apify Website Content Crawler #
It has the most dials and a risky default: maxCrawlPages is 9,999,999. It runs in an adaptive mode by default, and the bill depends heavily on whether pages go through raw HTTP or a headless browser. Apify.
Its strength is recrawling, covered further down. robots.txt is respected by default through respectRobotsTxtFile, but the crawler doesn’t support crawl-delay, so I’d keep it on and set a polite pace myself. Apify.
Crawl4AI deep crawl #
It’s free under Apache-2.0, with about 87.5k GitHub stars and v0.9.5 released October 14, 2026. GitHub.
It offers BFS, DFS and BestFirst strategies (the docs recommend BestFirst) and can stream results as they arrive. Crawl4AI deep crawling.
The catch is robots.txt. The deep-crawl docs say compliance is “disabled by default”, and check_robots_txt defaults to False, so switch it on. Crawl4AI deep crawling, Crawl4AI parameters.
There’s also a hosted Crawl4AI Cloud with “1,000 free credits every month” and “plans from $10”; I couldn’t find its per-credit price. GitHub. My open-source LLM crawlers roundup compares it with other self-hosted options.
Cloudflare Browser Run /crawl #
It reads all sources, sitemaps only or links only, renders pages in a browser by default, and returns HTML, Markdown or JSON. It respects robots.txt including crawl-delay, and otherwise waits 0.5 seconds between requests to the same domain. Cloudflare crawl docs.
Three limits matter. The user agent is CloudflareBrowserRenderingCrawler/1.0 and can’t be changed, it doesn’t bypass bot protection, and results are kept for 14 days. Cloudflare crawl docs.
Spider #
Spider respects robots.txt by default, but its limit defaults to 0, which means every page, and it skips the sitemap unless you turn sitemap on. Scope comes from regex whitelist and blacklist paths, and output is raw unless you ask for Markdown. Spider API docs.
It charges “$1 per GB” of bandwidth plus “$0.0001 per minute” of compute, quotes “~$0.0003” per /crawl request, and you can top up from $5. Its Unlimited plans start at $40 a month for 2 concurrent requests, and the tier the page shows by default is $350 a month for 25. Spider pricing.
What will the crawl cost at your page count? #
The formula is simple: pages × credits per page × price per credit, plus any discovery calls. For metered tools it’s GB transferred × price per GB, plus compute time.
| Tool and mode | 1,000 pages | 10,000 pages | 100,000 pages | Assumption |
|---|---|---|---|---|
| Firecrawl, Markdown | $0 (Free, 1,000 credits) | $99 (Standard) | $99 (Standard, 100,000 credits) | 1 credit per page; monthly price |
| Firecrawl, JSON | $19 (Hobby, 5,000 credits) | $99 (Standard) | $399 (Growth, 500,000 credits) | 5 credits per page (1 + 4 for JSON) |
| Apify, raw HTTP | $0.20 | $2 | $20 | Apify’s estimate of $0.2 per 1,000 pages |
| Apify, headless browser | $0.50-$5 | $5-$50 | $50-$500 | Apify’s estimate of $0.5-$5 per 1,000 pages |
| Cloudflare, render:false | $0 | $0 | $0 | Not billed during the beta |
| Cloudflare, render:true | Inside Workers Paid’s 10 included hours | Inside Workers Paid’s 10 included hours | About $6.60 plus the Workers Paid fee | My assumption: 3 s of browser time per page |
| Spider | $0.02-$0.30 | $0.20-$3 | $2-$30 | Range across Spider’s own figures; units unclear |
| Crawl4AI, self-hosted | $0 + your compute | $0 + your compute | $0 + your compute | Your own servers |
List prices verified 2026-11-17: Firecrawl, Apify crawler, Apify plans, Cloudflare crawl, Cloudflare pricing, Spider.
How I got the Firecrawl numbers #
Firecrawl’s plans are Free with 1,000 credits, Hobby at $19 for 5,000, Standard at $99 for 100,000, Growth at $399 for 500,000 and Scale at $749 for 1,000,000 (annual billing drops them to $16, $83, $333 and $599 a month). Firecrawl pricing. That works out to $3.80 per 1,000 pages on Hobby, $0.99 on Standard, about $0.80 on Growth and about $0.75 on Scale.
A crawl costs 1 credit per page, JSON adds 4 more, and PDFs cost 1 credit per PDF page. Firecrawl crawl docs. So 100,000 Markdown pages fit Standard exactly, and the same crawl as JSON needs 500,000 credits, which is Growth.
Mind the edges. If you spend 1 credit on /map first, a full 1,000-page or 100,000-page crawl no longer fits, so set the limit to 999 or 99,999 or expect a 402. My Firecrawl pricing breakdown covers extra-credit packs.
How I got the Apify, Cloudflare and Spider numbers #
Apify’s Website Content Crawler page estimates “$0.2 per 1,000 web pages” with raw HTTP and “$0.5 - $5 per 1,000 web pages” with a headless browser (Apify). The Free plan’s $5 of monthly usage covers about 25,000 raw-HTTP pages or 1,000 to 10,000 browser pages on those estimates. Apify pricing.
The two Apify pages disagree on compute-unit price: the crawler page mentions a baseline of $0.25, while the pricing page lists $0.20 on Free and Starter ($19 a month), $0.16 on Scale ($199) and $0.13 on Business ($999). I trust the pricing page for plan maths; my Apify pricing guide goes further.
For Cloudflare with rendering on, Workers Paid includes 10 browser hours a month, then $0.09 an hour, and Workers Free gets 10 minutes a day. Cloudflare pricing. At my assumed 3 seconds a page, 100,000 pages is 83.3 hours, so (83.3 − 10) × $0.09 ≈ $6.60, and Free’s 10 minutes covers about 200 pages a day.
That 3-second figure is an illustration, not a measurement; slower pages mean a bigger bill. My Cloudflare Browser Run pricing guide has more on browser hours.
Spider’s range comes from its own figures, which don’t agree. “$0.002 to $0.01” per 100 HTTP pages gives the low end, “~$0.0003” per crawl request gives the high end if a request means a page, and at an assumed 150 KB a page, 100,000 pages is 15 GB, or about $15 of bandwidth before compute. Spider pricing.
If credits in general confuse you, how scraping API credits work explains the unit.

How long should a polite crawl take? #
Hours, not minutes, for a large site on one host. RFC 9309 doesn’t define crawl-delay at all, but it does set rules a whole-site crawl will hit. RFC 9309.
Don’t use a cached robots.txt for more than 24 hours. If robots.txt returns a 4xx you may crawl, but a 5xx means “assume complete disallow”, so a struggling server is a reason to stop. RFC 9309.
Now the arithmetic. At Cloudflare’s 0.5-second default gap, 100,000 pages on one host take at least 50,000 seconds, or 13.9 hours; at one request a second it’s 27.8 hours.
Firecrawl’s delay setting forces concurrency to 1, so any delay of d seconds means at least 100,000 × d seconds for that crawl. Firecrawl crawl docs.
Your plan’s concurrency (2 on Free up to 100 on Scale) is your account’s ceiling, not the pace a site can take. Firecrawl pricing.
Crawl4AI’s defaults are a mean_delay of 0.1 seconds, a max_range of 0.3 and a semaphore_count of 5. Crawl4AI parameters. That’s quicker than Cloudflare’s 0.5-second gap, so I’d slow it down for any single host; my guide to web scraping rate limits has the policy I’d use.
Do you need JavaScript rendering for this site? #
Only if the pages you want don’t show their text without it, so check before you pay, because rendering is the biggest swing in the bill. On Apify’s own estimates, browser pages cost 2.5 to 25 times as much as raw HTTP pages ($0.5-$5 against $0.2 per 1,000). Apify.
Cloudflare’s /crawl renders by default, so set render: false for static docs. Cloudflare crawl docs. My JavaScript rendering check shows how to tell whether a site needs it, section by section.
What format should the output be? #
Markdown, for almost every RAG pipeline. Firecrawl, Cloudflare and Crawl4AI all return it, and Apify returns text, Markdown or an HTML file. Firecrawl, Cloudflare, Crawl4AI, Apify.
Ask for JSON only when you need structured fields, since on Firecrawl it adds 4 credits a page and turns a $99 crawl of 100,000 pages into a $399 one. How clean that Markdown is varies by tool; my website-to-Markdown comparison covers extraction quality, and web scraping for RAG covers chunking and the rest of the pipeline.
How do you keep it fresh without recrawling everything? #
Recrawl what changed, not the whole site. The sitemap’s lastmod is meant to show when each page last changed, which gives you the list. sitemaps.org.
Apify’s onlyCrawlPagesModifiedSince uses lastmod dates and HTTP 304 responses, and it returns only the delta. Apify. So keep your previous snapshot and merge the changes into it; the recrawl output alone isn’t the whole site.
Cloudflare’s /crawl offers modifiedSince and maxAge, which defaults to 86,400 seconds (24 hours). Cloudflare crawl docs.
Firecrawl’s changeTracking format works with /crawl and labels each page new, same, changed or removed against your last scrape, with no extra credits for the basic mode. Each page still costs its normal scrape credit, so it flags changes rather than cutting the bill (Firecrawl change tracking). I didn’t find a recrawl option in the Crawl4AI pages I read.
How often? No standard sets a number. I’d match the schedule to how often the docs really change, and let lastmod decide what gets fetched.
Where’s the legal and ethical line? #
robots.txt isn’t access control: RFC 9309 says its rules are not a form of access authorization, so a permissive file doesn’t settle whether you may reuse the content. RFC 9309. Read the site’s terms too, and keep a dated note of what you checked before the first run.
My ethical web scraping checklist and guide to whether web scraping is legal cover the rest.
So which should you pick? #
- A typical docs site under 100,000 pages: Firecrawl /crawl. Its defaults are the safest, and the price is predictable at 1 credit a page.
- Static pages, smallest bill: Cloudflare /crawl with
render: false, as long as you’re comfortable with a beta and a fixed user agent. Remember its default cap is 10 pages. - You’d rather run it yourself: Crawl4AI deep crawl, with
check_robots_txton,max_pagesset and the delay raised. - A large index you refresh often: Apify Website Content Crawler for its modified-since recrawls, with
maxCrawlPagesset andrespectRobotsTxtFileon.
What I would do for a 5,000-page docs site #
I’d read robots.txt, pull the sitemap index and note the URL count. Then I’d crawl with Firecrawl or Cloudflare, limited to /docs/ with a page cap a little above that count, Markdown only.
After that, a weekly modified-since recrawl keeps it current, and I’d pay for rendering only on the sections that need it. That’s a recommendation from the docs, not a result from my own checks; the crawl APIs hub covers the wider tool choice.
FAQ
How do I find all pages on a website?
Read robots.txt for Sitemap lines, open the sitemap index (each sitemap file holds up to 50,000 URLs), add llms.txt if the site publishes one, then run a map or URL-seeding call and let a link crawl catch the rest. Firecrawl's /map costs 1 credit a call but its docs say it may not capture every link, so the link crawl still matters.
Does crawling a whole site respect robots.txt automatically?
It depends on the tool. Firecrawl's /crawl, Cloudflare's /crawl and Spider respect it by default, and Cloudflare also honors crawl-delay. Apify's Website Content Crawler has respectRobotsTxtFile on by default, but it doesn't support crawl-delay. Crawl4AI leaves robots.txt checks off unless you set check_robots_txt to True.
How much does it cost to crawl 10,000 pages?
On list prices I checked on November 17, 2026: $99 on Firecrawl's Standard plan for Markdown, about $2 on Apify's raw-HTTP estimate or $5 to $50 with a headless browser, $0 on Cloudflare's /crawl with render:false while it's in beta, and $0 plus your own compute with self-hosted Crawl4AI.
How many pages can one crawl job handle?
Firecrawl's default limit is 10,000 pages. Cloudflare's /crawl defaults to just 10 and goes up to 100,000. Apify's maxCrawlPages defaults to 9,999,999, and Crawl4AI has no cap unless you set max_pages. Set your own cap a little above the sitemap count either way.
Should I use llms.txt instead of crawling?
Use it where the site publishes one, because it's the site owner's own list of what an LLM should read, often with Markdown copies of pages. It's still a proposal, so check for it, but plan on the sitemap and a crawl as well.
How often should I recrawl?
No standard sets a number. Recrawl on change instead: read sitemap lastmod dates, or use Apify's onlyCrawlPagesModifiedSince or Cloudflare's modifiedSince so you only fetch pages that changed, on a schedule that matches how often the docs are edited.
Sources
21 sources, accessed .
Official pricing, docs, changelogs and status pages 17
- sitemaps.org: Sitemaps XML protocol
- IETF RFC 9309: Robots Exclusion Protocol
- llmstxt.org: the llms.txt proposal
- Google Search Central: consolidate duplicate URLs
- Firecrawl docs: crawl
- Firecrawl docs: map
- Firecrawl docs: change tracking
- Firecrawl pricing
- Apify: Website Content Crawler
- Apify pricing
- Crawl4AI docs: deep crawling
- Crawl4AI docs: parameters
- Crawl4AI docs: URL seeding
- Cloudflare Browser Run: /crawl endpoint (updated Nov 3, 2026)
- Cloudflare Browser Run: pricing (updated Apr 21, 2026)
- Spider pricing
- Spider API docs


