Skip to content
CrawlfolioSourced research on web data for AI
Menu

Getting a docs site, help centre or knowledge base into an LLM index takes four decisions: how you find every URL, how you keep the crawl inside the part you want, which tool does the whole site in one call, and what it costs at your page count. This guide answers each one with the settings and prices I found in the vendors’ docs on November 17, 2026.

The short version: for most docs sites I’d start with Firecrawl’s /crawl. It respects robots.txt by default, stops at 10,000 pages unless you raise the limit, and charges a flat credit per page. Firecrawl crawl docs.

Short on time?

Short on time? Here's my take.

Most docs sites
Firecrawl /crawl
robots.txt on by default, a 10,000-page default cap, sitemap included, 1 credit a page. 100,000 pages fit the $99 Standard plan.
Cheapest hosted, static pages
Cloudflare Browser Run /crawl with render:false
Not billed during the beta, honors crawl-delay, returns Markdown. Caveats: beta, 100,000-page max, a user agent you can't change.
Self-hosted, no per-page bill
Crawl4AI deep crawl, with check_robots_txt switched on
Free and open source, but robots.txt checks are off by default and there's no page cap unless you set one.
Keeping a big index fresh
Apify Website Content Crawler
Its modified-since recrawl uses lastmod dates and 304s and returns only what changed. Cloudflare's modifiedSince does a similar job.

Prices verified . Sponsors cannot buy a verdict or a position.

These picks come from reading docs and price pages. I didn’t crawl anything for this guide.

Where do you find every URL first? #

From the site’s own lists, before any link crawl. I’d work in this order: robots.txt, the sitemap index, llms.txt where it exists, then a map or seeding call, and finally a link crawl to fill gaps.

Start with robots.txt and the sitemap index #

A site can point crawlers to its sitemaps with a Sitemap: line in robots.txt. sitemaps.org. Read the rest of the file at the same time, because it also lists the paths you’re asked to skip.

One sitemap file holds at most 50,000 URLs and 50 MB (52,428,800 bytes) uncompressed, and a sitemap index can list up to 50,000 sitemaps. sitemaps.org. So a 100,000-page site needs at least two sitemap files plus an index, and /sitemap.xml may be the index rather than the pages.

Two rules in the protocol help you trust what you find. URLs in a sitemap must share its protocol and host and sit at or below its path, and lastmod is “the date the linked page was last modified, not when the sitemap is generated”. sitemaps.org.

That second rule is what makes cheap recrawls possible later, so keep the lastmod values with each URL.

Check for llms.txt on docs sites #

Some docs sites publish an llms.txt file: Markdown with a required H1, H2 sections of links, an “Optional” section, and often .md copies of the pages. It’s a proposal by Jeremy Howard, first published September 3, 2024 and last modified August 10, 2026, and it can sit at the root or under a subpath. llmstxt.org.

Where it exists, it’s the site owner’s own list of what an LLM should read. My llms.txt explainer covers the format; here it’s one more URL source.

Let a map or seeding call fill the gaps #

The hosted tools can do discovery for you, cheaply. Firecrawl’s /map costs 1 credit per call “regardless of the number of URLs returned”, reading the sitemap first and adding search results and earlier crawls. Firecrawl map docs.

Its docs also warn it “may not capture all website links”, so treat the map as a starting list, not the full site. Firecrawl map docs.

Crawl4AI calls this step URL seeding. It pulls candidates from the sitemap, Common Crawl (“cc”) or both, with a pattern filter, a max_urls cap, an optional live check and BM25 scoring against a query. Crawl4AI URL seeding.

The same page notes seeding may miss very new pages. That’s why I’d still let a link crawl run after it, inside the scope you set next.

The bare upper branches of a leafless tree spreading into a pale, empty sky
Every branch ends in twigs a sitemap may never list: map the site first, then let a scoped link crawl reach the newer pages the map missed. Photo: Jei Lee / Unsplash · Unsplash License

How do you stop the crawl wandering? #

Set six things before you start: a page cap, a depth limit, include and exclude patterns, subdomains off, query strings ignored, and a plan for duplicates. The ranking tutorials I read skip several of these; the Hugging Face walkthrough sets an Apify include glob of https://help.obsidian.md/** and never mentions a page cap. Hugging Face tutorial.

What to set Firecrawl /crawl Apify Website Content Crawler Crawl4AI deep crawl Cloudflare /crawl
Page cap limit, default 10,000 maxCrawlPages, default 9,999,999 max_pages, no limit by default limit, default 10, max 100,000
Depth maxDiscoveryDepth maxCrawlDepth max_depth depth, default and max 100,000
Patterns includePaths / excludePaths: regex on the path, up to 1,000; regexOnFullURL for the whole URL includeUrlGlobs / excludeUrlGlobs: globs URLPatternFilter, DomainFilter, ContentTypeFilter includePatterns / excludePatterns: wildcards, and exclude wins
Leaving the site crawlEntireDomain, allowSubdomains, allowExternalLinks all default to false No dedicated input; scope it with globs include_external includeSubdomains, includeExternalLinks both default to false
Query strings ignoreQueryParameters No dedicated input; use exclude globs Not in the docs I read; use filters Not in the docs I read; use patterns
robots.txt Respected; ignoring it is Enterprise-only respectRobotsTxtFile, on by default; no crawl-delay support Off unless check_robots_txt is True Respected, including crawl-delay

Verified 2026-11-17: Firecrawl, Apify, Crawl4AI deep crawling, Crawl4AI parameters, Cloudflare.

The page cap is the setting I’d never leave at its default. Cloudflare’s 10 pages will stop you short, while Apify’s 9,999,999 and the no-limit defaults in Crawl4AI and Spider let a calendar or faceted search page run up the bill. I’d set it a little above the sitemap count, so a surprise shows up as a hit cap rather than a surprise invoice.

Watch the pattern syntax when you move rules between tools. Firecrawl takes regular expressions matched against the path, Apify takes URL globs and Cloudflare takes wildcards, so a rule written for one won’t mean the same thing in another.

What about duplicate pages? #

Let the site’s own canonical signals decide. Google treats permanent redirects and rel=canonical, in the HTML or an HTTP header, as strong canonical signals, and sitemap inclusion as a weak one. Google Search Central.

If I were building the pipeline, I’d key each document on its final URL after redirects, then collapse anything whose rel=canonical points elsewhere. ScrapingBee’s guide adds hash and MinHash checks for near-identical text, which makes a sensible second pass. ScrapingBee.

Which tools crawl a whole site in one call? #

Firecrawl, Apify’s Website Content Crawler, Crawl4AI, Cloudflare Browser Run and Spider all crawl a whole site from one start URL, and they differ more in defaults than in features. Here’s how they line up on the docs and price pages.

Tool Default page cap Scope syntax Sitemap use robots.txt by default Output Recrawl option Price unit
Firecrawl /crawl 10,000 Regex on path Include (default), skip or only Respected Markdown, HTML, JSON changeTracking format 1 credit per page
Apify Website Content Crawler 9,999,999 URL globs useSitemaps (off by default) Respected (respectRobotsTxtFile) Text, Markdown, HTML file, files onlyCrawlPagesModifiedSince Compute units; Apify estimates $0.20-$5 per 1,000 pages
Crawl4AI deep crawl None Pattern and domain filters Through URL seeding Off Markdown Not covered in the docs I read Free, self-hosted
Cloudflare Browser Run /crawl 10 (max 100,000) Wildcards All, sitemaps or links Respected, plus crawl-delay HTML, Markdown, JSON modifiedSince, maxAge Browser hours; render:false not billed in the beta
Spider None (limit 0 crawls all pages) Regex whitelist / blacklist sitemap (off by default) or sitemap_only Respected (respect_robots) Raw by default; Markdown, text, XML and more Not in the docs I read $1 per GB plus $0.0001 per compute minute

Verified 2026-11-17: Firecrawl, Firecrawl change tracking, Apify, Crawl4AI, Crawl4AI on GitHub, Cloudflare, Spider docs, Spider pricing.

Firecrawl /crawl #

It has the safest defaults of the five for a docs site: robots.txt respected, subdomains and external links off, the sitemap included and a 10,000-page cap. If your credits can’t cover the limit you set, it returns a 402 error. Firecrawl crawl docs.

Webhooks fire on crawl.started, each page and crawl.completed, signed with HMAC-SHA256. Results stay available through the API for 24 hours, so save them as they arrive. Firecrawl crawl docs.

My Crawl4AI vs Firecrawl comparison covers the head-to-head in more depth.

Apify Website Content Crawler #

It has the most dials and a risky default: maxCrawlPages is 9,999,999. It runs in an adaptive mode by default, and the bill depends heavily on whether pages go through raw HTTP or a headless browser. Apify.

Its strength is recrawling, covered further down. robots.txt is respected by default through respectRobotsTxtFile, but the crawler doesn’t support crawl-delay, so I’d keep it on and set a polite pace myself. Apify.

Crawl4AI deep crawl #

It’s free under Apache-2.0, with about 87.5k GitHub stars and v0.9.5 released October 14, 2026. GitHub.

It offers BFS, DFS and BestFirst strategies (the docs recommend BestFirst) and can stream results as they arrive. Crawl4AI deep crawling.

The catch is robots.txt. The deep-crawl docs say compliance is “disabled by default”, and check_robots_txt defaults to False, so switch it on. Crawl4AI deep crawling, Crawl4AI parameters.

There’s also a hosted Crawl4AI Cloud with “1,000 free credits every month” and “plans from $10”; I couldn’t find its per-credit price. GitHub. My open-source LLM crawlers roundup compares it with other self-hosted options.

Cloudflare Browser Run /crawl #

It reads all sources, sitemaps only or links only, renders pages in a browser by default, and returns HTML, Markdown or JSON. It respects robots.txt including crawl-delay, and otherwise waits 0.5 seconds between requests to the same domain. Cloudflare crawl docs.

Three limits matter. The user agent is CloudflareBrowserRenderingCrawler/1.0 and can’t be changed, it doesn’t bypass bot protection, and results are kept for 14 days. Cloudflare crawl docs.

Spider #

Spider respects robots.txt by default, but its limit defaults to 0, which means every page, and it skips the sitemap unless you turn sitemap on. Scope comes from regex whitelist and blacklist paths, and output is raw unless you ask for Markdown. Spider API docs.

It charges “$1 per GB” of bandwidth plus “$0.0001 per minute” of compute, quotes “~$0.0003” per /crawl request, and you can top up from $5. Its Unlimited plans start at $40 a month for 2 concurrent requests, and the tier the page shows by default is $350 a month for 25. Spider pricing.

What will the crawl cost at your page count? #

The formula is simple: pages × credits per page × price per credit, plus any discovery calls. For metered tools it’s GB transferred × price per GB, plus compute time.

Tool and mode 1,000 pages 10,000 pages 100,000 pages Assumption
Firecrawl, Markdown $0 (Free, 1,000 credits) $99 (Standard) $99 (Standard, 100,000 credits) 1 credit per page; monthly price
Firecrawl, JSON $19 (Hobby, 5,000 credits) $99 (Standard) $399 (Growth, 500,000 credits) 5 credits per page (1 + 4 for JSON)
Apify, raw HTTP $0.20 $2 $20 Apify’s estimate of $0.2 per 1,000 pages
Apify, headless browser $0.50-$5 $5-$50 $50-$500 Apify’s estimate of $0.5-$5 per 1,000 pages
Cloudflare, render:false $0 $0 $0 Not billed during the beta
Cloudflare, render:true Inside Workers Paid’s 10 included hours Inside Workers Paid’s 10 included hours About $6.60 plus the Workers Paid fee My assumption: 3 s of browser time per page
Spider $0.02-$0.30 $0.20-$3 $2-$30 Range across Spider’s own figures; units unclear
Crawl4AI, self-hosted $0 + your compute $0 + your compute $0 + your compute Your own servers

List prices verified 2026-11-17: Firecrawl, Apify crawler, Apify plans, Cloudflare crawl, Cloudflare pricing, Spider.

How I got the Firecrawl numbers #

Firecrawl’s plans are Free with 1,000 credits, Hobby at $19 for 5,000, Standard at $99 for 100,000, Growth at $399 for 500,000 and Scale at $749 for 1,000,000 (annual billing drops them to $16, $83, $333 and $599 a month). Firecrawl pricing. That works out to $3.80 per 1,000 pages on Hobby, $0.99 on Standard, about $0.80 on Growth and about $0.75 on Scale.

A crawl costs 1 credit per page, JSON adds 4 more, and PDFs cost 1 credit per PDF page. Firecrawl crawl docs. So 100,000 Markdown pages fit Standard exactly, and the same crawl as JSON needs 500,000 credits, which is Growth.

Mind the edges. If you spend 1 credit on /map first, a full 1,000-page or 100,000-page crawl no longer fits, so set the limit to 999 or 99,999 or expect a 402. My Firecrawl pricing breakdown covers extra-credit packs.

How I got the Apify, Cloudflare and Spider numbers #

Apify’s Website Content Crawler page estimates “$0.2 per 1,000 web pages” with raw HTTP and “$0.5 - $5 per 1,000 web pages” with a headless browser (Apify). The Free plan’s $5 of monthly usage covers about 25,000 raw-HTTP pages or 1,000 to 10,000 browser pages on those estimates. Apify pricing.

The two Apify pages disagree on compute-unit price: the crawler page mentions a baseline of $0.25, while the pricing page lists $0.20 on Free and Starter ($19 a month), $0.16 on Scale ($199) and $0.13 on Business ($999). I trust the pricing page for plan maths; my Apify pricing guide goes further.

For Cloudflare with rendering on, Workers Paid includes 10 browser hours a month, then $0.09 an hour, and Workers Free gets 10 minutes a day. Cloudflare pricing. At my assumed 3 seconds a page, 100,000 pages is 83.3 hours, so (83.3 − 10) × $0.09 ≈ $6.60, and Free’s 10 minutes covers about 200 pages a day.

That 3-second figure is an illustration, not a measurement; slower pages mean a bigger bill. My Cloudflare Browser Run pricing guide has more on browser hours.

Spider’s range comes from its own figures, which don’t agree. “$0.002 to $0.01” per 100 HTTP pages gives the low end, “~$0.0003” per crawl request gives the high end if a request means a page, and at an assumed 150 KB a page, 100,000 pages is 15 GB, or about $15 of bandwidth before compute. Spider pricing.

If credits in general confuse you, how scraping API credits work explains the unit.

An unfinished concrete building frame, floor slabs stacked one above another with brick infill between bare columns
Floor after floor of the same frame: a whole-site bill is one page price repeated 100,000 times, so a small per-page difference in credits or render time sets the total. Photo: Egor / Unsplash · Unsplash License

How long should a polite crawl take? #

Hours, not minutes, for a large site on one host. RFC 9309 doesn’t define crawl-delay at all, but it does set rules a whole-site crawl will hit. RFC 9309.

Don’t use a cached robots.txt for more than 24 hours. If robots.txt returns a 4xx you may crawl, but a 5xx means “assume complete disallow”, so a struggling server is a reason to stop. RFC 9309.

Now the arithmetic. At Cloudflare’s 0.5-second default gap, 100,000 pages on one host take at least 50,000 seconds, or 13.9 hours; at one request a second it’s 27.8 hours.

Firecrawl’s delay setting forces concurrency to 1, so any delay of d seconds means at least 100,000 × d seconds for that crawl. Firecrawl crawl docs.

Your plan’s concurrency (2 on Free up to 100 on Scale) is your account’s ceiling, not the pace a site can take. Firecrawl pricing.

Crawl4AI’s defaults are a mean_delay of 0.1 seconds, a max_range of 0.3 and a semaphore_count of 5. Crawl4AI parameters. That’s quicker than Cloudflare’s 0.5-second gap, so I’d slow it down for any single host; my guide to web scraping rate limits has the policy I’d use.

Do you need JavaScript rendering for this site? #

Only if the pages you want don’t show their text without it, so check before you pay, because rendering is the biggest swing in the bill. On Apify’s own estimates, browser pages cost 2.5 to 25 times as much as raw HTTP pages ($0.5-$5 against $0.2 per 1,000). Apify.

Cloudflare’s /crawl renders by default, so set render: false for static docs. Cloudflare crawl docs. My JavaScript rendering check shows how to tell whether a site needs it, section by section.

What format should the output be? #

Markdown, for almost every RAG pipeline. Firecrawl, Cloudflare and Crawl4AI all return it, and Apify returns text, Markdown or an HTML file. Firecrawl, Cloudflare, Crawl4AI, Apify.

Ask for JSON only when you need structured fields, since on Firecrawl it adds 4 credits a page and turns a $99 crawl of 100,000 pages into a $399 one. How clean that Markdown is varies by tool; my website-to-Markdown comparison covers extraction quality, and web scraping for RAG covers chunking and the rest of the pipeline.

How do you keep it fresh without recrawling everything? #

Recrawl what changed, not the whole site. The sitemap’s lastmod is meant to show when each page last changed, which gives you the list. sitemaps.org.

Apify’s onlyCrawlPagesModifiedSince uses lastmod dates and HTTP 304 responses, and it returns only the delta. Apify. So keep your previous snapshot and merge the changes into it; the recrawl output alone isn’t the whole site.

Cloudflare’s /crawl offers modifiedSince and maxAge, which defaults to 86,400 seconds (24 hours). Cloudflare crawl docs.

Firecrawl’s changeTracking format works with /crawl and labels each page new, same, changed or removed against your last scrape, with no extra credits for the basic mode. Each page still costs its normal scrape credit, so it flags changes rather than cutting the bill (Firecrawl change tracking). I didn’t find a recrawl option in the Crawl4AI pages I read.

How often? No standard sets a number. I’d match the schedule to how often the docs really change, and let lastmod decide what gets fetched.

robots.txt isn’t access control: RFC 9309 says its rules are not a form of access authorization, so a permissive file doesn’t settle whether you may reuse the content. RFC 9309. Read the site’s terms too, and keep a dated note of what you checked before the first run.

My ethical web scraping checklist and guide to whether web scraping is legal cover the rest.

So which should you pick? #

  • A typical docs site under 100,000 pages: Firecrawl /crawl. Its defaults are the safest, and the price is predictable at 1 credit a page.
  • Static pages, smallest bill: Cloudflare /crawl with render: false, as long as you’re comfortable with a beta and a fixed user agent. Remember its default cap is 10 pages.
  • You’d rather run it yourself: Crawl4AI deep crawl, with check_robots_txt on, max_pages set and the delay raised.
  • A large index you refresh often: Apify Website Content Crawler for its modified-since recrawls, with maxCrawlPages set and respectRobotsTxtFile on.

What I would do for a 5,000-page docs site #

I’d read robots.txt, pull the sitemap index and note the URL count. Then I’d crawl with Firecrawl or Cloudflare, limited to /docs/ with a page cap a little above that count, Markdown only.

After that, a weekly modified-since recrawl keeps it current, and I’d pay for rendering only on the sections that need it. That’s a recommendation from the docs, not a result from my own checks; the crawl APIs hub covers the wider tool choice.

FAQ

How do I find all pages on a website?

Read robots.txt for Sitemap lines, open the sitemap index (each sitemap file holds up to 50,000 URLs), add llms.txt if the site publishes one, then run a map or URL-seeding call and let a link crawl catch the rest. Firecrawl's /map costs 1 credit a call but its docs say it may not capture every link, so the link crawl still matters.

Does crawling a whole site respect robots.txt automatically?

It depends on the tool. Firecrawl's /crawl, Cloudflare's /crawl and Spider respect it by default, and Cloudflare also honors crawl-delay. Apify's Website Content Crawler has respectRobotsTxtFile on by default, but it doesn't support crawl-delay. Crawl4AI leaves robots.txt checks off unless you set check_robots_txt to True.

How much does it cost to crawl 10,000 pages?

On list prices I checked on November 17, 2026: $99 on Firecrawl's Standard plan for Markdown, about $2 on Apify's raw-HTTP estimate or $5 to $50 with a headless browser, $0 on Cloudflare's /crawl with render:false while it's in beta, and $0 plus your own compute with self-hosted Crawl4AI.

How many pages can one crawl job handle?

Firecrawl's default limit is 10,000 pages. Cloudflare's /crawl defaults to just 10 and goes up to 100,000. Apify's maxCrawlPages defaults to 9,999,999, and Crawl4AI has no cap unless you set max_pages. Set your own cap a little above the sitemap count either way.

Should I use llms.txt instead of crawling?

Use it where the site publishes one, because it's the site owner's own list of what an LLM should read, often with Markdown copies of pages. It's still a proposal, so check for it, but plan on the sitemap and a crawl as well.

How often should I recrawl?

No standard sets a number. Recrawl on change instead: read sitemap lastmod dates, or use Apify's onlyCrawlPagesModifiedSince or Cloudflare's modifiedSince so you only fetch pages that changed, on a schedule that matches how often the docs are edited.

Sources

21 sources, accessed .

Official pricing, docs, changelogs and status pages 17

  1. sitemaps.org: Sitemaps XML protocol sitemaps.org · accessed
  2. IETF RFC 9309: Robots Exclusion Protocol rfc-editor.org · accessed
  3. llmstxt.org: the llms.txt proposal llmstxt.org · accessed
  4. Google Search Central: consolidate duplicate URLs developers.google.com · accessed
  5. Firecrawl docs: crawl docs.firecrawl.dev · accessed
  6. Firecrawl docs: map docs.firecrawl.dev · accessed
  7. Firecrawl docs: change tracking docs.firecrawl.dev · accessed
  8. Firecrawl pricing firecrawl.dev · accessed
  9. Apify: Website Content Crawler apify.com · accessed
  10. Apify pricing apify.com · accessed
  11. Crawl4AI docs: deep crawling docs.crawl4ai.com · accessed
  12. Crawl4AI docs: parameters docs.crawl4ai.com · accessed
  13. Crawl4AI docs: URL seeding docs.crawl4ai.com · accessed
  14. Cloudflare Browser Run: /crawl endpoint (updated Nov 3, 2026) developers.cloudflare.com · accessed
  15. Cloudflare Browser Run: pricing (updated Apr 21, 2026) developers.cloudflare.com · accessed
  16. Spider pricing spider.cloud · accessed
  17. Spider API docs spider.cloud · accessed

GitHub code, issues and releases 1

  1. Crawl4AI on GitHub github.com · accessed

Community discussions 1

  1. Hugging Face community blog: chat with any full website (Oct 22, 2024) huggingface.co · accessed

Other blogs 2

  1. ScrapingBee blog: scrape all text from a website for LLM training (updated Sep 2, 2026) scrapingbee.com · accessed
  2. Indexical blog: crawl developer documentation (Aug 5, 2024) indexical-blog.pages.dev · accessed

What readers say

Used one of the tools in this guide? Tell other readers how it went. I read every review before it goes up, and I never turn one down for being negative. How reviews work.

No reader reviews yet. Yours could be the first.

Write a review
Your rating

What did you use it for, what worked, and what didn't? 30 to 1500 characters. Please don't include links.

Your connection to the company

Your review appears with the name you enter, after I approve it. I don't ask for your email. Privacy.