Research area
Best Web Scrapers for LLMs and RAG: How to Choose
Choose the fetch layer by output, rendering and operating model, then price a defined workload. Public sources checked October 31, 2026.
The best web scraper for an LLM or RAG pipeline is the one that returns the content you need within a defined budget. I choose output first, then rendering and crawl scope, then the operating model, cost and robots behaviour. I rechecked the primary sources retained below on October 31, 2026; no scraping or crawling jobs were run for this page.
Short on time?
My pick by workflow
- Managed scrape and crawl
- Firecrawl
- One managed API with markdown and optional structured extraction. Standard is $99/month for 100K credits; plain pages use one credit before extras.
- Run the software yourself
- Crawl4AI
- Control the source, browser settings and infrastructure. Apache-2.0 with an added attribution requirement; servers and providers still cost money.
- Known URLs, one at a time
- Jina Reader
- A URL prefix returns readable content. New keys get 10M non-commercial tokens; paid packs are $50 for 1B tokens or $500 for 11B, billed on output tokens.
- An Actor-based workflow
- Apify
- Choose the specific Actor and its billing model. Website Content Crawler is pay per usage; other Actors need separate pricing checks.
- Bandwidth-based billing
- Spider
- PAYG bills bandwidth and CPU, with free failed requests. Page weight and settings prevent a universal per-page price.
Prices verified . Documented fit and configured list prices, not a measured quality or speed ranking.
Web scrapers for LLMs and RAG compared #
The current short list below covers seven configurations across six projects. For more managed scraping products, use the separate scraping API guide; a search tool belongs in the search API guide.
| Option | Route | Documented output or scope | Billing constraint |
|---|---|---|---|
| Firecrawl Cloud | Managed API | Scrape, crawl, map and search; markdown or structured formats | Credits per endpoint and optional features |
| Crawl4AI library | Self-hosted | Browser crawler with markdown and extraction configuration | Your infrastructure, proxies and model providers |
| Crawl4AI Cloud | Managed API | Hosted scrape, search and extraction, plus /site crawl and /scan discovery | Monthly credits; plain/browser/hard engines use different units |
| Jina Reader | Known-URL reader | Readable content from a supplied URL | Output tokens; no fixed universal page price |
| Apify Website Content Crawler | Actor on a platform | HTTP or browser crawling; text, markdown and HTML | Usage-based compute and platform resources |
| Spider | Managed API | Crawl and scrape with selectable content formats | PAYG bandwidth and compute, or separate concurrency subscription |
| Tavily Extract/Crawl | Search and retrieval API | Extraction from URLs and controlled site crawling | Extraction credits and crawl mapping costs differ |
Source: Primary sources linked in the five steps below, checked October 31, 2026. Capability is documented, not executed. Scroll sideways for more columns.
Step 1: Do you need markdown, JSON or raw HTML? #
Choose the output your downstream parser needs. Markdown is useful for readable text, a schema is useful for selected fields, and HTML preserves page structure; none establishes extraction accuracy by itself.
Firecrawl’s scrape reference exposes multiple formats. Its billing guide charges one base credit per ordinary page and four extra for LLM JSON extraction. A plain page and a JSON page therefore do not have the same credit consumption.
Crawl4AI’s parameters expose content and extraction settings. Jina Reader converts a supplied URL to readable text, defaults to browser rendering and offers a direct HTTP engine. New keys receive 10M tokens labeled non-commercial use only, and paid packs are $50 for 1B tokens or $500 for 11B. Reader counts output tokens, and its jina-ocr-v1 mode costs 40 times more tokens, so I don’t infer a fixed per-page tariff.
Spider’s reference distinguishes response transport formats from the content returned for a page. Its pricing page lists HTML, text and markdown choices. Apify Website Content Crawler documents text, markdown and HTML for HTTP and browser workloads.
For the output decision in more detail, see website-to-markdown tools for LLMs.
Step 2: Do you need JavaScript rendering, and how much of the site? #
Inspect the permitted target pages before choosing a browser. A supplied-URL reader and a recursive crawler do different jobs; discovery calls and extracted result pages can be billed separately.
Firecrawl Crawl follows links and sitemaps and defaults to a 10,000-page limit. Set a smaller explicit limit for a bounded job; do not assume every discovered link is already included in the intended budget. Apify’s input schema exposes crawler type, scope and sitemap settings. Check the selected Actor version and inputs rather than assuming defaults across the entire platform.
Tavily Crawl is a separate endpoint from Extract. Its credit guide distinguishes basic extraction at one credit per five successful URLs from advanced extraction at two, with crawl mapping additional. Tavily Keyless documents free Search and Extract without an account, but numeric caps are unpublished and Crawl requires a key. I cover Tavily from documentation only.
Firecrawl’s pricing markdown describes PDF parsing as one credit per PDF page; its billing guide describes an added credit per PDF page. I leave exact normalized PDF costs unresolved rather than pick one interpretation. Check the PDF-specific contract before budgeting a document-heavy crawl.
Step 3: Build or buy, and which licence? #
Choose self-hosting when source and infrastructure control justify operating it. Licence rights in software do not grant rights to collect or reuse someone else’s data.
Firecrawl: the repository identifies an AGPL-3.0 core and MIT components. Its current deployment comparison includes core APIs and Playwright in the default self-hosted stack, while Agent, Browser, Interact and managed enterprise surfaces are excluded. Advanced services and model extraction need configuration. The self-host guide starts with authentication off for local evaluation; production needs its own security, storage, monitoring and recovery controls.
Crawl4AI: the repository is Apache-2.0, with an additional attribution requirement in its actual LICENSE. Follow the licence text for your pinned version. The hosted service’s legal terms are a separate contract.
Jina and Spider: the Reader repository is Apache-2.0, while the ReaderLM-v2 model card uses CC-BY-NC-4.0. A code licence is not the model licence. Jina AI is part of Elastic, which sells the commercial licence for the Reader models. The open-source Spider crate is MIT; its hosted service is separately priced.
The practical comparisons are Crawl4AI vs Firecrawl, Firecrawl vs Jina Reader and Firecrawl vs Apify.
Step 4: What do 1,000 pages cost at your volume? #
The following are configured monthly totals for known URLs, not universal minimum costs or observed invoices. They exclude taxes, discovery, JSON, PDFs, optional services and account-specific limits. For a wider price table use the scraping cost comparison.
| Configuration | 10K pages | 100K pages | 1M pages | Assumption |
|---|---|---|---|---|
| Firecrawl plain pages | $44 | $99 | $749 | Hobby $19 + $25 top-up; Standard $99; Scale $749, one credit/page |
| Crawl4AI Cloud plain engine | $10 | $50 | $395.25 | 0.5 credit/page; Starter plus overage for 100K; Builder plus overage for 1M |
| Crawl4AI Cloud browser engine | $10 | $99 | $666.50 | 1 credit/page; Starter, Builder, Scale plus 250K overage |
| Crawl4AI Cloud hard engine | $20 | $158.25 | $1,336.50 | 2 credits/page; Starter plus 10K, Builder plus 75K, Scale plus 1.25M overage |
| Spider PAYG | Workload-dependent | Workload-dependent | Workload-dependent | Bytes and CPU; not normalized to a guessed page size |
| Apify Website Content Crawler | Workload-dependent | Workload-dependent | Workload-dependent | Specific Actor compute, proxy, storage and plan rates |
| Jina Reader | Token-dependent | Token-dependent | Token-dependent | $50/1B or $500/11B output tokens; tokens per page are not published |
Source: Arithmetic from Firecrawl pricing/billing and the Crawl4AI homepage checked October 31. Minimum plan cash is included; free credits are not stacked with paid plans. Scroll sideways for more columns.
Firecrawl prices are $19/$99/$399/$749 monthly for Hobby/Standard/Growth/Scale. Billing sells $5 top-ups containing 1K/2K/2.5K/5K credits respectively. The pricing page says Scale rolls unused credits over for one month, while the billing docs limit that to annual Scale plans; confirm before counting on rollover with monthly billing. Annual billing requires the full yearly payment.
Crawl4AI Cloud now uses monthly subscriptions: Starter $10/10K credits with $1-per-1K overage, Builder $99/125K with $0.79, and Scale $499/750K with $0.67. Plain/browser/hard engines consume 0.5/1/2 credits per page. This replaces the former pack tariff and December promotion. The table models consumption plus overage; bought-credit timing and monthly usage caps need account confirmation.
Spider PAYG bills $1/GB of pre-transformation content plus $0.0001/CPU minute; failed requests cost zero and balance does not expire. Its Unlimited subscription is a different route, priced per concurrency slot, and does not establish pages/hour. Apify pricing separates plan credit and resource rates. Store Actors are pay-per-event or pay-per-usage, so the chosen Actor’s model, not another Actor’s tariff, sets its cost.
Step 5: Does the tool respect robots.txt, and is scraping legal? #
The defaults below come from documentation, without execution. A robots setting alone does not settle legal permission; review the target, the data and the intended use.
| Tool | Documented robots behaviour | Source |
|---|---|---|
| Firecrawl Crawl | Respects by default; ignore option is Enterprise-only | Crawl documentation |
| Crawl4AI library | check_robots_txt defaults to false; enable it |
Parameters |
| Apify Website Content Crawler | respectRobotsTxtFile defaults to false; the console form prefills true |
Actor input schema |
| Spider | Respects by default | Pricing FAQ |
| Jina Reader | Offers an X-Robots-Txt opt-in header |
Reader reference |
| Crawl4AI Cloud / Tavily | Default not established in this review | Review the selected endpoint and applicable contract |
Do not read a licence label or a provider’s existence as a legal opinion.
Which one should you pick? #
Start with Firecrawl for a managed crawl workflow, Crawl4AI when you can own the infrastructure, Jina for supplied single URLs, or Apify when a specific Actor fits the job. Evaluate Spider using realistic byte and compute requirements. I have not measured extraction quality, speed or success, so no row wins those categories.
If you’re leaving Firecrawl, the alternatives guide compares operating models. If finding URLs is the main job, use the search API guide or free search comparison. The Firecrawl/Tavily comparison separates discovery from retrieval.
FAQ
What is the best web scraper for LLMs?
I would start with Firecrawl for a managed scrape/crawl API, Crawl4AI for source and infrastructure control, or Jina Reader for known single URLs. Apify suits an Actor-based workflow and Spider uses bandwidth plus compute billing. These are documented workflow fits, not a measured quality ranking.
Is there a free web scraper for LLMs?
Crawl4AI can be self-hosted without a software fee, subject to its licence and attribution requirement. Firecrawl advertises 1,000 monthly credits with no card; Crawl4AI Cloud gives 20 guest credits, then 1,000 monthly credits after email verification. Jina gives new keys 10M tokens labeled non-commercial use only. Credits and tokens do not represent the same number of pages. Server and optional provider costs remain separate.
Can ChatGPT do web scraping?
For a repeatable pipeline, choose a fetch tool that exposes the output, scope and billing controls you need, then pass its content to your model. This guide makes no claim about the availability or performance of a particular chatbot browsing feature.
Is AI scraping illegal?
A crawler's existence or robots setting does not decide whether a particular collection is lawful. Check target terms, permissions, data rights and the intended use. Firecrawl Crawl and Spider document respecting robots by default; Crawl4AI and Apify Website Content Crawler document an opt-in check. This is a technical comparison, not legal advice.
Can Firecrawl be self-hosted?
Yes. Its core is AGPL-3.0, with some components under MIT. The default stack includes core APIs and Playwright processing, while Agent, Browser, Interact and managed enterprise surfaces are excluded. Extra extraction or advanced services need configuration. The documented local evaluation setup has API authentication off; production needs its own security and operations design.
Is Crawl4AI free for commercial use?
The repository uses Apache-2.0 with an added attribution requirement in LICENSE. Follow the actual licence for the version used and obtain any target-data permissions. The hosted Cloud service is separate: current subscriptions start at $10/month, not the former prepaid-pack promotion.
How much does it cost to crawl 1 million pages a month?
For one-credit plain pages, Firecrawl Scale is $749/month before extras. A configured Crawl4AI Cloud plain-page example consumes 500,000 credits and costs $395.25 on Builder plus overage; 1M browser pages use 1M credits and cost $666.50 on Scale plus overage. These exclude discovery, optional features, account caps and taxes. Spider and Apify need workload-specific byte or compute assumptions.
Sources
26 sources, accessed .
Official pricing, docs, changelogs and status pages 20
- Firecrawl pricing (markdown version of the pricing page)
- Firecrawl pricing page (effective date of the price list)
- Firecrawl API reference: scrape (default format)
- Firecrawl docs: billing and credit costs
- Firecrawl docs: crawl
- Firecrawl docs: self-hosting
- Firecrawl docs: open source or cloud
- Crawl4AI Cloud pricing
- Crawl4AI docs: parameters
- Crawl4AI Cloud terms and acceptable use policy
- Jina Reader
- Jina AI API pricing (token packs)
- Apify pricing
- Apify: Website Content Crawler
- Apify: Website Content Crawler input schema
- Spider pricing
- Spider docs: API reference
- Tavily docs: API credits
- Tavily docs: Crawl endpoint
- Tavily keyless Search and Extract
