How to choose
The Best Web Scraper for LLMs and RAG Starts With Output
Firecrawl, Crawl4AI, Jina Reader, Apify or Spider? How to pick a scraper for LLMs and RAG by output, rendering, licence, crawl scope and cost per page.

The best web scraper for an LLM or RAG pipeline is the one that returns the content you need within a defined budget. I choose output first, then rendering and crawl scope, then the operating model, cost and robots behaviour. I rechecked the primary pricing and feature sources retained below on November 13, 2026; no scraping or crawling jobs were run for this page.
Short on time?
My pick, by what you're building
- Managed scrape and crawl
- Firecrawl
- One managed API with markdown and optional structured extraction. Standard is $99/month for 100K credits; plain pages use one credit before extras.
- Run the software yourself
- Crawl4AI
- Control the source, browser settings and infrastructure. Apache-2.0 with an added attribution requirement; servers and providers still cost money.
- Known URLs, one at a time
- Jina Reader
- A URL prefix returns readable content. New keys get 10M non-commercial tokens; on 31 Oct paid packs were $50 for 1B tokens or $500 for 11B, billed on output tokens.
- An Actor-based workflow
- Apify
- Choose the specific Actor and its billing model. Website Content Crawler is pay per usage; other Actors need separate pricing checks.
- Bandwidth-based billing
- Spider
- PAYG bills bandwidth and CPU, and a failed attempt still bills what it used. Page weight and settings prevent a universal per-page price.
Prices verified . Documented fit and configured list prices, not a measured quality or speed ranking.
The short list, side by side #
The current short list of web scrapers for LLMs and RAG below covers seven configurations across six projects. For more managed scraping products, use the separate scraping API guide; a search tool belongs in the search API guide.
| Option | Route | Documented output or scope | Billing constraint |
|---|---|---|---|
| Firecrawl Cloud | Managed API | Scrape, crawl, map and search; markdown or structured formats | Credits per endpoint and optional features |
| Crawl4AI library | Self-hosted | Browser crawler with markdown and extraction configuration | Your infrastructure, proxies and model providers |
| Crawl4AI Cloud | Managed API | Hosted scrape, search and extraction, plus /site crawl and /scan discovery | Monthly credits; plain/browser/hard engines use different units |
| Jina Reader | Known-URL reader | Readable content from a supplied URL | Output tokens; no fixed universal page price |
| Apify Website Content Crawler | Actor on a platform | HTTP or browser crawling; text, markdown and HTML | Usage-based compute and platform resources |
| Spider | Managed API | Crawl and scrape with selectable content formats | PAYG bandwidth and compute, or separate concurrency subscription |
| Tavily Extract/Crawl | Search and retrieval API | Extraction from URLs and controlled site crawling | Extraction credits and crawl mapping costs differ |
Source: Primary sources linked in the five steps below, checked November 13, 2026. Capability is documented, not executed. Scroll sideways for more columns.
Step 1: What should the scraper hand your model? #
Choose the output your downstream parser needs: markdown, JSON or raw HTML. Markdown is useful for readable text, a schema is useful for selected fields, and HTML preserves page structure; none establishes extraction accuracy by itself.

Firecrawl’s scrape reference exposes multiple formats. Its billing guide charges one base credit per ordinary page and four extra for LLM JSON extraction. A plain page and a JSON page therefore do not have the same credit consumption.
Crawl4AI’s parameters expose content and extraction settings. Jina Reader converts a supplied URL to readable text, defaults to browser rendering and offers a direct HTTP engine. New keys receive 10M tokens, which Jina’s pricing page labeled non-commercial use only on 31 October, when its paid packs were $50 for 1B tokens or $500 for 11B. Reader counts output tokens, and its jina-ocr-v1 mode costs 40 times more tokens, so I don’t infer a fixed per-page tariff.
Spider’s reference distinguishes response transport formats from the content returned for a page. Its pricing page lists HTML, text and markdown choices. Apify Website Content Crawler documents text, markdown and HTML for HTTP and browser workloads.
For the output decision in more detail, see website-to-markdown tools for LLMs and, for schema fields, LLM structured data extraction.
Step 2: Do you need a browser, and how much of the site? #
Inspect the permitted target pages before choosing a browser for JavaScript rendering. A supplied-URL reader and a recursive crawler do different jobs; discovery calls and extracted result pages can be billed separately.
Firecrawl Crawl follows links and sitemaps and defaults to a 10,000-page limit. Set a smaller explicit limit for a bounded job; do not assume every discovered link is already included in the intended budget. Apify’s input schema exposes crawler type, scope and sitemap settings. Check the selected Actor version and inputs rather than assuming defaults across the entire platform.
Tavily Crawl is a separate endpoint from Extract. Its credit guide distinguishes basic extraction at one credit per five successful URLs from advanced extraction at two, with crawl mapping additional. Tavily Keyless documents free Search and Extract without an account, but numeric caps are unpublished and Crawl requires a key. I cover Tavily from documentation only.
Firecrawl’s pricing markdown describes PDF parsing as one credit per PDF page; its billing guide describes an added credit per PDF page. I leave exact normalized PDF costs unresolved rather than pick one interpretation. Check the PDF-specific contract before budgeting a document-heavy crawl.
Step 3: Build or buy, and which licence? #
Choose self-hosting when source and infrastructure control justify operating it. Licence rights in software do not grant rights to collect or reuse someone else’s data.

Firecrawl: the repository identifies an AGPL-3.0 core and MIT components. Its current deployment comparison includes core APIs and Playwright in the default self-hosted stack, while Agent, Browser, Interact and managed enterprise surfaces are excluded. Advanced services and model extraction need configuration. The self-host guide starts with authentication off for local evaluation; production needs its own security, storage, monitoring and recovery controls.
Crawl4AI: the repository is Apache-2.0, with an additional attribution requirement in its actual LICENSE. Follow the licence text for your pinned version. The hosted service’s legal terms are a separate contract.
Jina and Spider: the Reader repository is Apache-2.0, while the ReaderLM-v2 model card uses CC-BY-NC-4.0. A code licence is not the model licence. Jina AI is part of Elastic, which sells the commercial licence for the Reader models. The open-source Spider crate is MIT; its hosted service is separately priced.
The practical comparisons are Crawl4AI vs Firecrawl, Firecrawl vs Jina Reader and Firecrawl vs Apify.
Step 4: What will your pages actually cost? #
The following are configured monthly totals for known URLs at 10K, 100K and 1M pages, not universal minimum costs or observed invoices. They exclude taxes, discovery, JSON, PDFs, optional services and account-specific limits. For a wider price table use the scraping cost comparison.
| Configuration | 10K pages | 100K pages | 1M pages | Assumption |
|---|---|---|---|---|
| Firecrawl plain pages | $44 | $99 | $749 | Hobby $19 + $25 top-up; Standard $99; Scale $749, one credit/page |
| Crawl4AI Cloud plain engine | $10 | $10 | $99 | 0.1 credit/page; Starter for 10K and 100K; Builder for 1M, since 100K credits exceed Starter’s 50K monthly cap |
| Crawl4AI Cloud browser engine | $10 | $50 | $395.25 | 0.5 credit/page; Starter, Starter plus 40K overage, Builder plus 375K overage |
| Crawl4AI Cloud hard engine | $20 | $158.25 | $1,336.50 | 2 credits/page; Starter plus 10K, Builder plus 75K, Scale plus 1.25M overage |
| Spider PAYG | Workload-dependent | Workload-dependent | Workload-dependent | Bytes and CPU; not normalized to a guessed page size |
| Apify Website Content Crawler | Workload-dependent | Workload-dependent | Workload-dependent | Specific Actor compute, proxy, storage and plan rates |
| Jina Reader | Token-dependent | Token-dependent | Token-dependent | $50/1B or $500/11B output tokens (31 Oct); tokens per page are not published |
Source: Arithmetic from Firecrawl pricing/billing and the Crawl4AI homepage and price API checked November 13. Minimum plan cash is included; free credits are not stacked with paid plans. Scroll sideways for more columns.
Firecrawl prices are $19/$99/$399/$749 monthly for Hobby/Standard/Growth/Scale. Billing sells $5 top-ups containing 1K/2K/2.5K/5K credits respectively. The pricing page says Scale rolls unused credits over for one month, while the billing docs limit that to annual Scale plans; confirm before counting on rollover with monthly billing. Annual billing requires the full yearly payment.
Crawl4AI Cloud now uses monthly subscriptions: Starter $10/10K credits with $1-per-1K overage, Builder $99/125K with $0.79, and Scale $499/750K with $0.67. Plain/browser/hard engines consume 0.1/0.5/2 credits per page (13 November; 0.5/1/2 on 31 October). This replaces the former pack tariff and December promotion. The table models consumption plus overage; bought-credit timing and monthly usage caps need account confirmation.
Spider PAYG bills $1/GB of pre-transformation content plus $0.0001/CPU minute; a failed attempt still bills the bandwidth and compute it used, whatever status the page returns, and balance does not expire. Its Unlimited subscription is a different route, priced per concurrency slot, and does not establish pages/hour. Apify pricing separates plan credit and resource rates. Store Actors are pay-per-event or pay-per-usage, so the chosen Actor’s model, not another Actor’s tariff, sets its cost.
Step 5: What about robots.txt and the law? #
The robots.txt defaults below come from documentation, without execution. A robots setting alone does not settle legal permission; review the target, the data and the intended use.
| Tool | Documented robots behaviour | Source |
|---|---|---|
| Firecrawl Crawl | Respects by default; ignore option is Enterprise-only | Crawl documentation |
| Crawl4AI library | check_robots_txt defaults to false; enable it |
Parameters |
| Apify Website Content Crawler | respectRobotsTxtFile defaults to false; the console form prefills true |
Actor input schema |
| Spider | Respects by default | Pricing FAQ |
| Jina Reader | Offers an X-Robots-Txt opt-in header |
Reader reference |
| Crawl4AI Cloud / Tavily | Default not established in this review | Review the selected endpoint and applicable contract |
Do not read a licence label or a provider’s existence as a legal opinion.
Which one should you pick? #
Start with Firecrawl for a managed crawl workflow, Crawl4AI when you can own the infrastructure, Jina for supplied single URLs, or Apify when a specific Actor fits the job. Evaluate Spider using realistic byte and compute requirements. I have not measured extraction quality, speed or success, so no row wins those categories.
If you’re leaving Firecrawl, the alternatives guide compares operating models. If finding URLs is the main job, use the search API guide or free search comparison. The Firecrawl/Tavily comparison separates discovery from retrieval, and web scraping for RAG covers what to settle before indexing.
FAQ
What's the best web scraper for LLMs?
I would start with Firecrawl for a managed scrape/crawl API, Crawl4AI for source and infrastructure control, or Jina Reader for known single URLs. Apify suits an Actor-based workflow and Spider uses bandwidth plus compute billing. These are documented workflow fits, not a measured quality ranking.
Is there a free web scraper for LLMs?
Crawl4AI can be self-hosted without a software fee, subject to its licence and attribution requirement. Firecrawl advertises 1,000 monthly credits with no card; Crawl4AI Cloud lists 20 guest credits and advertises 1,000 free monthly credits with no card. Jina gives new keys 10M tokens labeled non-commercial use only. Credits and tokens do not represent the same number of pages. Server and optional provider costs remain separate.
Can ChatGPT do web scraping?
For a repeatable pipeline, choose a fetch tool that exposes the output, scope and billing controls you need, then pass its content to your model. This guide makes no claim about the availability or performance of a particular chatbot browsing feature.
Is AI scraping illegal?
A crawler's existence or robots setting does not decide whether a particular collection is lawful. Check target terms, permissions, data rights and the intended use. Firecrawl Crawl and Spider document respecting robots by default; Crawl4AI and Apify Website Content Crawler document an opt-in check. This is a technical comparison, not legal advice.
Can Firecrawl be self-hosted?
Yes. Its core is AGPL-3.0, with some components under MIT. The default stack includes core APIs and Playwright processing, while Agent, Browser, Interact and managed enterprise surfaces are excluded. Extra extraction or advanced services need configuration. The documented local evaluation setup has API authentication off; production needs its own security and operations design.
Is Crawl4AI free for commercial use?
The repository uses Apache-2.0 with an added attribution requirement in LICENSE. Follow the actual licence for the version used and obtain any target-data permissions. The hosted Cloud service is separate: current subscriptions start at $10/month, not the former prepaid-pack promotion.
How much does it cost to crawl a million pages a month?
For one-credit plain pages, Firecrawl Scale is $749/month before extras. A configured Crawl4AI Cloud plain-page example consumes 100,000 credits and costs $99 on Builder; 1M browser pages use 500,000 credits and cost $395.25 on Builder plus overage. These exclude discovery, optional features, account caps and taxes. Spider and Apify need workload-specific byte or compute assumptions.
Sources
26 sources, accessed .
Official pricing, docs, changelogs and status pages 20
- Firecrawl pricing (markdown version of the pricing page)
- Firecrawl pricing page (effective date of the price list)
- Firecrawl API reference: scrape (default format)
- Firecrawl docs: billing and credit costs
- Firecrawl docs: crawl
- Firecrawl docs: self-hosting
- Firecrawl docs: open source or cloud
- Crawl4AI Cloud pricing
- Crawl4AI docs: parameters
- Crawl4AI Cloud terms and acceptable use policy
- Jina Reader
- Jina AI API pricing (token packs)
- Apify pricing
- Apify: Website Content Crawler
- Apify: Website Content Crawler input schema
- Spider pricing
- Spider docs: API reference
- Tavily docs: API credits
- Tavily docs: Crawl endpoint
- Tavily keyless Search and Extract


