Roundup
Which Open-Source Web Crawler for LLMs Fits Your Stack?
Crawl4AI, Firecrawl, Crawlee, Scrapy, Trafilatura and ScrapeGraphAI compared by language, output, license and the work self-hosting leaves you.
I would start with Crawl4AI for a Python browser-to-Markdown workflow, Crawlee for a TypeScript crawling pipeline, and Firecrawl when its HTTP API shape is the deciding requirement. Scrapy, Trafilatura and ScrapeGraphAI fit narrower jobs; I would choose them for those jobs rather than call every repository a replacement for a managed crawl API.
My order below reflects the reader’s likely workload, not a speed or accuracy ranking. I compared public repositories, licenses, documentation and version-specific reports; I did not run any of these tools.
Short on time?
Here's my take
- If you work in Python
- Crawl4AI
- A documented browser/extraction route with deterministic and model-based options; inspect its attribution appendix and deployment defaults.
- If you work in TypeScript
- Crawlee
- Queues, storage and HTTP/browser crawlers fit an application you maintain yourself.
- If you want a Firecrawl-style API
- Firecrawl core
- Use its documented self-hosted interface while budgeting the features and operations the default stack does not supply.
Prices verified . Repository and documentation review, 13 October 2026, re-checked 11 November 2026. No measured performance winner is established.
The six, and what each one is for #
| My shortlist | Best starting job | License evidence | Responsibility I would budget |
|---|---|---|---|
| Crawl4AI | Python/browser-to-Markdown and structured extraction | Apache text plus attribution appendix | Browser resources, policy configuration, updates and optional model bills |
| Firecrawl core | A self-hosted crawl/scrape API | AGPL-3.0 | Auth, persistence, missing managed capabilities and optional model services |
| Crawlee | TypeScript queues and HTTP/browser workflows | Apache-2.0 | Your extraction code, browser dependencies, scheduling and storage |
| Scrapy | Maintained Python spider projects | BSD-3-Clause | Spider rules, deployment and explicit project settings |
| Trafilatura | Text/metadata extraction inside a pipeline | Apache-2.0 | Fetching policy, integration and output validation |
| ScrapeGraphAI | Model-driven extraction graphs | MIT license body | Model configuration, tokens and graph/output checks |
Checked 13 October 2026 and re-checked 11 November 2026. License names describe the documents I read; they do not settle the obligations of a particular deployment. I would review the full license before distributing or offering a modified service.
Crawl4AI #
Crawl4AI is my first shortlist when Python, browser rendering and LLM-facing output belong in one workflow. Its repository and docs cover browser crawling, deep traversal and both deterministic and LLM extraction strategies. Repository, deep crawling, extraction choices.
I would start with CSS/XPath for stable fields, then add a supplied model when the extraction problem is semantic rather than a fixed page structure. A model-based strategy still has its own provider configuration and token bill. LLM strategies.
Its license includes an attribution appendix after the Apache-2.0 text. I would keep that visible in a comparison instead of copying an old one-word license label. Current license.
For Docker, the guide calls for at least 4 GB available to the container and explains token/binding behavior. That is a prerequisite, not proof of how many pages your host can process. Self-host guide.
Firecrawl #
Firecrawl core fits when a familiar API interface matters more than a general crawler framework. The self-host guide and repository describe the local stack, while the cloud comparison explains what is absent by default. Repository, self-host guide, cloud versus open source.
Do not budget for cloud parity from the repository name. The default stack does not supply Fire-engine, screenshots or page actions, or the managed Agent/Browser/interact capabilities; LLM formats require a configured provider or Ollama. Default-stack scope, self-host guide.
The core is AGPL-3.0. I would examine the actual license and my deployment requirements before choosing it for a hosted service. The more detailed buying question is in Firecrawl self-host versus cloud, with Crawl4AI versus Firecrawl for the two-product comparison.
Crawlee #
Crawlee is my TypeScript shortlist when I want to own the crawl queue and extraction pipeline. Its JS repository supplies HTTP/browser crawlers, queues and storage; its documentation frames the application configuration you still maintain. Repository, introduction.
Playwright is a separately installed dependency, so I would include browser setup in the project rather than assume it appears with every package install. Its license is Apache-2.0. Repository setup, license.
I would choose this route when a maintained application is the goal. If the requirement is simply “send a URL and receive a cleaned document,” compare that engineering work with an API before making the repository’s license price your total budget. My build-versus-buy breakdown lists the work to count, and Crawlee vs Scrapy helps if those two are your finalists.
Scrapy #
Scrapy fits a Python spider project whose rules and maintenance you intend to own. I would shortlist it for that framework role rather than assume it is an LLM-ready cloud endpoint. Repository, BSD-3-Clause license.
Its robots settings deserve an explicit check. The docs distinguish the generated-project setting from the historical fallback, so a broad statement that every Scrapy crawl follows one default would be misleading. Settings reference.
My acceptance check would include the crawl scope, extracted fields and behavior on a rejected or unexpected page. A spider that returns something is not necessarily returning the document your downstream system intended to ingest.
Trafilatura #
Trafilatura fits as a text and metadata extraction component. Its Python usage docs describe fetching and extracting with output choices, and the repository includes crawler utilities; that does not make it a drop-in managed API. Usage, repository.
The current license is Apache-2.0. I would use the current file rather than carry forward an older label from a comparison. License.
I would pair it with explicit fetch permissions, provenance and validation in a pipeline I control. I would also budget the metadata and refresh work that a clean text string alone cannot settle; my guide to web scraping for RAG walks through that work.
ScrapeGraphAI #
ScrapeGraphAI is worth a shortlist when a graph and model-driven extraction match the task. Its repository documents that approach, and the current raw license body confirms MIT. I have not independently measured extraction quality or token use. Repository.
The current license body, re-checked 11 November 2026, includes the copyright/permission-notice condition. I would also confirm the model, graph inputs and output validation rather than infer that “AI” prevents incorrect fields.
For the schema decision itself, read LLM structured data extraction. It separates model-generated shape from a field value supported by the source page.
Check the defaults before you crawl #
Crawl4AI’s check_robots_txt defaults to false; Firecrawl crawl respects robots by default, with an Enterprise-only override. I would set and verify the intended policy explicitly rather than assume that two commands with “crawl” in their name enforce identical scope. Crawl4AI parameters, Firecrawl crawl.
Robots is not an access authorization system. Establish permission and allowed targets separately from the robots setting. RFC 9309.
Crawl4AI’s changelog records security releases in v0.9.3, v0.9.4 and v0.9.5 (14 October 2026, six advisories closed). Its memory-accounting, lifecycle and auth issue threads are version-specific; I would inspect the installed version and fix status before calling an old complaint a current defect. Changelog, memory report, lifecycle report, closed auth report.
So which one should you run? #
My first split is who will own the running application. Shortlist Crawl4AI for Python/browser extraction, Crawlee for TypeScript queues and Scrapy for maintained Python spiders; use Trafilatura when extraction is a component inside another system.
Shortlist Firecrawl core for its API shape with the default-stack gaps made explicit. Consider ScrapeGraphAI for a model-driven graph after reviewing its actual license and validating the fields you need.
For a managed option, see Firecrawl alternatives. For output format, see website-to-Markdown choices; the crawl API hub connects the wider buying routes, and web scraping MCP servers are a separate decision from the crawler shortlist.
FAQ #
Is self-hosted Firecrawl the same as the cloud version? #
The documented default stack has meaningful exclusions. Compare the needed capability with the cloud/open-source table before choosing a deployment route.
Do I need an LLM for every extracted field? #
No. Crawl4AI documents LLM-free CSS/XPath extraction; I would use it first for a stable known structure. Deterministic strategies.
Is an open-source crawler free to run? #
I would budget servers, storage, optional model calls and maintenance separately from the source license. The self-host cost guide makes that distinction without inventing throughput.
Which one is most accurate? #
This research cannot establish that. The shortlist compares documented fit, licenses and responsibilities, with no logged fair output-quality comparison.
Sources
31 sources, accessed .
Official pricing, docs, changelogs and status pages 12
- Default stack limits and production duties
- Cloud versus default stack
- Scope, robots and result expiry
- RAM, auth and local MCP endpoints
- Robots and browser/run configuration
- BFS/DFS/best-first bounded traversal
- Model providers and LLM extraction
- CSS/XPath and deterministic extraction
- Framework configuration and responsibilities
- Robots/project defaults and concurrency
- Fetch/extract API and output choices
- Robots protocol is not access authorization
GitHub code, issues and releases 19
- Core repository and SDK inventory
- Core AGPL license
- Local stack companion instructions
- Python crawler and output capabilities
- Apache text plus attribution appendix
- Dated security and behavior changes
- Published security advisory inventory
- Queues, HTTP/browser crawlers and storage
- Apache-2.0 license
- Python crawler framework
- BSD-3-Clause license
- Text/metadata extractor and crawler utilities
- Current Apache-2.0 license
- Graph extraction; README states MIT
- ScrapeGraphAI: current MIT license body
- User report: cgroup memory guard
- Version-specific lifecycle report and replies
- Compose/auth complaint closed in v0.9.3
- Current limits and hardening settings
Last updated: 13 October 2026. Researched and drafted with AI tools; human editorial review and publication approval are pending.
