Comparison
Crawlee vs Scrapy: Pick by Language First, Then the Job
Crawlee now comes in JavaScript and Python. How it and Scrapy differ on browsers, resumable jobs, pipelines and security, and which to pick.
I would choose Crawlee JS for a fitting JS/TS crawler composition, Python Crawlee for its native asyncio workflow, or Scrapy when its Python scheduler/middleware/pipeline/export design fits the job. I would not choose from stars or an unmeasured speed claim, and Crawlee is no longer an exclusively JavaScript comparison. JS introduction, Python introduction, Scrapy architecture.
Start with the existing team’s language and required HTTP/browser, state and output behavior. All three still need hosting, target permission, network/runtime security and recovery work; source availability is not a managed fleet or a proven deployment.
Short on time?
Here's my take
- If your team writes JS or TS
- Crawlee JS
- Inspect the stable 3.x API and selected HTTP/browser path, keeping 4.0 RC separate.
- If you want native Python asyncio
- Python Crawlee
- It is a distinct implementation; do not assume JS feature parity.
- If you need Python pipelines and exports
- Scrapy
- Scheduler/middleware/JOBDIR and pipeline/export composition can be the decisive fit.
Prices verified . Sponsors cannot buy a verdict or a position.
Which versions am I comparing? #
This comparison covers Crawlee JS/TypeScript 3.18.2, Python Crawlee 1.11.2 and Python Scrapy 2.20.0. Versions/specifications verified: 2026-11-11. I use stable tagged metadata instead of silently applying a development branch’s requirement or installing a release candidate. Crawlee JS tag, Python Crawlee tag, Scrapy tag.
| Implementation | Stable release reviewed | Published minimum / licence | Version caveat |
|---|---|---|---|
| Crawlee JS | 3.18.2, September 29 | Tagged Node 16 minimum; Apache-2.0 | Metadata minimum is not maintained-runtime advice; 4.0 RC separate. Repo, releases, tag |
| Crawlee Python | 1.11.2, November 10 | Python 3.10+; Apache-2.0 | Distinct package/API, no assumed JS parity. Repo, releases, tag |
| Scrapy | 2.20.0, November 6 | Python 3.11+; BSD-3-Clause | 2.20 raised the minimum from 3.10 to 3.11; check your runtime before upgrading. Repo, releases, tag, install |
I would select a maintained runtime appropriate to the deployment, not treat the lowest permitted package version as security advice. Stable release notes and requirements also need to remain attached to the implementation rather than blended into one “Crawlee” version. JS tag, Python tag, Scrapy install.
Do you need a browser at all? #
Choose static HTTP when it supplies the required fields, and add browser work for a demonstrated rendering/interaction requirement. Crawlee JS’s Cheerio crawler and Playwright/Puppeteer browser paths are different capabilities with different operating costs. Cheerio, Playwright, JS introduction.
Python Crawlee offers Parsel/BeautifulSoup/Playwright and optional extras with native asyncio. That supports a Python choice, but I cannot infer full JS/Python feature or recovery parity from the name. Python introduction, repository.
Scrapy’s core overview/architecture describes scheduler, downloader, middleware and pipelines, not automatic rendering on every request. The optional scrapy-playwright plugin requires Python 3.10, Scrapy 2.7 and Playwright 1.40 at minimum, with separate browser binaries and reactor/Windows constraints. Overview, architecture, plugin, asyncio.
I would verify the selected environment in isolation before adopting an integration recipe. No browser/plugin was installed for this research, so the documented path is not a successful runtime check. If rendering does turn out to be essential, my guide to self-hosted vs managed headless browsers counts the upkeep, and search, fetch or browser helps confirm you need one.
Does Scrapy support asyncio? #
Current Scrapy documentation includes coroutine process/runner and asyncio support, with experimental aiohttp download handling and a RemoteControl extension added in 2.19.0. Describing it as “Twisted-only, no asyncio” would miss the reviewed release’s actual interfaces and environment requirements. Asyncio, extensions, installation.
Python Crawlee is native asyncio, while JS Crawlee has its own resource-aware pool and crawler composition. I would choose from actual client/middleware behavior and the team’s language rather than convert that design distinction into a speed or reliability rank. Python intro, AutoscaledPool.
How do they handle queues, resuming and output? #
Crawlee’s request storage, sessions and resource-aware pool help compose a crawler. They do not automatically supply managed fleet operations, deployment rollback or proof that every queue resumes correctly on your storage backend. Request storage, sessions, pool.
Scrapy offers scheduler/middleware composition, JOBDIR persistence, item pipelines and feed exports. I would inspect those contracts when resumable jobs, item validation or a specific export path is the requirement, then rehearse interruption/recovery separately. Architecture, jobs, pipelines, exports. If you’d rather not host Scrapy yourself, Scrapy Cloud is a separately billed product; I cover it near the end of my Zyte API pricing breakdown.
| Need | Crawlee side to inspect | Scrapy side to inspect | Verification still needed |
|---|---|---|---|
| Work state | Request storage/queue | Scheduler/JOBDIR | Resume, duplicates and storage durability |
| Session/retry policy | Sessions and crawler options | Middleware/retry composition | Actual target and failed-attempt outcomes |
| Resource controls | AutoscaledPool/options | Settings and throttle | Environment-specific limits/throughput |
| Data acceptance | Your validation/export code | Item pipelines/feed exports | Required fields, freshness and distinct records |
| Browser lifecycle | Selected crawler implementation | Optional Playwright plugin | Page/context closure and isolation |
Sources: storage, sessions, options, Scrapy jobs, middleware, settings, exports, plugin.
Set politeness and request limits yourself #
JS Crawlee’s request limits can overshoot under parallelism; three request retries and ten session rotations are separate controls. I would bound the client work and target scope rather than equate a setting with an exact global request/billing count. Crawler options.
Scrapy robots documentation distinguishes a True value from fallback False and generated-project True, so check the actual settings rather than quote one ambiguous default. AutoThrottle, redirect/retry/proxy middleware and concurrency settings are separate from target permission. Settings, throttle, middleware.
Python Crawlee robots is opt-in false. Same-host checks and robots do not supply target rights or private-network isolation, so I would make politeness and allowed-target settings explicit in the deployed project. Security guide.
Security is your job, not the framework’s #
Python Crawlee accepts HTTP(S)/same hostname by default but follows intermediate redirects and does not itself block private IPs. For untrusted input, constrain schemes, hosts, DNS, redirects and response streams, backed by isolated egress; a hostname check alone is not SSRF protection. Crawlee security, OWASP.
Browser pages execute untrusted code, so preserve the browser sandbox and isolate runtime/network/credentials. I would not copy insecure container/root flags or disabled TLS settings as a production shortcut. Playwright container guidance, Crawlee security.
Scrapy’s RemoteControl extension (added in 2.19) exposes Python execution through HTTP in a running crawl, with confidential token-bearing job files. Optional coding-agent/MCP integration can connect to a live local process; this is a control plane to protect, not a plugin I installed or a public endpoint to expose. RemoteControl, agent integration.
I would validate/escape outputs and defend downstream model prompts against page instructions. A retrieved page is untrusted content, not authority to change the crawl’s permissions or credentials. Crawlee security.
What do the releases and old issues tell you? #
Python release notes describe HTML/XML/base-URL/content-type and container-resource/429 fixes. Those are changes in the project, not my output-quality or reliability measurements and not a user-sentiment sample. Python releases.
Crawlee issue 1829 is a specific 2023 page-close hang linked from a later September fix; scrapy-playwright issue 191 is an April 2023 lifecycle callback race. I would use them as concrete reasons to verify page/context cleanup, without claiming recurring current failure rates or general customer consensus. Crawlee issue, JS releases, plugin issue.
Unclosed browser pages/contexts can stall jobs according to the plugin documentation. That is a documented lifecycle concern to test in the selected setup, not evidence that either whole framework is broadly unreliable. Plugin.
Crawlee or Scrapy: which would I use? #
I would use Crawlee JS when its JS/TS HTTP/browser composition fits the existing stack, Python Crawlee when its asyncio interface matches the Python job, or Scrapy when scheduler/middleware/pipeline/export composition is the better fit. I would not migrate a working pipeline merely because another project has more stars or a newer version number. If the output feeds an LLM, my roundup of open-source web crawlers for LLMs puts both next to Crawl4AI, Firecrawl and Trafilatura.
Before choosing, validate representative authorized output, queue/resume behavior, browser lifecycle, actual robots/throttle settings and isolated network/runtime controls. This article supplies no matched crawl, production deployment or independent Reddit consensus that makes one the universal winner. If you pick Scrapy but don’t want to host it, Apify vs Scrapy looks at running your spider on Apify instead.
The scraping API pillar connects owned-code and purchased-delivery choices. Build versus buy models operating work, Apify alternatives discusses platform migration, Firecrawl self-hosting covers an ingestion path, and web scraping for RAG defines usable downstream data.
FAQ #
Is Crawlee only JavaScript? #
No. JS and Python Crawlee have distinct stable versions and APIs; full parity was not verified. JS, Python.
Is it true Scrapy doesn’t work with asyncio? #
That simplification is obsolete for the reviewed docs, which include coroutine/asyncio interfaces and explicit runtime requirements. Asyncio.
Does Scrapy render JavaScript pages by default? #
No such claim is made. Its optional Playwright path has separate dependencies, browser binaries and reactor/OS constraints. Plugin, asyncio.
Do Crawlee’s host checks stop every unsafe request? #
No. Python Crawlee follows intermediate redirects and does not block private IPs itself; isolate egress and constrain URL/DNS/redirect behavior. Security, OWASP.
Do old GitHub issues mean one is unreliable? #
No. The two historical lifecycle reports are specific cases, not failure rates or a systematic review corpus. Crawlee issue, plugin issue.
Sources
35 sources, accessed .
Official pricing, docs, changelogs and status pages 23
- JS Crawlee intro
- AutoscaledPool
- Request storage
- Session management
- Crawler options
- PlaywrightCrawler
- CheerioCrawler
- Python Crawlee intro
- Crawlee Python security
- Scrapy overview
- Scrapy installation docs
- Scrapy architecture
- Scrapy jobs
- Item pipeline
- Feed exports
- AutoThrottle
- Middleware
- Scrapy asyncio
- Scrapy settings
- Coding-agent integration
- RemoteControl extension
- OWASP SSRF
- Playwright Docker
