Explainer
Monitoring a web scraper: catching the quiet failures
Your scraper can run green and still ship bad data. What to monitor, starter alert rules, and the tools that check it, from Spidermon to Apify.

To monitor a web scraper, check the data, not just the job. A scraper can finish green and still ship empty, partial or wrong records, so start with three checks: a heartbeat that proves it ran, a record count compared with last week’s runs, and the fill rate of your key fields.
I read the Spidermon, Scrapy, Crawlee, Apify and Great Expectations docs, the billing rules of two scraping APIs, and the pages that rank for this topic. Below is what to measure, which alerts should wake someone up, and the tool I’d pick for each stack.
Short on time?
Short on time? Here's my take.
- Scrapy project
- Spidermon
- Turn on the spider-close suite, add a JSON Schema per item type and send alerts to Slack. Add CLOSESPIDER_TIMEOUT_NO_ITEM so empty runs end loudly.
- Running on Apify
- Apify's built-in monitoring
- Free per Apify's docs. Give the dataset a schema so the field-statistics alerts work.
- Data lands in a warehouse or Spark
- GX Core Checkpoints
- Validate the output table on every load and send failures to Slack or Teams.
- Small mixed fleet, any language
- Heartbeat, record-count baseline and schema check
- Files, SQLite and email are enough, per one practitioner running 77 scrapers.
- You want a hosted dashboard
- ScrapeOps Monitoring
- SDKs for Scrapy and Python Requests. Check the price first; I couldn't verify it.
Prices verified . Sponsors cannot buy a verdict or a position.
Why does a scraper fail without telling you? #
Because the scheduler judges the job, and nobody judges the data. The process starts, fetches pages, writes a file and exits with code zero, while the price field has been empty since the site moved it into a new element.
Shifter’s monitoring guide puts it in one line: “A scraper fails silently: the cron fires, the logs say 200, the data goes wrong.” Apify’s 2023 post describes the same thing from the other end: “Your scraper might be working fine for weeks, then suddenly, it breaks, yielding zero results.”
It doesn’t take much. A Hacker News commenter noted back in 2013 that “cosmetic changes break scrapers silently”. The pages I read describe the same three shapes: a layout change empties a field, a block page comes back as a normal response, or a listing quietly returns fewer items.
PromptCloud draws the line I find most useful: “Monitoring answers whether the job ran. Observability answers whether the data still reflects reality.” It also claims infrastructure failures get caught “within minutes” while missing fields surface days later, but it shows no method, so read that as their anecdote.
The Zyte guide that ranks first here describes monitors as “additional code segments integrated into your web scraping project alongside the scraper’s code” (Zyte KB). That’s the right model: the checks live next to the scraper and run every time it does. What none of the ranking pages give you is the actual settings, so that’s what the rest of this page is.
One boundary. Blocks appear here only as a metric: if your status codes fill with 403s and 429s, the answer is to slow down or stop, which my rate limit guide covers.
What should you actually measure? #
Two layers. Job health tells you the scraper ran and ended normally; data health tells you that what it produced is complete and correct. Every ranking page says teams miss the second layer, and none shows where the numbers come from.
| Metric | What it catches | Where you get it |
|---|---|---|
| Heartbeat / last successful run | The job didn’t run or didn’t finish | Your own heartbeat record; Apify run-status alerts |
| Finish reason | A run that ended on a timeout, error or close condition | Scrapy’s finish_reason stat; Spidermon FinishReason (expects finished) |
| Items vs recent runs | Layout changes, block pages, shrinking listings | Scrapy item_scraped_count; Spidermon ItemCount, PeriodicItemCount; Apify “Lower than” alert |
| Field coverage | One selector broke while the rest still work | Spidermon FieldCoverage; Apify dataset field statistics; ScrapeOps |
| Schema errors | Wrong types, malformed values | Spidermon item validation (JSON Schema); GX Expectations |
| Status-code mix | Blocks, rate limits, server errors | Scrapy downloader/response_status_count/{code}; Spidermon UnwantedHTTPCodes; Crawlee registerStatusCode |
| Retries | A target getting slower or stricter | Crawlee requestRetryHistogram; Spidermon RetryCount |
| Errors | Exceptions in parsing or downloading | Scrapy log_count/ERROR; Spidermon ErrorCount; Crawlee errorTracker |
| Duration | Hangs and slowdowns | Scrapy elapsed_time_seconds; Spidermon PeriodicExecutionTime; Apify “Higher than” alert |
| Freshness | Data that stopped updating | The timestamp of the newest good record in your store |
| Cost per good record | Paying for pages that produced nothing | Your own sum: credits spent ÷ records that pass validation |
Features verified 2026-11-17: Scrapy stats, Scrapy extensions, Spidermon monitors, Crawlee Statistics, Apify monitoring, GX Core, ScrapeOps.
If you only add two data checks, make them item count and field coverage. A block page or a layout change can leave the job layer looking perfectly normal, and these two are where it shows.
Compare each target against its own history, not one global number. Shifter argues that “absolute thresholds either miss real problems or cry wolf constantly”, and recommends a baseline per target plus p50, p95 and p99 latency (Shifter). Its example figures are illustrations, not measurements.
Schema checks apply just as much when a model does the parsing. If you use an LLM for structured data extraction, validate its output against the same JSON Schema you’d use for a CSS-selector scraper.

Which alerts should wake someone up? #
Very few. Alex Spinov, who describes monitoring 77 scrapers with about 200 lines of Python, a JSON heartbeat per run and counts in SQLite, puts it bluntly: “The biggest mistake is alerting on everything.”
Here are the starter rules I’d use, sorted into three tiers. The tiers are my house rule; each number is someone else’s, named next to it.
Starter alert rules (start here, then tune per target)
Page someone now
- No heartbeat: the run didn’t happen or didn’t finish. A May 2026 HN comment says a monitoring system “has to treat absence of data as a signal”.
- Zero items two runs in a row (Spinov’s critical rule).
- The run failed, or its finish reason isn’t
finished(Spidermon FinishReason default; Apify run-status alert).Daily digest
- Item count below half the trailing 7-day average (Spinov’s flag).
- A key field’s coverage falls below your rule (Spidermon FieldCoverage, tolerance 0 by default).
- More than 10 unwanted HTTP codes in a run (Spidermon’s default maximum).
- A Bright Data scraper’s success rate crosses your threshold (its docs use 25% as the example).
Weekly report
- Cost per good record, p95 duration, the retry histogram and the trend in schema errors.
Sources: HN, Spinov, Spidermon monitors, Apify monitoring, Bright Data.
Treat those as a first draft. Spinov reports going from 3-4 fires a week to 0-1 after setting up his rules, but that’s one person’s self-report, and he also sells Apify actors.
Then grow the list from real incidents. Apify’s advice is “Every time something breaks, add a new alert”, which beats guessing a dozen thresholds on day one (Apify blog).
If you don’t run Scrapy or Apify, the two count rules fit in a few lines. This is my sketch of Spinov’s rules, assuming one run a day; I haven’t run it.
# A sketch of two starter rules from one practitioner's setup (dev.to, 2026). Not run.
def check_counts(today, previous):
"""today: items in this run. previous: item counts of earlier runs, newest first."""
alerts = []
if today == 0 and previous and previous[0] == 0:
alerts.append(("page", "zero items two runs in a row"))
week = previous[:7]
if week:
baseline = sum(week) / len(week)
if today < baseline / 2:
alerts.append(("digest", f"{today} items vs a 7-run average of {baseline:.0f}"))
return alerts
What does your stack already give you? #
Quite a lot, and much of it is free. Here’s what each option covers, and what it costs where the docs say.
| Your stack | Tool | What you get | Cost or licence |
|---|---|---|---|
| Scrapy | Spidermon | Close-of-run monitor suite, JSON Schema validation, alerts by email, Slack, Telegram or Discord | Open source, BSD-3-Clause |
| Scrapy, nothing added | Built-in stats and CloseSpider | Stats keys, no-item close settings, periodic JSON stats | Built in |
| Crawlee | Statistics |
Failed requests per minute, retry histogram, status codes, error tracker | Built in |
| Apify | Console monitoring | Run-status, lower-than, higher-than and field-statistics alerts | Free per Apify’s docs |
| Warehouse or Spark | GX Core | Expectations checked by Checkpoints, with email, Slack or Teams actions | Open source, Apache-2.0 |
| Scrapy or Python Requests | ScrapeOps Monitoring | Hosted dashboard, health checks, daily reports | Not shown on the page I read |
Verified 2026-11-17: Spidermon on GitHub, Scrapy extensions, Crawlee, Apify, GX on GitHub, ScrapeOps.
Scrapy with Spidermon #
Spidermon calls itself “a framework to build monitors for Scrapy spiders”, and it’s actively maintained: the newest release note is 1.27.0, dated August 10, 2026, and the docs footer credits Zyte (Spidermon docs). The repo is BSD-3-Clause with 564 stars (GitHub).
It ships more than a dozen built-in monitors. The SpiderCloseMonitorSuite, switched on through SPIDERMON_SPIDER_CLOSE_MONITORS, bundles eleven: ItemCount, ItemValidation, ErrorCount, WarningCount, FinishReason, UnwantedHTTPCodes, FieldCoverage, RetryCount, DownloaderException, SuccessfulRequests and TotalRequests (monitors).
The defaults tell you what you still have to decide. ItemCount (SPIDERMON_MIN_ITEMS) and ErrorCount (SPIDERMON_MAX_ERRORS) need a value from you or they don’t run; FinishReason expects ['finished']; RetryCount is off at -1.
UnwantedHTTPCodes watches [400, 407, 429, 500, 502, 503, 504, 523, 540, 541] and allows up to 10 of them per run. FieldCoverage reads its rules from SPIDERMON_FIELD_COVERAGE_RULES, and PeriodicItemCount checks growth during a run through SPIDERMON_ITEM_COUNT_INCREASE (monitors).
Item validation is “based on the jsonschema library” (Spidermon docs). You add ItemValidationPipeline as the last pipeline and point SPIDERMON_VALIDATION_SCHEMAS at local, S3 or HTTPS schema files; errors land in stats under spidermon/validation/fields/errors/* (item validation).
By default it neither tags bad items nor drops them: SPIDERMON_VALIDATION_ADD_ERRORS_TO_ITEMS and SPIDERMON_VALIDATION_DROP_ITEMS_WITH_ERRORS both start False. I’d turn on the first, so a _validation field travels with each bad record.
Alerts go to email, Slack, Telegram or Discord, and actions include Sentry, SNS, S3 or file reports, and job tags (Spidermon docs). Periodic suites run during the crawl via SPIDERMON_PERIODIC_MONITORS, timed in seconds. One monitor, ZyteJobsComparison, compares with past jobs on Zyte’s platform; the rest work anywhere Scrapy does.
Scrapy on its own #
Even without Spidermon, “The Stats Collector API is always available” (Scrapy stats). The keys you want are item_scraped_count, finish_reason, downloader/response_status_count/{code} and log_count/{level}.
The cheapest guard is CloseSpider. CLOSESPIDER_TIMEOUT_NO_ITEM closes the spider if no items arrive for a set number of seconds, and CLOSESPIDER_PAGECOUNT_NO_ITEM closes it after a set number of responses with no item; both default to 0, which means off (Scrapy extensions).
My inference from the two docs: a run stopped that way ends with a different finish_reason, so Spidermon’s FinishReason monitor fires on it. That turns an empty run from a quiet success into a loud failure.
Core Stats adds elapsed_time_seconds, item_dropped_count and response_received_count. Periodic Log (PERIODIC_LOG_STATS, PERIODIC_LOG_DELTA) writes stats as JSON during the run, which is handy if you ship logs to another system (Scrapy extensions).
Crawlee’s statistics #
Crawlee keeps the numbers in its Statistics class: requestsFailedPerMinute, requestsFinishedPerMinute, requestAvgFailedDurationMillis and a requestRetryHistogram, where “Index 0 means 0 retries” (Crawlee). Status codes are counted through registerStatusCode, and errorTracker collects the errors from final retries.
The Statistics page I read is about collecting numbers, not sending alerts, so read these values at the end of a run and apply the same rules as above. My Crawlee vs Scrapy comparison covers the frameworks more broadly.
Monitoring on Apify #
Apify’s docs are plain about it: “The monitoring system is free for all users.” It works for any Actor or task, with a 30-day status chart and a metrics chart over the last 200 runs; the same page suggests watching duration, results, fields, usage cost and proxy usage (Apify monitoring).
The alerts map neatly onto the tiers. “Lower than” checks after a run, “Higher than” checks about every five minutes and at the end, run-status alerts fire on the first failure and then aggregate every 20 minutes, and field-statistics alerts need a dataset schema. Apify calls Slack “more reliable” than email for delivery.
A separate monitoring suite adds schema and duplicate checks, but it runs on platform resources you pay for, and I found no price for it. For the platform’s costs overall, see my Apify pricing breakdown.
GX Core where the data lands #
If your scraped data ends up in a SQL table or a Spark DataFrame, validate it there. GX Core is a Python library where an Expectation is a “verifiable assertion about data”, and a Checkpoint is “the primary means for validating data in a production deployment of GX” (GX overview).
Checkpoints trigger Actions, which can send email, Slack or Microsoft Teams messages, and Data Docs give you a readable report. The docs I read are version 1.25.0 (GX introduction), and the project is Apache-2.0 with 11.9k stars (GitHub).
This is the right layer when several scrapers feed one table. One Checkpoint on the table flags a broken field, if an Expectation covers it, no matter which scraper caused it.
ScrapeOps for a hosted dashboard #
ScrapeOps Monitoring shows job stats, pages and items scraped and missed, field coverage, status codes, success rates, latency and bandwidth, with Slack health checks, daily reports and comparison against past runs (ScrapeOps).
Its SDKs cover Scrapy and Python Requests, with Node.js and Go listed as on the roadmap. The page I read shows no monitoring price, so check that before you plan around it.
What the scraping APIs offer #
Less than you’d hope, and none of it replaces your own checks. Bright Data’s Scrapers Library sends per-scraper success-rate emails on a drop and on recovery, with the threshold set in its Control Panel, but “there is no API for them”, and by default they fire “only when the scraper is in outage” (Bright Data). To watch it in code, you poll GET /datasets/v3/progress/{snapshot_id}.
Scrapfly’s Crawler Monitoring API offers aggregated and per-domain metrics, but plans below Enterprise get HTTP 402 (Scrapfly). Firecrawl’s monitors are something else: they detect changes on pages, not the health of your scraper, and I cover them under canaries below.
Why your API’s success rate isn’t your success rate #
Because “success” there means “billable”, and a billable response can still give you no data. The two billing pages I read show it clearly.
| ScraperAPI | Zyte API | |
|---|---|---|
| What’s charged | “successful requests (200 and 404 status codes)” | “only charged for successful responses” |
| A target page’s 404 | Charged | Returned as a successful response with statusCode 404 |
| Ban errors | Not covered on the page I read | 520 (“usually a ban”) and 521 are unsuccessful and free |
| Requests you cancel | Charged if cancelled before 70 seconds | Not covered on the page I read |
Billing rules verified 2026-11-17: ScraperAPI credits, Zyte pricing, Zyte errors.
Zyte also says “Rate-limiting and unsuccessful responses are free.” So by my reading of those two pages, Zyte’s dashboard would count that target 404 as a success you paid for, while a ban costs nothing. Either way, the API’s success rate and yours are different numbers.
So record the target’s status inside the payload, not just the API’s response code, and count records that pass validation. Then divide.
A worked example (hypothetical counts, ScraperAPI’s documented weights): 10,000 normal requests at 1 credit each cost 10,000 credits. If 1,500 come back as billed 404s and 500 parse into empty items, you have 8,000 good records, so you paid 1.25 credits per good record, not 1.0.
That 25% markup stays invisible if you only watch request counts. Credit weights make every wasted request dearer on harder targets: ScraperAPI charges 5 credits for e-commerce pages, 10 for bot-protected sites and 25 for SERP pages, and lists render=true as an extra cost of 10 credits per request (ScraperAPI).
For how the weights work across vendors, see what scraping API credits really buy, and the per-vendor detail in Zyte API pricing and ScraperAPI pricing. Once cost per good record is on your weekly report, how to reduce web scraping costs is the next read.
Canaries and heartbeats catch what counts miss #
Counts tell you something changed; a canary tells you whether the values are still right. Pick a few pages whose correct values you know, scrape them on a schedule, and check that the values still match, which is the approach Shifter recommends alongside per-target baselines (Shifter).
If you want practice pages built for this, my guide on how to test a scraping API lists them. When a canary’s fields suddenly go empty after a site redesign, the page may have moved to client-side rendering, and the JavaScript rendering check helps you tell.
A heartbeat covers the opposite failure: nothing ran at all. Each run writes a small record (finished, item count, time), and a separate check alerts when that record is missing or stale, because a monitoring system “has to treat absence of data as a signal”.
Firecrawl’s monitors can watch a target page for content changes. They run on a cron or plain-text schedule no more often than every 5 minutes, label results same, new, changed, removed or error, and send webhooks such as monitor.page (Firecrawl).
There’s no separate per-monitor fee; you pay the underlying scrape credits. A changed result on a page you scrape is an early warning that your selectors may be next.

Which setup should you pick? #
Go by where your scrapers run and where the data ends up.
- Scrapy project: Spidermon, with the spider-close suite, a JSON Schema per item type and Slack alerts. Add
CLOSESPIDER_TIMEOUT_NO_ITEMso an empty run ends with a failure you’ll see. - Running on Apify: Apify’s built-in monitoring, because it’s free and already wired to your runs. Define a dataset schema so field-statistics alerts work.
- Data lands in a warehouse or Spark: GX Core Checkpoints on the output table, alerting to Slack or Teams. One check covers every scraper that writes there.
- Small mixed fleet, any language: a heartbeat, a record-count baseline and a schema check. Spinov’s files-SQLite-and-email setup shows that’s enough at 77 scrapers, by his own account.
- You want a hosted dashboard for Scrapy or Requests: ScrapeOps Monitoring, once you’ve confirmed its price.
Whatever you pick, add cost per good record yourself. No tool on this list computes it from your vendor’s billing rules.
What I would do #
I’d start with three checks on every scraper before adding any dashboard: did it run (heartbeat), did it return roughly as many records as last week, and are the key fields filled. On Scrapy that’s Spidermon plus the no-item close setting; elsewhere a small script does it.
Then I’d compute cost per good record from the vendor’s billing rules. A 404 that’s billed and an empty item that passed are the two failures no vendor dashboard shows you.
That’s a plan built from the docs, not from my own runs. My guides to web scraping APIs pick up from here if you’re still choosing the API underneath.
FAQ
What is a silent failure in web scraping?
A run that finishes without errors but ships empty, partial or wrong data. Typical causes are a layout change that empties a field, a block page that comes back as a normal response, or a listing that quietly returns fewer items.
What's the minimum monitoring a scraper needs?
Three checks: a heartbeat that proves the run happened, a record count compared with recent runs, and the fill rate of your key fields. Add schema validation next.
Is Spidermon free, and does it work without Zyte?
Yes. It's an open-source Scrapy extension under the BSD-3-Clause licence, and it runs wherever Scrapy runs. One monitor, ZyteJobsComparison, compares against past jobs on Zyte's platform; the rest don't need Zyte.
How do I monitor a scraper that isn't built on Scrapy?
On Crawlee, read its Statistics class (failed requests per minute, retry histogram, error tracker). On Apify, turn on the free Console monitoring. If the data lands in SQL or Spark, validate it there with GX Core. For anything else, a small script that writes a heartbeat and compares counts is enough.
Why does my scraping API report success when my data is wrong?
Because its success is about billing, not your data. ScraperAPI charges for 200 and 404 responses, and Zyte API returns a target site's 404 as a successful response with statusCode 404. Check the status inside the payload and validate the parsed record.
How do I avoid alert fatigue?
Tier your alerts. Page someone only when data stops arriving or drops sharply, send softer warnings to a daily digest, and keep trends like cost per good record for a weekly report. Add a new alert after each real incident instead of guessing up front.
Does Apify charge for monitoring?
Apify's docs say its built-in monitoring system is free for all users. The separate monitoring suite, which adds schema and duplicate checks, runs on platform resources that you pay for.
Sources
25 sources, accessed .
Official pricing, docs, changelogs and status pages 16
- Spidermon docs: overview and release notes (1.27.0, August 10, 2026)
- Spidermon docs: built-in monitors and their settings
- Spidermon docs: item validation with JSON Schema
- Scrapy docs: Stats Collection
- Scrapy docs: extensions (CloseSpider, Core Stats, Periodic Log)
- Crawlee API: Statistics class
- Apify docs: monitoring and alerts
- GX Core docs: overview (Expectations, Checkpoints, Actions)
- GX Core docs: introduction (version 1.25.0)
- ScraperAPI docs: credits and request costs
- Zyte API docs: pricing
- Zyte API docs: errors (target 404, 520, 521)
- Bright Data docs: Scrapers Library success-rate alerts
- ScrapeOps: Monitoring product page
- Firecrawl docs: monitoring (page change checks)
- Scrapfly docs: Crawler Monitoring API
GitHub code, issues and releases 2
Community discussions 3
Other blogs 4
- Zyte knowledge base: monitoring web scrapers with Spidermon (June 19, 2023)
- Apify blog: why you need to monitor long-running scraping projects (April 6, 2023)
- Shifter blog: how to monitor a web scraping pipeline (updated August 4, 2026)
- PromptCloud blog: web scraping monitoring challenges (February 18, 2026)


