Skip to content
CrawlfolioSourced research on web data for AI
Menu

To monitor a web scraper, check the data, not just the job. A scraper can finish green and still ship empty, partial or wrong records, so start with three checks: a heartbeat that proves it ran, a record count compared with last week’s runs, and the fill rate of your key fields.

I read the Spidermon, Scrapy, Crawlee, Apify and Great Expectations docs, the billing rules of two scraping APIs, and the pages that rank for this topic. Below is what to measure, which alerts should wake someone up, and the tool I’d pick for each stack.

Short on time?

Short on time? Here's my take.

Scrapy project
Spidermon
Turn on the spider-close suite, add a JSON Schema per item type and send alerts to Slack. Add CLOSESPIDER_TIMEOUT_NO_ITEM so empty runs end loudly.
Running on Apify
Apify's built-in monitoring
Free per Apify's docs. Give the dataset a schema so the field-statistics alerts work.
Data lands in a warehouse or Spark
GX Core Checkpoints
Validate the output table on every load and send failures to Slack or Teams.
Small mixed fleet, any language
Heartbeat, record-count baseline and schema check
Files, SQLite and email are enough, per one practitioner running 77 scrapers.
You want a hosted dashboard
ScrapeOps Monitoring
SDKs for Scrapy and Python Requests. Check the price first; I couldn't verify it.

Prices verified . Sponsors cannot buy a verdict or a position.

Why does a scraper fail without telling you? #

Because the scheduler judges the job, and nobody judges the data. The process starts, fetches pages, writes a file and exits with code zero, while the price field has been empty since the site moved it into a new element.

Shifter’s monitoring guide puts it in one line: “A scraper fails silently: the cron fires, the logs say 200, the data goes wrong.” Apify’s 2023 post describes the same thing from the other end: “Your scraper might be working fine for weeks, then suddenly, it breaks, yielding zero results.”

It doesn’t take much. A Hacker News commenter noted back in 2013 that “cosmetic changes break scrapers silently”. The pages I read describe the same three shapes: a layout change empties a field, a block page comes back as a normal response, or a listing quietly returns fewer items.

PromptCloud draws the line I find most useful: “Monitoring answers whether the job ran. Observability answers whether the data still reflects reality.” It also claims infrastructure failures get caught “within minutes” while missing fields surface days later, but it shows no method, so read that as their anecdote.

The Zyte guide that ranks first here describes monitors as “additional code segments integrated into your web scraping project alongside the scraper’s code” (Zyte KB). That’s the right model: the checks live next to the scraper and run every time it does. What none of the ranking pages give you is the actual settings, so that’s what the rest of this page is.

One boundary. Blocks appear here only as a metric: if your status codes fill with 403s and 429s, the answer is to slow down or stop, which my rate limit guide covers.

What should you actually measure? #

Two layers. Job health tells you the scraper ran and ended normally; data health tells you that what it produced is complete and correct. Every ranking page says teams miss the second layer, and none shows where the numbers come from.

Metric What it catches Where you get it
Heartbeat / last successful run The job didn’t run or didn’t finish Your own heartbeat record; Apify run-status alerts
Finish reason A run that ended on a timeout, error or close condition Scrapy’s finish_reason stat; Spidermon FinishReason (expects finished)
Items vs recent runs Layout changes, block pages, shrinking listings Scrapy item_scraped_count; Spidermon ItemCount, PeriodicItemCount; Apify “Lower than” alert
Field coverage One selector broke while the rest still work Spidermon FieldCoverage; Apify dataset field statistics; ScrapeOps
Schema errors Wrong types, malformed values Spidermon item validation (JSON Schema); GX Expectations
Status-code mix Blocks, rate limits, server errors Scrapy downloader/response_status_count/{code}; Spidermon UnwantedHTTPCodes; Crawlee registerStatusCode
Retries A target getting slower or stricter Crawlee requestRetryHistogram; Spidermon RetryCount
Errors Exceptions in parsing or downloading Scrapy log_count/ERROR; Spidermon ErrorCount; Crawlee errorTracker
Duration Hangs and slowdowns Scrapy elapsed_time_seconds; Spidermon PeriodicExecutionTime; Apify “Higher than” alert
Freshness Data that stopped updating The timestamp of the newest good record in your store
Cost per good record Paying for pages that produced nothing Your own sum: credits spent ÷ records that pass validation

Features verified 2026-11-17: Scrapy stats, Scrapy extensions, Spidermon monitors, Crawlee Statistics, Apify monitoring, GX Core, ScrapeOps.

If you only add two data checks, make them item count and field coverage. A block page or a layout change can leave the job layer looking perfectly normal, and these two are where it shows.

Compare each target against its own history, not one global number. Shifter argues that “absolute thresholds either miss real problems or cry wolf constantly”, and recommends a baseline per target plus p50, p95 and p99 latency (Shifter). Its example figures are illustrations, not measurements.

Schema checks apply just as much when a model does the parsing. If you use an LLM for structured data extraction, validate its output against the same JSON Schema you’d use for a CSS-selector scraper.

Wide view of a massive grey concrete bunker with stacked, overhanging slabs and dark slit openings under a pale sky
From the outside, a sealed concrete shell looks the same whether its rooms are full or empty. A scraper's job stats work the same way, which is why item count and field coverage, not a clean exit, are where a block page or a layout change shows up. Photo: Michael Jasmund / Unsplash · Unsplash License

Which alerts should wake someone up? #

Very few. Alex Spinov, who describes monitoring 77 scrapers with about 200 lines of Python, a JSON heartbeat per run and counts in SQLite, puts it bluntly: “The biggest mistake is alerting on everything.”

Here are the starter rules I’d use, sorted into three tiers. The tiers are my house rule; each number is someone else’s, named next to it.

Starter alert rules (start here, then tune per target)

Page someone now

  • No heartbeat: the run didn’t happen or didn’t finish. A May 2026 HN comment says a monitoring system “has to treat absence of data as a signal”.
  • Zero items two runs in a row (Spinov’s critical rule).
  • The run failed, or its finish reason isn’t finished (Spidermon FinishReason default; Apify run-status alert).

Daily digest

  • Item count below half the trailing 7-day average (Spinov’s flag).
  • A key field’s coverage falls below your rule (Spidermon FieldCoverage, tolerance 0 by default).
  • More than 10 unwanted HTTP codes in a run (Spidermon’s default maximum).
  • A Bright Data scraper’s success rate crosses your threshold (its docs use 25% as the example).

Weekly report

  • Cost per good record, p95 duration, the retry histogram and the trend in schema errors.

Sources: HN, Spinov, Spidermon monitors, Apify monitoring, Bright Data.

Treat those as a first draft. Spinov reports going from 3-4 fires a week to 0-1 after setting up his rules, but that’s one person’s self-report, and he also sells Apify actors.

Then grow the list from real incidents. Apify’s advice is “Every time something breaks, add a new alert”, which beats guessing a dozen thresholds on day one (Apify blog).

If you don’t run Scrapy or Apify, the two count rules fit in a few lines. This is my sketch of Spinov’s rules, assuming one run a day; I haven’t run it.

# A sketch of two starter rules from one practitioner's setup (dev.to, 2026). Not run.
def check_counts(today, previous):
    """today: items in this run. previous: item counts of earlier runs, newest first."""
    alerts = []
    if today == 0 and previous and previous[0] == 0:
        alerts.append(("page", "zero items two runs in a row"))
    week = previous[:7]
    if week:
        baseline = sum(week) / len(week)
        if today < baseline / 2:
            alerts.append(("digest", f"{today} items vs a 7-run average of {baseline:.0f}"))
    return alerts

What does your stack already give you? #

Quite a lot, and much of it is free. Here’s what each option covers, and what it costs where the docs say.

Your stack Tool What you get Cost or licence
Scrapy Spidermon Close-of-run monitor suite, JSON Schema validation, alerts by email, Slack, Telegram or Discord Open source, BSD-3-Clause
Scrapy, nothing added Built-in stats and CloseSpider Stats keys, no-item close settings, periodic JSON stats Built in
Crawlee Statistics Failed requests per minute, retry histogram, status codes, error tracker Built in
Apify Console monitoring Run-status, lower-than, higher-than and field-statistics alerts Free per Apify’s docs
Warehouse or Spark GX Core Expectations checked by Checkpoints, with email, Slack or Teams actions Open source, Apache-2.0
Scrapy or Python Requests ScrapeOps Monitoring Hosted dashboard, health checks, daily reports Not shown on the page I read

Verified 2026-11-17: Spidermon on GitHub, Scrapy extensions, Crawlee, Apify, GX on GitHub, ScrapeOps.

Scrapy with Spidermon #

Spidermon calls itself “a framework to build monitors for Scrapy spiders”, and it’s actively maintained: the newest release note is 1.27.0, dated August 10, 2026, and the docs footer credits Zyte (Spidermon docs). The repo is BSD-3-Clause with 564 stars (GitHub).

It ships more than a dozen built-in monitors. The SpiderCloseMonitorSuite, switched on through SPIDERMON_SPIDER_CLOSE_MONITORS, bundles eleven: ItemCount, ItemValidation, ErrorCount, WarningCount, FinishReason, UnwantedHTTPCodes, FieldCoverage, RetryCount, DownloaderException, SuccessfulRequests and TotalRequests (monitors).

The defaults tell you what you still have to decide. ItemCount (SPIDERMON_MIN_ITEMS) and ErrorCount (SPIDERMON_MAX_ERRORS) need a value from you or they don’t run; FinishReason expects ['finished']; RetryCount is off at -1.

UnwantedHTTPCodes watches [400, 407, 429, 500, 502, 503, 504, 523, 540, 541] and allows up to 10 of them per run. FieldCoverage reads its rules from SPIDERMON_FIELD_COVERAGE_RULES, and PeriodicItemCount checks growth during a run through SPIDERMON_ITEM_COUNT_INCREASE (monitors).

Item validation is “based on the jsonschema library” (Spidermon docs). You add ItemValidationPipeline as the last pipeline and point SPIDERMON_VALIDATION_SCHEMAS at local, S3 or HTTPS schema files; errors land in stats under spidermon/validation/fields/errors/* (item validation).

By default it neither tags bad items nor drops them: SPIDERMON_VALIDATION_ADD_ERRORS_TO_ITEMS and SPIDERMON_VALIDATION_DROP_ITEMS_WITH_ERRORS both start False. I’d turn on the first, so a _validation field travels with each bad record.

Alerts go to email, Slack, Telegram or Discord, and actions include Sentry, SNS, S3 or file reports, and job tags (Spidermon docs). Periodic suites run during the crawl via SPIDERMON_PERIODIC_MONITORS, timed in seconds. One monitor, ZyteJobsComparison, compares with past jobs on Zyte’s platform; the rest work anywhere Scrapy does.

Scrapy on its own #

Even without Spidermon, “The Stats Collector API is always available” (Scrapy stats). The keys you want are item_scraped_count, finish_reason, downloader/response_status_count/{code} and log_count/{level}.

The cheapest guard is CloseSpider. CLOSESPIDER_TIMEOUT_NO_ITEM closes the spider if no items arrive for a set number of seconds, and CLOSESPIDER_PAGECOUNT_NO_ITEM closes it after a set number of responses with no item; both default to 0, which means off (Scrapy extensions).

My inference from the two docs: a run stopped that way ends with a different finish_reason, so Spidermon’s FinishReason monitor fires on it. That turns an empty run from a quiet success into a loud failure.

Core Stats adds elapsed_time_seconds, item_dropped_count and response_received_count. Periodic Log (PERIODIC_LOG_STATS, PERIODIC_LOG_DELTA) writes stats as JSON during the run, which is handy if you ship logs to another system (Scrapy extensions).

Crawlee’s statistics #

Crawlee keeps the numbers in its Statistics class: requestsFailedPerMinute, requestsFinishedPerMinute, requestAvgFailedDurationMillis and a requestRetryHistogram, where “Index 0 means 0 retries” (Crawlee). Status codes are counted through registerStatusCode, and errorTracker collects the errors from final retries.

The Statistics page I read is about collecting numbers, not sending alerts, so read these values at the end of a run and apply the same rules as above. My Crawlee vs Scrapy comparison covers the frameworks more broadly.

Monitoring on Apify #

Apify’s docs are plain about it: “The monitoring system is free for all users.” It works for any Actor or task, with a 30-day status chart and a metrics chart over the last 200 runs; the same page suggests watching duration, results, fields, usage cost and proxy usage (Apify monitoring).

The alerts map neatly onto the tiers. “Lower than” checks after a run, “Higher than” checks about every five minutes and at the end, run-status alerts fire on the first failure and then aggregate every 20 minutes, and field-statistics alerts need a dataset schema. Apify calls Slack “more reliable” than email for delivery.

A separate monitoring suite adds schema and duplicate checks, but it runs on platform resources you pay for, and I found no price for it. For the platform’s costs overall, see my Apify pricing breakdown.

GX Core where the data lands #

If your scraped data ends up in a SQL table or a Spark DataFrame, validate it there. GX Core is a Python library where an Expectation is a “verifiable assertion about data”, and a Checkpoint is “the primary means for validating data in a production deployment of GX” (GX overview).

Checkpoints trigger Actions, which can send email, Slack or Microsoft Teams messages, and Data Docs give you a readable report. The docs I read are version 1.25.0 (GX introduction), and the project is Apache-2.0 with 11.9k stars (GitHub).

This is the right layer when several scrapers feed one table. One Checkpoint on the table flags a broken field, if an Expectation covers it, no matter which scraper caused it.

ScrapeOps for a hosted dashboard #

ScrapeOps Monitoring shows job stats, pages and items scraped and missed, field coverage, status codes, success rates, latency and bandwidth, with Slack health checks, daily reports and comparison against past runs (ScrapeOps).

Its SDKs cover Scrapy and Python Requests, with Node.js and Go listed as on the roadmap. The page I read shows no monitoring price, so check that before you plan around it.

What the scraping APIs offer #

Less than you’d hope, and none of it replaces your own checks. Bright Data’s Scrapers Library sends per-scraper success-rate emails on a drop and on recovery, with the threshold set in its Control Panel, but “there is no API for them”, and by default they fire “only when the scraper is in outage” (Bright Data). To watch it in code, you poll GET /datasets/v3/progress/{snapshot_id}.

Scrapfly’s Crawler Monitoring API offers aggregated and per-domain metrics, but plans below Enterprise get HTTP 402 (Scrapfly). Firecrawl’s monitors are something else: they detect changes on pages, not the health of your scraper, and I cover them under canaries below.

Why your API’s success rate isn’t your success rate #

Because “success” there means “billable”, and a billable response can still give you no data. The two billing pages I read show it clearly.

ScraperAPI Zyte API
What’s charged “successful requests (200 and 404 status codes)” “only charged for successful responses”
A target page’s 404 Charged Returned as a successful response with statusCode 404
Ban errors Not covered on the page I read 520 (“usually a ban”) and 521 are unsuccessful and free
Requests you cancel Charged if cancelled before 70 seconds Not covered on the page I read

Billing rules verified 2026-11-17: ScraperAPI credits, Zyte pricing, Zyte errors.

Zyte also says “Rate-limiting and unsuccessful responses are free.” So by my reading of those two pages, Zyte’s dashboard would count that target 404 as a success you paid for, while a ban costs nothing. Either way, the API’s success rate and yours are different numbers.

So record the target’s status inside the payload, not just the API’s response code, and count records that pass validation. Then divide.

A worked example (hypothetical counts, ScraperAPI’s documented weights): 10,000 normal requests at 1 credit each cost 10,000 credits. If 1,500 come back as billed 404s and 500 parse into empty items, you have 8,000 good records, so you paid 1.25 credits per good record, not 1.0.

That 25% markup stays invisible if you only watch request counts. Credit weights make every wasted request dearer on harder targets: ScraperAPI charges 5 credits for e-commerce pages, 10 for bot-protected sites and 25 for SERP pages, and lists render=true as an extra cost of 10 credits per request (ScraperAPI).

For how the weights work across vendors, see what scraping API credits really buy, and the per-vendor detail in Zyte API pricing and ScraperAPI pricing. Once cost per good record is on your weekly report, how to reduce web scraping costs is the next read.

Canaries and heartbeats catch what counts miss #

Counts tell you something changed; a canary tells you whether the values are still right. Pick a few pages whose correct values you know, scrape them on a schedule, and check that the values still match, which is the approach Shifter recommends alongside per-target baselines (Shifter).

If you want practice pages built for this, my guide on how to test a scraping API lists them. When a canary’s fields suddenly go empty after a site redesign, the page may have moved to client-side rendering, and the JavaScript rendering check helps you tell.

A heartbeat covers the opposite failure: nothing ran at all. Each run writes a small record (finished, item count, time), and a separate check alerts when that record is missing or stale, because a monitoring system “has to treat absence of data as a signal”.

Firecrawl’s monitors can watch a target page for content changes. They run on a cron or plain-text schedule no more often than every 5 minutes, label results same, new, changed, removed or error, and send webhooks such as monitor.page (Firecrawl).

There’s no separate per-monitor fee; you pay the underlying scrape credits. A changed result on a page you scrape is an early warning that your selectors may be next.

Fine streaks of soft grey light sweeping diagonally across a dark background, fading out at both edges
A heartbeat is a faint trace each run should leave behind. The alert watches for its absence: when the newest record is missing or stale, nothing ran, even though no error was ever logged. Photo: MARIOLA GROBELSKA / Unsplash · Unsplash License

Which setup should you pick? #

Go by where your scrapers run and where the data ends up.

  • Scrapy project: Spidermon, with the spider-close suite, a JSON Schema per item type and Slack alerts. Add CLOSESPIDER_TIMEOUT_NO_ITEM so an empty run ends with a failure you’ll see.
  • Running on Apify: Apify’s built-in monitoring, because it’s free and already wired to your runs. Define a dataset schema so field-statistics alerts work.
  • Data lands in a warehouse or Spark: GX Core Checkpoints on the output table, alerting to Slack or Teams. One check covers every scraper that writes there.
  • Small mixed fleet, any language: a heartbeat, a record-count baseline and a schema check. Spinov’s files-SQLite-and-email setup shows that’s enough at 77 scrapers, by his own account.
  • You want a hosted dashboard for Scrapy or Requests: ScrapeOps Monitoring, once you’ve confirmed its price.

Whatever you pick, add cost per good record yourself. No tool on this list computes it from your vendor’s billing rules.

What I would do #

I’d start with three checks on every scraper before adding any dashboard: did it run (heartbeat), did it return roughly as many records as last week, and are the key fields filled. On Scrapy that’s Spidermon plus the no-item close setting; elsewhere a small script does it.

Then I’d compute cost per good record from the vendor’s billing rules. A 404 that’s billed and an empty item that passed are the two failures no vendor dashboard shows you.

That’s a plan built from the docs, not from my own runs. My guides to web scraping APIs pick up from here if you’re still choosing the API underneath.

FAQ

What is a silent failure in web scraping?

A run that finishes without errors but ships empty, partial or wrong data. Typical causes are a layout change that empties a field, a block page that comes back as a normal response, or a listing that quietly returns fewer items.

What's the minimum monitoring a scraper needs?

Three checks: a heartbeat that proves the run happened, a record count compared with recent runs, and the fill rate of your key fields. Add schema validation next.

Is Spidermon free, and does it work without Zyte?

Yes. It's an open-source Scrapy extension under the BSD-3-Clause licence, and it runs wherever Scrapy runs. One monitor, ZyteJobsComparison, compares against past jobs on Zyte's platform; the rest don't need Zyte.

How do I monitor a scraper that isn't built on Scrapy?

On Crawlee, read its Statistics class (failed requests per minute, retry histogram, error tracker). On Apify, turn on the free Console monitoring. If the data lands in SQL or Spark, validate it there with GX Core. For anything else, a small script that writes a heartbeat and compares counts is enough.

Why does my scraping API report success when my data is wrong?

Because its success is about billing, not your data. ScraperAPI charges for 200 and 404 responses, and Zyte API returns a target site's 404 as a successful response with statusCode 404. Check the status inside the payload and validate the parsed record.

How do I avoid alert fatigue?

Tier your alerts. Page someone only when data stops arriving or drops sharply, send softer warnings to a daily digest, and keep trends like cost per good record for a weekly report. Add a new alert after each real incident instead of guessing up front.

Does Apify charge for monitoring?

Apify's docs say its built-in monitoring system is free for all users. The separate monitoring suite, which adds schema and duplicate checks, runs on platform resources that you pay for.

Sources

25 sources, accessed .

Official pricing, docs, changelogs and status pages 16

  1. Spidermon docs: overview and release notes (1.27.0, August 10, 2026) spidermon.readthedocs.io · accessed
  2. Spidermon docs: built-in monitors and their settings spidermon.readthedocs.io · accessed
  3. Spidermon docs: item validation with JSON Schema spidermon.readthedocs.io · accessed
  4. Scrapy docs: Stats Collection docs.scrapy.org · accessed
  5. Scrapy docs: extensions (CloseSpider, Core Stats, Periodic Log) docs.scrapy.org · accessed
  6. Crawlee API: Statistics class crawlee.dev · accessed
  7. Apify docs: monitoring and alerts docs.apify.com · accessed
  8. GX Core docs: overview (Expectations, Checkpoints, Actions) docs.greatexpectations.io · accessed
  9. GX Core docs: introduction (version 1.25.0) docs.greatexpectations.io · accessed
  10. ScraperAPI docs: credits and request costs docs.scraperapi.com · accessed
  11. Zyte API docs: pricing docs.zyte.com · accessed
  12. Zyte API docs: errors (target 404, 520, 521) docs.zyte.com · accessed
  13. Bright Data docs: Scrapers Library success-rate alerts docs.brightdata.com · accessed
  14. ScrapeOps: Monitoring product page scrapeops.io · accessed
  15. Firecrawl docs: monitoring (page change checks) docs.firecrawl.dev · accessed
  16. Scrapfly docs: Crawler Monitoring API scrapfly.io · accessed

GitHub code, issues and releases 2

  1. GitHub: scrapinghub/spidermon (BSD-3-Clause) github.com · accessed
  2. GitHub: great-expectations/great_expectations (Apache-2.0) github.com · accessed

Community discussions 3

  1. Alex Spinov on dev.to: how I monitor 77 web scrapers (March 2026) dev.to · accessed
  2. Hacker News comment: treat absence of data as a signal (May 13, 2026) news.ycombinator.com · accessed
  3. Hacker News comment: cosmetic changes break scrapers silently (December 4, 2013) news.ycombinator.com · accessed

What readers say

Used one of the tools in this guide? Tell other readers how it went. I read every review before it goes up, and I never turn one down for being negative. How reviews work.

No reader reviews yet. Yours could be the first.

Write a review
Your rating

What did you use it for, what worked, and what didn't? 30 to 1500 characters. Please don't include links.

Your connection to the company

Your review appears with the name you enter, after I approve it. I don't ask for your email. Privacy.