Skip to content
CrawlfolioSourced research on web data for AI
Menu

If your scraping bill is too high, cut the number of requests before you cut the price per request. Removing duplicate URLs, refetching only pages that changed and rendering only where needed can shrink a bill far more than a cheaper vendor.

This guide gives you eight fixes in the order I’d make them, each tied to a standard or a vendor’s own docs. A worked bill shows which one moves the number most, and the last section tells you when switching really is the answer.

Short on time?

Short on time? Here's my take.

Recrawling the same sites on a schedule
Refetch only what changed
Drive recrawls from accurate sitemap lastmod and conditional requests. In my worked bill this removes the most fetches.
Paying for JavaScript rendering on every page
Render per template, not per site
ScraperAPI charges 10 credits for a rendered request and 1 for a plain one; ScrapingBee renders by default at 5.
Paying per GB for browser traffic
Block images and unneeded resources
Playwright's route.abort drops them before they download. This saves bytes, not per-request credits.
Still too expensive after the checklist
Switch vendor
Measure cost per validated record first, then compare vendors on that number.

Prices verified . Sponsors cannot buy a verdict or a position.

Where does the money actually go? #

On a scraping API, you pay for requests times the credit weight of each one. On a per-GB proxy, you pay for bytes. Every fix below cuts one of those two numbers.

Credit weights vary a lot by request type. On ScraperAPI a normal request costs 1 credit and a rendered one 10, and ZenRows charges 1 for a standard request and 5 with JavaScript. ScraperAPI, ZenRows. The full multiplier tables live in my guide to how scraping API credits work.

Proxy plans bill bandwidth instead. Decodo’s residential pricing is “bandwidth-based”, at $2.75 per GB on its 100 GB plan, and IPRoyal charges $5.25 per GB on its 10 GB subscription. Decodo, IPRoyal. Prices verified 2026-11-17.

That split decides which fixes matter to you. Fewer requests help on both kinds of plan; fewer bytes per request only help when you pay per GB. My guide to how proxies bill covers the per-GB side, and the web scraping cost calculator estimates a monthly bill at your volume from dated prices.

The checklist, in the order I’d work through it #

Fewest requests first, then fewest credits per request, then fewest bytes, then ceilings. Switching vendor comes last.

Eight fixes, in order

  1. Drop duplicate URLs before they’re queued. Canonicalize URLs and let the framework’s dupe filter do its job. Every duplicate you drop is a request you don’t pay for.
  2. Refetch only pages that changed. Use sitemap lastmod where it’s accurate, and change detection where your vendor offers it.
  3. Make refetches conditional. Send If-None-Match or If-Modified-Since and accept a 304 with no body, on direct fetches and per-GB proxies.
  4. Render JavaScript only where needed. Decide per page template, not per site.
  5. Cut bytes in browser sessions. Block images and resources you don’t need. This matters on per-GB plans.
  6. Bound retries. Retry a small, fixed number of times, and only on statuses worth retrying.
  7. Put a ceiling on spend. Use per-request cost caps and plan limits where your vendor offers them.
  8. Then decide whether to switch. Compare vendors on cost per validated record, not list price.

Built from RFC 9110, Scrapy, Google’s sitemap guide, Playwright and ScraperAPI’s cost control. The order is my recommendation.

Why this order? The first three remove whole requests, which saves money on every pricing model. Rendering and bytes come next because they only cut the cost of requests you still send.

The top three ranking guides miss most of the first three. Bright Data’s docs focus on IP types, browsers and bandwidth, ScrapeOps on compute, proxies and bandwidth, and Oxylabs’ case study reports a cut for an anonymous client with no baseline, so I don’t use its figure.

Are you paying twice for the same page? #

Possibly, if the same page reaches your queue under several URLs. Tracking parameters, reordered query strings and trailing slashes all make one page look like many.

Scrapy handles this by default. Its standard dupe filter, RFPDupeFilter, “filters out requests with the canonical (w3lib.url.canonicalize_url()) url, method and body”. Scrapy settings. Keep it on, and canonicalize any URLs you add from outside the crawl before they’re queued.

Crawlee’s request queue works the same way: “The queue can only contain unique URLs.” It adds that “By default, uniqueKey is generated from the URL, but it can also be overridden”, and a duplicate “will not be updated”. Crawlee.

Override the uniqueKey when two URLs are really one page, such as the same product with different tracking tags. That’s a one-line change, and each duplicate it catches is one fewer request on the bill.

Why pay again for pages that stayed the same? #

You don’t have to, if the site tells you what changed. A sitemap’s lastmod date is the cheapest change feed there is, because one file covers thousands of pages.

The sitemaps.org protocol defines lastmod as “The date of last modification of the page”, and says “the date must be set to the date the linked page was last modified, not when the sitemap is generated”. changefreq is only “a hint and not a command”. sitemaps.org.

Not every site keeps it honest, and Google treats it accordingly. “Google uses the <lastmod> value if it’s consistently and verifiably (for example by comparing to the last modification of the page) accurate”, and “Google ignores <priority> and <changefreq> values”. A single sitemap is limited to “50MB (uncompressed) or 50,000 URLs”. Google.

So copy Google’s caution. Spot-check a handful of pages against their lastmod before you trust it, and fall back to a full recrawl on a slower schedule for sites where it’s wrong. I found no independent data on how many sites keep it accurate.

Change tracking on Firecrawl #

If you’re on Firecrawl, its change tracking “compares the current content of a page against the last time you scraped it” and labels each page new, same, changed or removed. It needs the markdown format, and the git-diff mode adds no extra cost, while “JSON mode uses LLM extraction and costs 5 credits per page”. Firecrawl.

As I read it, the page is still scraped, so this saves downstream work, such as re-embedding or re-extracting unchanged pages, rather than the fetch itself. Pair it with lastmod if you want to skip fetches too. The Firecrawl pricing breakdown covers its credit costs.

What does a conditional request save? #

The whole body of any page that hasn’t changed. You send the validator you stored last time, and if nothing changed the server answers 304 with no content.

RFC 9110, the HTTP standard, defines it. A 304 means the request “would have resulted in a 200 (OK) response if it were not for the fact that the condition evaluated to false”, so “there is no need for the server to transfer a representation”. A 304 response “cannot contain content or trailers”. RFC 9110.

Send If-None-Match with the stored ETag where you have one. The RFC says “A recipient MUST ignore If-Modified-Since if the request contains an If-None-Match header field”, so the ETag wins when you send both. RFC 9110.

GET /product/123 HTTP/1.1
Host: example.com
If-None-Match: "abc123"
If-Modified-Since: Mon, 16 Nov 2026 08:00:00 GMT

The ETag and date are placeholders; use the values the server sent with the last full response.

This is where one ranking guide goes wrong. ScrapeOps calls a Last-Modified check “conditional requests”, and says “Retrieving the headers first will consume a lot less bandwidth than retrieving the entire page”. But its example sends a HEAD request and prints the header; it never sends If-None-Match or If-Modified-Since. ScrapeOps.

A HEAD check is still a request, and when the page has changed you send a second one; a conditional GET does both jobs in one.

Scrapy does this for you, once you switch it on #

Scrapy’s HttpCacheMiddleware “provides low-level cache to all HTTP requests and responses”, but HTTPCACHE_ENABLED defaults to False. Its default policy, DummyPolicy, “has no awareness of any HTTP Cache-Control directives”. Scrapy.

The one you want is RFC2616Policy, which Scrapy describes as “aimed at production” and “used in continuous runs to avoid downloading unmodified data (to save bandwidth and speed up crawls)”. It revalidates with ETag and Last-Modified. Scrapy.

So the change is two settings: turn the cache on, and pick RFC2616Policy as its policy. Note that an HTTPCACHE_EXPIRATION_SECS of 0 means cached pages never expire. Scrapy.

Through a scraping API, it’s unverified #

I couldn’t confirm that a scraping API passes your conditional headers to the site, or what it bills if a 304 comes back. ScrapingBee has forward_headers and ScraperAPI lists keep_headers at no extra cost, so the header can go out. ScrapingBee, ScraperAPI.

But ScraperAPI says “We only charge for successful requests (200 and 404 status codes)”, and no doc I read mentions 304. So count conditional requests as a sure saving on direct fetches and per-GB proxies, and as an open question on per-request APIs until the vendor confirms it.

Is a vendor’s cache the same as yours? #

No. Your cache skips requests; a vendor’s cache may only make them faster.

Firecrawl is the clearest example. It returns a recent cached copy by default, with a maxAge of “172800000 (2 days)”, and maxAge: 0 forces a fresh fetch.

But its docs say plainly: “Cached results still cost 1 credit per page. Caching improves speed and latency, not credit usage.” Firecrawl.

Firecrawl also says caching can be “up to 500% faster”, which is a vendor claim about speed. Useful, but if your goal is a smaller bill, a cache you control, plus lastmod and conditional requests, is what saves money.

Render less, and load less #

Turn rendering on per page template, and in a real browser block what you don’t need. The first cuts credits; the second cuts bytes.

Rendering is the biggest per-request multiplier most people pay without noticing. ScraperAPI charges 10 credits for render=true against 1 for a normal request, and ScrapingBee’s render_js defaults to True and “costs 5 credits per request”. ScraperAPI, ScrapingBee.

My JavaScript rendering check walks through how to tell, in five steps, whether a page needs it. Run it once per template and set rendering per URL pattern.

Bytes matter once you drive your own browser over a per-GB proxy. Playwright says “You can abort requests using page.route() and route.abort()”, for example by matching image extensions or resourceType() === 'image'. Service workers can hide requests from routing, so set serviceWorkers to 'block'. Playwright.

// A sketch written from Playwright's network docs, not run.
await page.route('**/*', route =>
  route.request().resourceType() === 'image' ? route.abort() : route.continue());

On ScrapingBee, block_resources already defaults to true and “blocks all images and CSS”. ScrapingBee. On a per-request API that doesn’t lower the credit count, so this lever is mainly for per-GB plans.

Bright Data’s docs add two more: stop loading once your data is on the page, and use a browser only when plain HTTP isn’t enough. They also say to start with the cheapest IP type that works. Bright Data. For that choice, see residential vs datacenter proxies.

On Scrapy, keep HttpCompressionMiddleware on (it handles gzip, deflate, brotli and zstd), and consider lowering DOWNLOAD_MAXSIZE. Its default is 1 GiB, and “Bigger responses are aborted”; the warning size is 32 MiB. Scrapy middleware, Scrapy settings.

A patch of sunlight falling across a dark floor of broken marble tiles in front of a ribbed metal shutter, leaving the rest of the floor in shadow
Light only where it's needed: rendering, images and oversized responses cost the most, so turn them on per URL pattern instead of for every page. Photo: Marija Zaric / Unsplash · Unsplash License

How do you cap retries and spend? #

Retry a fixed, small number of times, and put a hard ceiling on what one request or one month can cost. Whether retries cost money depends on the vendor, so check before you tune.

Scrapy’s built-in default is RETRY_TIMES = 2, “in addition to the first download”, on statuses 408, 429, 500, 502, 503, 504, 522 and 524. Scrapy. Two is a sensible ceiling, and a 429 is a cue to slow down, not just retry; my guide to web scraping rate limits covers backoff.

Billing for failures differs. ZenRows says “Failed or retried requests don’t consume your balance”, and counts 404 and 410 as successful. ScraperAPI charges 200 and 404 responses, plus requests you cancel before giving it 70 seconds to finish. ZenRows, ScraperAPI.

ScrapingBee’s Auto-Mode costs 0 credits if every configuration fails. ScrapingBee. Don’t generalize from any one of these; my credits guide has the full table.

Then set ceilings. ScraperAPI’s max_cost sets “the maximum number of API credits used per scrape”, and a request over the cap gets a 403: “This request exceeds your max_cost”. It also returns an sa-credit-cost header, which is the simplest way to log cost per request. ScraperAPI cost control, ScraperAPI credits.

On Apify, if usage exceeds your plan limits, “Apify platform services will be suspended” unless you’ve turned on overage. Overage is invoiced right away once it reaches $20 on Starter or $200 on higher plans, otherwise on your next invoice; you get notified as you approach the limit. Apify.

On ZenRows, a crawl’s max_pages “caps what the crawl costs”. ZenRows.

What does this do to a 100,000-page bill? #

This is my estimate, not a measurement: the job, the duplicate share, the change rate and the JavaScript share are assumptions I chose for illustration. The credit weights are ScraperAPI’s, verified 2026-11-17.

My assumptions: 100,000 queued URLs, recrawled weekly (4 runs a month); the baseline renders every page; 10% of queued URLs are duplicates after canonicalization; lastmod is accurate and 25% of pages change each week; 20% of pages need JavaScript rendering. I found no public figure for any of these shares.

Step What changes Fetches a month Credits per fetch Credits a month Versus baseline
Baseline Render everything, 4 full runs 4 × 100,000 = 400,000 10 4,000,000
1. Drop duplicates 10% removed 4 × 90,000 = 360,000 10 3,600,000 10% fewer
2. Refetch only changed pages One full run, then 25% in each of the other three 90,000 + (3 × 22,500) = 157,500 10 1,575,000 61% fewer
3. Render only where needed 20% rendered, 80% plain 157,500 (0.2 × 10) + (0.8 × 1) = 2.8 157,500 × 2.8 = 441,000 89% fewer

My arithmetic on ScraperAPI’s published credit costs, verified 2026-11-17. No job was run.

The percentages are 1 minus the new bill over the baseline: 1,575,000 ÷ 4,000,000 is about 39%, so 61% fewer, and 441,000 ÷ 4,000,000 is about 11%, so 89% fewer.

The same chain on ZenRows’ weights (5 credits rendered, 1 plain) starts at 400,000 × 5 = 2,000,000 credits and ends at 157,500 × ((0.2 × 5) + (0.8 × 1)) = 157,500 × 1.8 = 283,500 credits, about 86% fewer. ZenRows.

No single lever gets you there. Selective rendering alone on the baseline gives 400,000 × 2.8 = 1,120,000 credits, 72% fewer; it takes the step 2 cut in requests as well to bring the bill down to 441,000.

Conditional requests aren’t in the table, because I couldn’t confirm how scraping APIs bill a 304. To turn credits into dollars, use the plan rates in my ScraperAPI pricing breakdown, or my Scrapfly pricing breakdown if that’s your vendor.

And on a per-GB proxy #

Bright Data says “A single Amazon product page loads over 20 MB of resources” and that “Blocking images alone cuts page load bandwidth by 50% or more”, but shows no method. Bright Data. Treat both as its claims.

My estimate, using those claims as assumptions: 10,000 browser page loads a month at 20 MB is 200 GB. At 10 MB after blocking images, it’s 100 GB. The 100 GB saved × $2.75 per GB (Decodo’s 100 GB tier, verified 2026-11-17) is $275 a month. Decodo.

A row of narrow white steps climbing diagonally across a pale plastered wall, each step casting a soft shadow
Step by step, not in one jump: fewer requests first, then selective rendering, is what takes the worked bill from 4 million credits down to 441,000. Photo: Stéphane Mingot / Unsplash · Unsplash License

When is switching the real answer? #

When the checklist is done and the bill is still too high. Three signals tell you the problem is the vendor, not your crawl.

Your cost per validated record stays high after fixes 1-6. List price per credit doesn’t tell you this; cost per record you can actually use does. My guide to testing a scraping API shows how to measure it, and the cheapest web scraping API comparison ranks vendors once you know it.

Most of the bill is a premium tier for protected sites. On ScraperAPI, premium plus rendering is 25 credits and ultra-premium plus rendering is 75. ScraperAPI. Another vendor may price that tier differently, which is a real reason to compare.

You’re paying an API for pages a plain fetch could get. If most of your targets are easy, a proxy plan or your own fetcher may be cheaper. My guides to scraping API vs proxies and building your own vs buying an API work through the trade.

What I would do #

I’d turn on URL canonicalization and dedup first, then drive recrawls from sitemaps and a local cache with ETag revalidation. Next, I’d switch rendering on only for the templates that need it, and block images in any browser that runs over a per-GB proxy.

Before comparing a single vendor price, I’d set a per-request max_cost where it’s offered, keep retries at two, and log the credit cost of every request. Only then would I look at switching, using cost per validated record as the yardstick.

That’s a recommendation from the docs, not a result from my own checks. The scraping APIs hub picks up from here if you do decide to shop around.

FAQ

What's the fastest way to cut a scraping API bill?

If you render JavaScript on every page, turn it off for the page types that don't need it: on ScraperAPI a rendered request costs 10 credits and a plain one costs 1. Otherwise, drop duplicate URLs and refetch only pages that changed.

Do conditional requests work through a scraping API?

I couldn't confirm it. ScraperAPI and ScrapingBee can forward headers you choose, but no doc I read says whether the target's 304 comes back to you or how it's billed. On direct fetches and per-GB proxies, a 304 carries no content, so it saves bytes.

Does Firecrawl's cache save credits?

No. Firecrawl's docs say cached results still cost 1 credit per page; caching improves speed and latency, not credit usage. Its default cache age is 2 days.

Can I trust sitemap lastmod?

Only where the site keeps it accurate. Google says it uses lastmod only if it's consistently and verifiably accurate, and ignores changefreq and priority. Spot-check a few pages before you skip refetches on the strength of it.

Do failed requests and retries cost money?

It depends on the vendor. ZenRows says failed or retried requests don't consume your balance. ScraperAPI charges for 200 and 404 responses and for requests you cancel before giving it 70 seconds to finish. My scraping API credits guide has the full table.

Is it cheaper to build my own scraper?

It can be, but count all the work. Buying an API still leaves parsing and data checks with you, and building means owning hosting and repairs. My guide on building vs buying a scraping API has a cost model for both, plus a hybrid.

Sources

19 sources, accessed .

Official pricing, docs, changelogs and status pages 17

  1. IETF RFC 9110: HTTP Semantics (conditional requests, 304) rfc-editor.org · accessed
  2. Scrapy: downloader middleware (HTTP cache, retries, compression) docs.scrapy.org · accessed
  3. Scrapy: settings (dupe filter, download size limits) docs.scrapy.org · accessed
  4. Crawlee: RequestQueue crawlee.dev · accessed
  5. Google Search Central: build and submit a sitemap developers.google.com · accessed
  6. sitemaps.org: sitemap protocol sitemaps.org · accessed
  7. Firecrawl: faster scraping (cache and maxAge) docs.firecrawl.dev · accessed
  8. Firecrawl: change tracking docs.firecrawl.dev · accessed
  9. ScrapingBee: API documentation scrapingbee.com · accessed
  10. ScraperAPI: credits and request costs docs.scraperapi.com · accessed
  11. ScraperAPI: cost control (max_cost) docs.scraperapi.com · accessed
  12. ZenRows: pricing and credit costs docs.zenrows.com · accessed
  13. Playwright: network (route.abort) playwright.dev · accessed
  14. Apify: billing and plan limits docs.apify.com · accessed
  15. Bright Data docs: reducing scraping costs docs.brightdata.com · accessed
  16. Decodo: residential proxy pricing decodo.com · accessed
  17. IPRoyal: residential proxy pricing iproyal.com · accessed

Other blogs 2

  1. ScrapeOps: how to minimize web scraping costs with Python (Jul 8, 2024) scrapeops.io · accessed
  2. Oxylabs: case study on reducing data acquisition costs oxylabs.io · accessed

What readers say

Used one of the tools in this guide? Tell other readers how it went. I read every review before it goes up, and I never turn one down for being negative. How reviews work.

No reader reviews yet. Yours could be the first.

Write a review
Your rating

What did you use it for, what worked, and what didn't? 30 to 1500 characters. Please don't include links.

Your connection to the company

Your review appears with the name you enter, after I approve it. I don't ask for your email. Privacy.