Explainer
Web scraping rate limits: how fast is too fast?
No standard sets a safe request rate. Here's what RFCs, Google, Bing, Cloudflare and Scrapy actually say, plus a backoff policy you can copy.

Before you ship a crawler you have to pick its pace: how many connections per site, how long between requests, how to back off, and when to stop. No standard gives you those numbers, so you set them yourself, and this guide shows you what to base them on.
I read the RFCs, the crawler operators’ own pages, the WAF docs sites use to count you, and the defaults in Scrapy, Crawlee and Heritrix. Every number below is labelled as a standard, a framework default, an operator’s statement or my house rule.
Short on time?
Short on time? Here's my take.
- The site gives no signal
- One connection per host, about a 1-second gap
- My house rule, and the same starting point a new Scrapy project ships with. Not a standard.
- robots.txt sets a Crawl-delay
- Honor it
- Bing, ClaudeBot and CCBot say they do. Google doesn't, but you aren't Google.
- You get a 429 or 503
- Wait at least Retry-After, then halve your pace
- RFC 6585 and RFC 9110 define the signal; halving is my house rule.
- Errors keep coming
- Capped exponential backoff with full jitter, then park the host
- AWS's full-jitter approach, with Google Cloud Storage's 60-second cap and a hard limit on retries.
Prices verified . Sponsors cannot buy a verdict or a position.
What do the standards actually say? #
Less than you’d hope. They define the signals a site can send, but none of them sets a safe request rate.
| Standard | What it covers | What it says | What it doesn’t say |
|---|---|---|---|
| RFC 9309 (robots.txt) | Rules for crawlers | Crawlers “MAY interpret other records that are not part of the robots.txt protocol”; don’t use a cached copy “for more than 24 hours”; on a 5xx the crawler “MUST assume complete disallow” | Crawl-delay isn’t mentioned at all |
| RFC 6585 | Status 429 | 429 means “the user has sent too many requests in a given amount of time”; it “MAY include a Retry-After header” | How the server “identifies the user” or “counts requests” |
| RFC 9110 | Retry-After, 503 | Retry-After says “how long the user agent ought to wait”; it “can be either an HTTP-date or a number of seconds”; 503 is “temporary overload or scheduled maintenance” | Any request rate |
Verified 2026-11-17: RFC 9309, RFC 6585, RFC 9110.
So Crawl-delay is a non-standard record. A crawler may read it, and the polite ones do, but no RFC defines what it means. Whether robots.txt binds you in law is a separate question, covered in is robots.txt legally binding.
The robots.txt status rules matter more than they look. RFC 9309 says a 4xx on robots.txt means the crawler “MAY access any resources”, while a 5xx means “complete disallow”. RFC 9309. A server that’s struggling to serve robots.txt is a server you shouldn’t crawl.
Not every operator follows that. Amazon says its bots use “a cached copy from the last 30 days”, and “When a file can’t be fetched, they will behave as if it does not exist”. Amazon. My policy follows the RFC: 24 hours at most, and a 5xx means stop.
Google sits between the two. It caches robots.txt “for up to 24 hours”, treats “all 4xx errors, except 429, as if a valid robots.txt file didn’t exist”, and stops crawling the site for the first 12 hours of a 5xx. Google.
Which crawlers honor Crawl-delay? #
Bing, ClaudeBot, Common Crawl’s CCBot and Heritrix say they do. Google and Amazon say they don’t, and Yandex no longer lists it.
| Crawler | Honors Crawl-delay? | What its operator says |
|---|---|---|
| Googlebot | No | “other fields such as crawl-delay aren’t supported” |
| Bingbot | Yes | A “time window (from 1 to 30 seconds) during which BingBot will crawl your web site only once” (2012 post) |
| YandexBot | Not listed | Crawl-delay is missing from its supported directives; set crawl rate in Yandex Webmaster instead |
| ClaudeBot | Yes | “we support the non-standard Crawl-delay extension to robots.txt” |
| CCBot | Yes | “We obey the Crawl-delay parameter”, plus back-off on 429 and 5xx |
| Amazonbot | No | Its bots “do not support the crawl-delay directive” |
| Heritrix 3 | Yes | Honors Crawl-delay up to 300 seconds by default |
| Scrapy | Unverified | I didn’t open its robots.txt middleware docs |
Verified 2026-11-17: Google, Bing, Yandex directives, Anthropic, Common Crawl, Amazon, Heritrix 3 profile.
Two caveats on that table. Bing’s page is a Webmaster blog post from May 3, 2012, and it’s the only official one I found; it also says a robots.txt crawl delay “will override the direction set within the Bing Webmaster Tool”. Bing.
For Yandex, I’m inferring. Its help page lists User-agent, Disallow, Sitemap, Clean-param and Allow, and doesn’t list Crawl-delay; site owners set the pace under Indexing, Crawl rate, “Set manually”. Yandex directives, Yandex crawl rate. I didn’t find a page that dates the change.
Common Crawl’s FAQ adds two useful details: “By default our crawler waits few seconds before sending the next request to the same site”, and it uses an “adaptive back-off algorithm” when a server answers 429 or 5xx. Common Crawl. That’s a sensible model for a small crawler too.
What does a Crawl-delay cost you? Simple division: a day has 86,400 seconds, so a delay of 5 caps you at 17,280 fetches a day per host, and a delay of 10 at 8,640. A 50,000-page site takes about 13.9 hours at one request a second, and about 5.8 days at Crawl-delay 10 (500,000 seconds).
Slow, but that’s the site’s call. Google doesn’t honor the field, but you aren’t Google, and Cloudflare’s verified-bot policy lists “An AI Crawler that does not respect the crawl-delay directive” as a breach. Cloudflare policy. For the full list of AI crawlers and their tokens, see AI crawler user agents.
What should you do with a 429 or 503? #
Wait at least as long as Retry-After says, then slow down for good. If there’s no Retry-After, back off with jitter. And if you get a 403 or a challenge page instead, stop.
Parse both forms of Retry-After. RFC 9110 gives one example of each: Fri, 31 Dec 1999 23:59:59 GMT, and 120, where “the delay is 2 minutes”. RFC 9110.
RFC 6585’s own 429 example uses Retry-After: 3600, an hour. RFC 6585.
When Retry-After and your own backoff disagree, take the longer of the two. If the header says 120 seconds and your backoff says 4, wait 120.
Then change your pace, not just your next request. My house rule is the same one in my ethical web scraping checklist: halve your pace on the first 429 or 503. At one request a second that means a 2-second gap, and a second signal makes it 4 seconds.
A 403 or a CAPTCHA page is different. It’s the site refusing access, not asking you to slow down, and the right response is to stop and check whether you have permission. The ethical checklist covers that decision, and my JavaScript rendering check explains why a challenge page isn’t a rendering problem either.
The ranking guides fall short here. ZenRows frames limits as something to bypass and calls proxies “the ultimate solution”, with no Retry-After handling at all. ZenRows blog.
Oxylabs shows Retry-After: 3600 and says to respect rate-limit headers, but its retry advice is generic exponential backoff that never mentions Retry-After, while Browse AI gets the core right: “When you get a 429, wait at least as long as the server suggests”. Oxylabs, Browse AI.
How should backoff work? #
Exponentially, with a cap, with randomness, and with a limit on how many times you try. Each of those four parts has a reason.
AWS’s 2015 analysis found that plain exponential backoff still leaves “clusters of calls”, because every client retries on the same schedule. Its conclusion: “The ‘Full Jitter’ approach uses less work, but slightly more time”, and jittered backoff “should be considered a standard approach for remote clients”. AWS Architecture Blog.
The formulas on that page are images, so here it is in words. Full jitter means each wait is a random amount between zero and the capped exponential delay, where the delay doubles on each retry until it hits the cap.
AWS’s Builders’ Library adds the warning: retries are “selfish”, and under overload they “can make matters significantly worse”. It says to cap backoff at “a maximum value” and to “limit the number of times that the client retries”, and it suggests a token bucket to limit retries across a whole client. AWS Builders’ Library.
For real numbers, Google Cloud Storage publishes its client defaults: in Python, an initial wait of 1 second, a multiplier of 2, a maximum of 60 seconds and a 120-second deadline (Go caps at 30 seconds). It retries 408, 429 and 5xx responses. Google Cloud Storage.
Here’s what full jitter does with those numbers. The schedule is my arithmetic, with GCS’s Python defaults as the assumption.
| Retry | Ceiling for the random wait | Expected wait (half the ceiling) | Total so far, expected | Total so far, worst case |
|---|---|---|---|---|
| 1 | 1 s | 0.5 s | 0.5 s | 1 s |
| 2 | 2 s | 1 s | 1.5 s | 3 s |
| 3 | 4 s | 2 s | 3.5 s | 7 s |
| 4 | 8 s | 4 s | 7.5 s | 15 s |
| 5 | 16 s | 8 s | 15.5 s | 31 s |
| 6 | 32 s | 16 s | 31.5 s | 63 s |
Based on Google Cloud Storage’s defaults and AWS’s full-jitter description. The cap of 60 seconds kicks in from the seventh retry.
# A sketch written from RFC 9110 and AWS's full-jitter description, not run.
import random
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
def retry_after_seconds(value):
"""Read Retry-After in either form: seconds ("120") or an HTTP-date."""
if not value:
return None
value = value.strip()
if value.isdigit():
return int(value)
try:
when = parsedate_to_datetime(value)
except (TypeError, ValueError):
return None
return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())
def wait_before_retry(retry_number, retry_after_header, base=1, cap=60):
"""Full jitter: a random wait between zero and the capped exponential delay."""
jittered = random.uniform(0, min(cap, base * 2 ** (retry_number - 1)))
server_says = retry_after_seconds(retry_after_header)
return jittered if server_says is None else max(jittered, server_says)
How many retries? Scrapy’s built-in default is 2 per request, “in addition to the first download”, on 408, 429, 500, 502, 503, 504, 522 and 524. Scrapy middleware. My house rule is never more than 5 for one URL, and if a host keeps answering 429 or 503, I park the whole host for the rest of the run instead of cycling through its URLs.

How many requests at once per site? #
One, to start. That’s my house rule, and it’s also where a new Scrapy project begins, though Scrapy’s built-in fallback is much higher.
| Framework | Per-site concurrency | Delay between requests | Adaptive pacing |
|---|---|---|---|
| Scrapy, new project template | 1 per domain (built-in fallback: 8) | 1 s, with ±50% jitter (built-in fallback: 0) | AutoThrottle, off unless enabled: start 5 s, max 60 s, target concurrency 1.0 |
| Crawlee | Starts at 1, scales up to 200 requests at a time | maxRequestsPerMinute defaults to Infinity |
Autoscaling on your machine’s available resources |
| Heritrix 3, default profile | Not covered here | 5 × the last fetch time, between 3 and 30 s | Built into the delay; Crawl-delay honored up to 300 s |
Framework defaults, verified 2026-11-17: Scrapy settings, Scrapy AutoThrottle, Crawlee, Heritrix 3 profile.
Scrapy: the template and the fallback differ #
Scrapy’s docs mark two values for some settings: the one the startproject template writes, and the built-in fallback you get if the setting is missing. CONCURRENT_REQUESTS_PER_DOMAIN is 1 in the template and 8 as a fallback; DOWNLOAD_DELAY, the “Minimum seconds to wait between 2 consecutive requests to the same domain”, is 1 in the template and 0 as a fallback. Scrapy settings.
So if you build settings from scratch, or delete the template’s lines, you get 8 parallel requests per domain with no delay. DOWNLOAD_DELAY_JITTER of 0.5 spreads the delay by ±50%, so a 1-second delay lands between 0.5 and 1.5 seconds. The template also sets ROBOTSTXT_OBEY to True, where the fallback is False. Scrapy settings.
AutoThrottle is off unless you enable it. When on, it aims for a delay of “latency / N”, where N is the target concurrency, and each new delay is the average of the previous one and that target. “latencies of non-200 responses are not allowed to decrease the delay”, and it never goes below DOWNLOAD_DELAY. Scrapy AutoThrottle.
My arithmetic, assuming a steady 0.8-second latency and N = 1: the target is 0.8 s, so the delay goes 5.0 → 2.9 → 1.85 → 1.33 → 1.06 s. The next step would be 0.93 s, which is below the template’s 1-second DOWNLOAD_DELAY, so it settles at 1 s. AutoThrottle makes you slower when the site slows down, never faster than your floor.
Crawlee and Heritrix #
Crawlee starts “with one parallel request at a time” and scales “to a maximum of 200 requests at a time”, based on your machine’s resources. maxRequestsPerMinute defaults to Infinity. Crawlee.
That scaling follows your machine’s resources, not the site’s health, so set maxConcurrency and maxRequestsPerMinute yourself when you crawl one host. My Crawlee vs Scrapy comparison covers the frameworks more broadly.
Heritrix 3, the Internet Archive’s crawler, shows what a cautious default looks like. Its default profile sets delayFactor 5.0, minDelayMs 3000, maxDelayMs 30000 and respectCrawlDelayUpToSeconds 300, with robots.txt obeyed. Heritrix 3 profile.
The wiki describes the delay factor as “a multiple of the last completion”. Heritrix wiki.
My arithmetic on those defaults: a 0.4-second fetch gives 2 s, raised to the 3-second floor; a 2-second fetch gives 10 s; an 8-second fetch gives 40 s, capped at 30. The slower the site answers, the longer Heritrix waits.
Other numbers you’ll see are house numbers too. Browse AI suggests 2 to 5 seconds between pages without a source. Browse AI. None of these is a standard.
What does the site see? #
A counter per IP, or per something else, over a short window, and usually a 429 when you cross it. Google’s own advice to site owners tells you what those errors mean.
Cloudflare’s rate-limiting rules count requests by characteristics over a period. On the Free plan they count by IP, with a 10-second period and a 10-second block. Pro plans can block for up to an hour, Business and Enterprise for up to a day, and Enterprise periods run up to 65,535 seconds. Cloudflare rate limiting.
The response code is the site’s choice: “The default value is 429 (Too many requests)”, and it can be changed to anything from 400 to 499. Counting can use IP, headers, cookies, path, ASN, country and JA3/JA4 fingerprints. Cloudflare parameters. Neither page mentions Retry-After, so don’t count on getting one.
Google tells site owners who need Googlebot to slow down urgently to “return a 500, 503, or 429” for “a short period of time (for example, a couple of hours, or 1-2 days)”. Google then “reduces your site’s crawl rate across the whole hostname”, and speeds up again when the errors stop. Google.
That’s the clearest statement of what these codes mean: slow down across the whole host, not just for one URL. It’s also why I halve the per-host pace rather than retrying the one URL that failed.
Cloudflare’s verified-bot policy sets the bar from the other side. A verified bot must obey “robots.txt and crawl directives” and keep “reasonable request rates”; a breach gets it removed from the allowlist, and repeated breaches may cost every bot an organisation runs its verified status. Cloudflare policy. If you run a site and want the other side of this, see how to block AI crawlers.
Your scraping API has its own limits #
They cap your account, not the site’s tolerance. A plan that allows hundreds of parallel requests says nothing about what one small site can take.
| Service | Limit | What happens over it |
|---|---|---|
| ScraperAPI | Rendered requests “set to 10req/sec by default”; at 25-second renders, “up to 250 rendered requests running concurrently” | Not stated on the page I read |
| Firecrawl | /scrape per minute: Free 10, Hobby 100, Standard 500, Growth 5,000, Scale 10,000 |
“returns a 429” |
| ZenRows | Concurrency: Free 5, Build 20, Launch 50, Growth 100, Scale 200-400 | Not covered here |
Account limits, verified 2026-11-17: ScraperAPI, Firecrawl, ZenRows.
So keep the per-host limits in your own code, even behind an API. If Firecrawl itself answers 429, that’s your plan’s ceiling; back off the same way, but don’t confuse it with the target site asking you to slow down.

The policy I would ship #
Six steps, each labelled. Copy it into your crawler’s README and change the numbers only on purpose.
A rate-limit policy for one crawler
- Before the crawl: fetch robots.txt, cache it for 24 hours at most, and treat a 5xx on it as “disallow everything” (RFC 9309). Honor any Crawl-delay (house rule; Bing, ClaudeBot and CCBot do).
- Start pace: one connection per host, about a 1-second gap with jitter (house rule; matches Scrapy’s template).
- On a 429 or 503: parse Retry-After in both forms and wait at least that long (RFC 9110). With no header, use capped exponential backoff with full jitter: 1 s doubling to 60 s (AWS, GCS defaults).
- After the first 429 or 503: halve the per-host pace, and raise it only slowly while responses stay healthy (house rule, same as my ethical checklist).
- Limit retries: at most 5 per URL, and park a host that keeps answering 429 or 503 (house rule; Scrapy’s default is 2). On a 403 or a challenge page, stop that host (house rule).
- Identify yourself: a named user agent with a contact URL. Don’t rotate IPs to get around a limit (house rule).
Sources: RFC 9309, RFC 9110, AWS, Google Cloud Storage, Scrapy.
This is the same house rule as in my ethical web scraping checklist: one connection per host to start, honor any Crawl-delay, and halve your pace on the first 429 or 503. Raise it only if the site stays healthy and nothing tells you to stop.
What I would do #
On Scrapy, I’d keep the template’s one request per domain and 1-second delay with jitter, turn AutoThrottle on, and leave ROBOTSTXT_OBEY set to True. On Crawlee, I’d set maxConcurrency and maxRequestsPerMinute low for any single-host crawl instead of trusting the default of Infinity.
In both, I’d add the Retry-After parser above, back off with full jitter capped at 60 seconds, and park a host after repeated 429s or 503s rather than retry forever. Any 403 or challenge page would stop that host for the run.
That’s a recommendation from the docs, not a result from my own checks. If you’re routing through proxies, sticky vs rotating proxies covers session settings, and rotation is never a fix for a rate limit. The crawl APIs hub and the glossary entry on rate limits pick up from here.
FAQ
How many requests per second is safe for web scraping?
No standard says. My house rule is one connection per host to start, about a one-second gap, any Crawl-delay honored, and half the pace on the first 429 or 503. That matches the starting settings in a new Scrapy project, but it's a choice, not a rule anyone sets.
Does Google respect Crawl-delay?
No. Google's robots.txt docs say other fields such as crawl-delay aren't supported. Google slows down when a site returns 500, 503 or 429 errors instead. Bing, ClaudeBot and Common Crawl's CCBot say they honor it.
What does a Retry-After header look like?
Either a number of seconds, such as 120 for two minutes, or an HTTP date, such as Fri, 31 Dec 1999 23:59:59 GMT. RFC 9110 allows both, so your client has to parse both.
Is a 429 always a rate limit?
It's the standard signal for too many requests, but RFC 6585 doesn't define how a server counts them, and Cloudflare lets sites send any 4xx code from a rate-limit rule. If you get a 403 or a challenge page, treat it as a block and stop.
Should I rotate proxies to get around rate limits?
No. A rate limit is the site telling you to slow down. Switching IPs to get past it ignores that request, and Cloudflare's verified-bot policy expects crawlers to keep reasonable request rates and respect crawl-delay.
Does my scraping API handle rate limits for me?
Only its own. ScraperAPI, Firecrawl and ZenRows cap how fast your account can send requests, and those caps can be far above what a small site can take. The per-site pace is still your call.
Sources
29 sources, accessed .
Official pricing, docs, changelogs and status pages 24
- IETF RFC 9309: Robots Exclusion Protocol
- IETF RFC 6585: 429 Too Many Requests
- IETF RFC 9110: HTTP Semantics (Retry-After, 503)
- Google Search Central: how Google interprets robots.txt
- Google Search Central: reduce the Googlebot crawl rate
- Bing Webmaster blog: crawl delay and Bingbot (May 3, 2012)
- Yandex Webmaster: robots.txt directives
- Yandex Webmaster: crawl rate setting
- Anthropic: does Anthropic crawl data from the web?
- Common Crawl: FAQ (CCBot)
- Amazon: Amazonbot
- Cloudflare: verified bots policy
- Cloudflare: rate limiting rules
- Cloudflare: rate limiting rule parameters
- AWS Architecture Blog: exponential backoff and jitter (Mar 4, 2015)
- AWS Builders' Library: timeouts, retries and backoff with jitter (Dec 5, 2019)
- Google Cloud Storage: retry strategy
- Scrapy: settings (concurrency, download delay)
- Scrapy: AutoThrottle extension
- Scrapy: downloader middleware (retries)
- Crawlee: scaling crawlers
- ScraperAPI: JS rendering FAQ (rendering rate limit)
- Firecrawl: rate limits
- ZenRows: pricing and concurrency per plan


