Skip to content
CrawlfolioSourced research on web data for AI
Menu

Before you ship a crawler you have to pick its pace: how many connections per site, how long between requests, how to back off, and when to stop. No standard gives you those numbers, so you set them yourself, and this guide shows you what to base them on.

I read the RFCs, the crawler operators’ own pages, the WAF docs sites use to count you, and the defaults in Scrapy, Crawlee and Heritrix. Every number below is labelled as a standard, a framework default, an operator’s statement or my house rule.

Short on time?

Short on time? Here's my take.

The site gives no signal
One connection per host, about a 1-second gap
My house rule, and the same starting point a new Scrapy project ships with. Not a standard.
robots.txt sets a Crawl-delay
Honor it
Bing, ClaudeBot and CCBot say they do. Google doesn't, but you aren't Google.
You get a 429 or 503
Wait at least Retry-After, then halve your pace
RFC 6585 and RFC 9110 define the signal; halving is my house rule.
Errors keep coming
Capped exponential backoff with full jitter, then park the host
AWS's full-jitter approach, with Google Cloud Storage's 60-second cap and a hard limit on retries.
You see a 403 or a challenge page
Stop
That's a block, not a speed problem.

Prices verified . Sponsors cannot buy a verdict or a position.

What do the standards actually say? #

Less than you’d hope. They define the signals a site can send, but none of them sets a safe request rate.

Standard What it covers What it says What it doesn’t say
RFC 9309 (robots.txt) Rules for crawlers Crawlers “MAY interpret other records that are not part of the robots.txt protocol”; don’t use a cached copy “for more than 24 hours”; on a 5xx the crawler “MUST assume complete disallow” Crawl-delay isn’t mentioned at all
RFC 6585 Status 429 429 means “the user has sent too many requests in a given amount of time”; it “MAY include a Retry-After header” How the server “identifies the user” or “counts requests”
RFC 9110 Retry-After, 503 Retry-After says “how long the user agent ought to wait”; it “can be either an HTTP-date or a number of seconds”; 503 is “temporary overload or scheduled maintenance” Any request rate

Verified 2026-11-17: RFC 9309, RFC 6585, RFC 9110.

So Crawl-delay is a non-standard record. A crawler may read it, and the polite ones do, but no RFC defines what it means. Whether robots.txt binds you in law is a separate question, covered in is robots.txt legally binding.

The robots.txt status rules matter more than they look. RFC 9309 says a 4xx on robots.txt means the crawler “MAY access any resources”, while a 5xx means “complete disallow”. RFC 9309. A server that’s struggling to serve robots.txt is a server you shouldn’t crawl.

Not every operator follows that. Amazon says its bots use “a cached copy from the last 30 days”, and “When a file can’t be fetched, they will behave as if it does not exist”. Amazon. My policy follows the RFC: 24 hours at most, and a 5xx means stop.

Google sits between the two. It caches robots.txt “for up to 24 hours”, treats “all 4xx errors, except 429, as if a valid robots.txt file didn’t exist”, and stops crawling the site for the first 12 hours of a 5xx. Google.

Which crawlers honor Crawl-delay? #

Bing, ClaudeBot, Common Crawl’s CCBot and Heritrix say they do. Google and Amazon say they don’t, and Yandex no longer lists it.

Crawler Honors Crawl-delay? What its operator says
Googlebot No “other fields such as crawl-delay aren’t supported”
Bingbot Yes A “time window (from 1 to 30 seconds) during which BingBot will crawl your web site only once” (2012 post)
YandexBot Not listed Crawl-delay is missing from its supported directives; set crawl rate in Yandex Webmaster instead
ClaudeBot Yes “we support the non-standard Crawl-delay extension to robots.txt”
CCBot Yes “We obey the Crawl-delay parameter”, plus back-off on 429 and 5xx
Amazonbot No Its bots “do not support the crawl-delay directive”
Heritrix 3 Yes Honors Crawl-delay up to 300 seconds by default
Scrapy Unverified I didn’t open its robots.txt middleware docs

Verified 2026-11-17: Google, Bing, Yandex directives, Anthropic, Common Crawl, Amazon, Heritrix 3 profile.

Two caveats on that table. Bing’s page is a Webmaster blog post from May 3, 2012, and it’s the only official one I found; it also says a robots.txt crawl delay “will override the direction set within the Bing Webmaster Tool”. Bing.

For Yandex, I’m inferring. Its help page lists User-agent, Disallow, Sitemap, Clean-param and Allow, and doesn’t list Crawl-delay; site owners set the pace under Indexing, Crawl rate, “Set manually”. Yandex directives, Yandex crawl rate. I didn’t find a page that dates the change.

Common Crawl’s FAQ adds two useful details: “By default our crawler waits few seconds before sending the next request to the same site”, and it uses an “adaptive back-off algorithm” when a server answers 429 or 5xx. Common Crawl. That’s a sensible model for a small crawler too.

What does a Crawl-delay cost you? Simple division: a day has 86,400 seconds, so a delay of 5 caps you at 17,280 fetches a day per host, and a delay of 10 at 8,640. A 50,000-page site takes about 13.9 hours at one request a second, and about 5.8 days at Crawl-delay 10 (500,000 seconds).

Slow, but that’s the site’s call. Google doesn’t honor the field, but you aren’t Google, and Cloudflare’s verified-bot policy lists “An AI Crawler that does not respect the crawl-delay directive” as a breach. Cloudflare policy. For the full list of AI crawlers and their tokens, see AI crawler user agents.

What should you do with a 429 or 503? #

Wait at least as long as Retry-After says, then slow down for good. If there’s no Retry-After, back off with jitter. And if you get a 403 or a challenge page instead, stop.

Parse both forms of Retry-After. RFC 9110 gives one example of each: Fri, 31 Dec 1999 23:59:59 GMT, and 120, where “the delay is 2 minutes”. RFC 9110.

RFC 6585’s own 429 example uses Retry-After: 3600, an hour. RFC 6585.

When Retry-After and your own backoff disagree, take the longer of the two. If the header says 120 seconds and your backoff says 4, wait 120.

Then change your pace, not just your next request. My house rule is the same one in my ethical web scraping checklist: halve your pace on the first 429 or 503. At one request a second that means a 2-second gap, and a second signal makes it 4 seconds.

A 403 or a CAPTCHA page is different. It’s the site refusing access, not asking you to slow down, and the right response is to stop and check whether you have permission. The ethical checklist covers that decision, and my JavaScript rendering check explains why a challenge page isn’t a rendering problem either.

The ranking guides fall short here. ZenRows frames limits as something to bypass and calls proxies “the ultimate solution”, with no Retry-After handling at all. ZenRows blog.

Oxylabs shows Retry-After: 3600 and says to respect rate-limit headers, but its retry advice is generic exponential backoff that never mentions Retry-After, while Browse AI gets the core right: “When you get a 429, wait at least as long as the server suggests”. Oxylabs, Browse AI.

How should backoff work? #

Exponentially, with a cap, with randomness, and with a limit on how many times you try. Each of those four parts has a reason.

AWS’s 2015 analysis found that plain exponential backoff still leaves “clusters of calls”, because every client retries on the same schedule. Its conclusion: “The ‘Full Jitter’ approach uses less work, but slightly more time”, and jittered backoff “should be considered a standard approach for remote clients”. AWS Architecture Blog.

The formulas on that page are images, so here it is in words. Full jitter means each wait is a random amount between zero and the capped exponential delay, where the delay doubles on each retry until it hits the cap.

AWS’s Builders’ Library adds the warning: retries are “selfish”, and under overload they “can make matters significantly worse”. It says to cap backoff at “a maximum value” and to “limit the number of times that the client retries”, and it suggests a token bucket to limit retries across a whole client. AWS Builders’ Library.

For real numbers, Google Cloud Storage publishes its client defaults: in Python, an initial wait of 1 second, a multiplier of 2, a maximum of 60 seconds and a 120-second deadline (Go caps at 30 seconds). It retries 408, 429 and 5xx responses. Google Cloud Storage.

Here’s what full jitter does with those numbers. The schedule is my arithmetic, with GCS’s Python defaults as the assumption.

Retry Ceiling for the random wait Expected wait (half the ceiling) Total so far, expected Total so far, worst case
1 1 s 0.5 s 0.5 s 1 s
2 2 s 1 s 1.5 s 3 s
3 4 s 2 s 3.5 s 7 s
4 8 s 4 s 7.5 s 15 s
5 16 s 8 s 15.5 s 31 s
6 32 s 16 s 31.5 s 63 s

Based on Google Cloud Storage’s defaults and AWS’s full-jitter description. The cap of 60 seconds kicks in from the seventh retry.

# A sketch written from RFC 9110 and AWS's full-jitter description, not run.
import random
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime

def retry_after_seconds(value):
    """Read Retry-After in either form: seconds ("120") or an HTTP-date."""
    if not value:
        return None
    value = value.strip()
    if value.isdigit():
        return int(value)
    try:
        when = parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return None
    return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())

def wait_before_retry(retry_number, retry_after_header, base=1, cap=60):
    """Full jitter: a random wait between zero and the capped exponential delay."""
    jittered = random.uniform(0, min(cap, base * 2 ** (retry_number - 1)))
    server_says = retry_after_seconds(retry_after_header)
    return jittered if server_says is None else max(jittered, server_says)

How many retries? Scrapy’s built-in default is 2 per request, “in addition to the first download”, on 408, 429, 500, 502, 503, 504, 522 and 524. Scrapy middleware. My house rule is never more than 5 for one URL, and if a host keeps answering 429 or 503, I park the whole host for the rest of the run instead of cycling through its URLs.

A water drop bouncing up from a still surface, with dark and light rings spreading out evenly around it
Each ring sits further out than the last: exponential backoff spaces retries the same way, and jitter keeps many clients from landing on the same ring at once. Photo: Koen Emmers / Unsplash · Unsplash License

How many requests at once per site? #

One, to start. That’s my house rule, and it’s also where a new Scrapy project begins, though Scrapy’s built-in fallback is much higher.

Framework Per-site concurrency Delay between requests Adaptive pacing
Scrapy, new project template 1 per domain (built-in fallback: 8) 1 s, with ±50% jitter (built-in fallback: 0) AutoThrottle, off unless enabled: start 5 s, max 60 s, target concurrency 1.0
Crawlee Starts at 1, scales up to 200 requests at a time maxRequestsPerMinute defaults to Infinity Autoscaling on your machine’s available resources
Heritrix 3, default profile Not covered here 5 × the last fetch time, between 3 and 30 s Built into the delay; Crawl-delay honored up to 300 s

Framework defaults, verified 2026-11-17: Scrapy settings, Scrapy AutoThrottle, Crawlee, Heritrix 3 profile.

Scrapy: the template and the fallback differ #

Scrapy’s docs mark two values for some settings: the one the startproject template writes, and the built-in fallback you get if the setting is missing. CONCURRENT_REQUESTS_PER_DOMAIN is 1 in the template and 8 as a fallback; DOWNLOAD_DELAY, the “Minimum seconds to wait between 2 consecutive requests to the same domain”, is 1 in the template and 0 as a fallback. Scrapy settings.

So if you build settings from scratch, or delete the template’s lines, you get 8 parallel requests per domain with no delay. DOWNLOAD_DELAY_JITTER of 0.5 spreads the delay by ±50%, so a 1-second delay lands between 0.5 and 1.5 seconds. The template also sets ROBOTSTXT_OBEY to True, where the fallback is False. Scrapy settings.

AutoThrottle is off unless you enable it. When on, it aims for a delay of “latency / N”, where N is the target concurrency, and each new delay is the average of the previous one and that target. “latencies of non-200 responses are not allowed to decrease the delay”, and it never goes below DOWNLOAD_DELAY. Scrapy AutoThrottle.

My arithmetic, assuming a steady 0.8-second latency and N = 1: the target is 0.8 s, so the delay goes 5.0 → 2.9 → 1.85 → 1.33 → 1.06 s. The next step would be 0.93 s, which is below the template’s 1-second DOWNLOAD_DELAY, so it settles at 1 s. AutoThrottle makes you slower when the site slows down, never faster than your floor.

Crawlee and Heritrix #

Crawlee starts “with one parallel request at a time” and scales “to a maximum of 200 requests at a time”, based on your machine’s resources. maxRequestsPerMinute defaults to Infinity. Crawlee.

That scaling follows your machine’s resources, not the site’s health, so set maxConcurrency and maxRequestsPerMinute yourself when you crawl one host. My Crawlee vs Scrapy comparison covers the frameworks more broadly.

Heritrix 3, the Internet Archive’s crawler, shows what a cautious default looks like. Its default profile sets delayFactor 5.0, minDelayMs 3000, maxDelayMs 30000 and respectCrawlDelayUpToSeconds 300, with robots.txt obeyed. Heritrix 3 profile.

The wiki describes the delay factor as “a multiple of the last completion”. Heritrix wiki.

My arithmetic on those defaults: a 0.4-second fetch gives 2 s, raised to the 3-second floor; a 2-second fetch gives 10 s; an 8-second fetch gives 40 s, capped at 30. The slower the site answers, the longer Heritrix waits.

Other numbers you’ll see are house numbers too. Browse AI suggests 2 to 5 seconds between pages without a source. Browse AI. None of these is a standard.

What does the site see? #

A counter per IP, or per something else, over a short window, and usually a 429 when you cross it. Google’s own advice to site owners tells you what those errors mean.

Cloudflare’s rate-limiting rules count requests by characteristics over a period. On the Free plan they count by IP, with a 10-second period and a 10-second block. Pro plans can block for up to an hour, Business and Enterprise for up to a day, and Enterprise periods run up to 65,535 seconds. Cloudflare rate limiting.

The response code is the site’s choice: “The default value is 429 (Too many requests)”, and it can be changed to anything from 400 to 499. Counting can use IP, headers, cookies, path, ASN, country and JA3/JA4 fingerprints. Cloudflare parameters. Neither page mentions Retry-After, so don’t count on getting one.

Google tells site owners who need Googlebot to slow down urgently to “return a 500, 503, or 429” for “a short period of time (for example, a couple of hours, or 1-2 days)”. Google then “reduces your site’s crawl rate across the whole hostname”, and speeds up again when the errors stop. Google.

That’s the clearest statement of what these codes mean: slow down across the whole host, not just for one URL. It’s also why I halve the per-host pace rather than retrying the one URL that failed.

Cloudflare’s verified-bot policy sets the bar from the other side. A verified bot must obey “robots.txt and crawl directives” and keep “reasonable request rates”; a breach gets it removed from the allowlist, and repeated breaches may cost every bot an organisation runs its verified status. Cloudflare policy. If you run a site and want the other side of this, see how to block AI crawlers.

Your scraping API has its own limits #

They cap your account, not the site’s tolerance. A plan that allows hundreds of parallel requests says nothing about what one small site can take.

Service Limit What happens over it
ScraperAPI Rendered requests “set to 10req/sec by default”; at 25-second renders, “up to 250 rendered requests running concurrently” Not stated on the page I read
Firecrawl /scrape per minute: Free 10, Hobby 100, Standard 500, Growth 5,000, Scale 10,000 “returns a 429”
ZenRows Concurrency: Free 5, Build 20, Launch 50, Growth 100, Scale 200-400 Not covered here

Account limits, verified 2026-11-17: ScraperAPI, Firecrawl, ZenRows.

So keep the per-host limits in your own code, even behind an API. If Firecrawl itself answers 429, that’s your plan’s ceiling; back off the same way, but don’t confuse it with the target site asking you to slow down.

A long row of pale stone columns on square plinths receding along a covered walkway, their shadows striping the floor at even intervals
A steady, even pace: your own per-host limit sets the gap between requests, whatever concurrency your scraping API's plan allows. Photo: Jesse Bauer / Unsplash · Unsplash License

The policy I would ship #

Six steps, each labelled. Copy it into your crawler’s README and change the numbers only on purpose.

A rate-limit policy for one crawler

  1. Before the crawl: fetch robots.txt, cache it for 24 hours at most, and treat a 5xx on it as “disallow everything” (RFC 9309). Honor any Crawl-delay (house rule; Bing, ClaudeBot and CCBot do).
  2. Start pace: one connection per host, about a 1-second gap with jitter (house rule; matches Scrapy’s template).
  3. On a 429 or 503: parse Retry-After in both forms and wait at least that long (RFC 9110). With no header, use capped exponential backoff with full jitter: 1 s doubling to 60 s (AWS, GCS defaults).
  4. After the first 429 or 503: halve the per-host pace, and raise it only slowly while responses stay healthy (house rule, same as my ethical checklist).
  5. Limit retries: at most 5 per URL, and park a host that keeps answering 429 or 503 (house rule; Scrapy’s default is 2). On a 403 or a challenge page, stop that host (house rule).
  6. Identify yourself: a named user agent with a contact URL. Don’t rotate IPs to get around a limit (house rule).

Sources: RFC 9309, RFC 9110, AWS, Google Cloud Storage, Scrapy.

This is the same house rule as in my ethical web scraping checklist: one connection per host to start, honor any Crawl-delay, and halve your pace on the first 429 or 503. Raise it only if the site stays healthy and nothing tells you to stop.

What I would do #

On Scrapy, I’d keep the template’s one request per domain and 1-second delay with jitter, turn AutoThrottle on, and leave ROBOTSTXT_OBEY set to True. On Crawlee, I’d set maxConcurrency and maxRequestsPerMinute low for any single-host crawl instead of trusting the default of Infinity.

In both, I’d add the Retry-After parser above, back off with full jitter capped at 60 seconds, and park a host after repeated 429s or 503s rather than retry forever. Any 403 or challenge page would stop that host for the run.

That’s a recommendation from the docs, not a result from my own checks. If you’re routing through proxies, sticky vs rotating proxies covers session settings, and rotation is never a fix for a rate limit. The crawl APIs hub and the glossary entry on rate limits pick up from here.

FAQ

How many requests per second is safe for web scraping?

No standard says. My house rule is one connection per host to start, about a one-second gap, any Crawl-delay honored, and half the pace on the first 429 or 503. That matches the starting settings in a new Scrapy project, but it's a choice, not a rule anyone sets.

Does Google respect Crawl-delay?

No. Google's robots.txt docs say other fields such as crawl-delay aren't supported. Google slows down when a site returns 500, 503 or 429 errors instead. Bing, ClaudeBot and Common Crawl's CCBot say they honor it.

What does a Retry-After header look like?

Either a number of seconds, such as 120 for two minutes, or an HTTP date, such as Fri, 31 Dec 1999 23:59:59 GMT. RFC 9110 allows both, so your client has to parse both.

Is a 429 always a rate limit?

It's the standard signal for too many requests, but RFC 6585 doesn't define how a server counts them, and Cloudflare lets sites send any 4xx code from a rate-limit rule. If you get a 403 or a challenge page, treat it as a block and stop.

Should I rotate proxies to get around rate limits?

No. A rate limit is the site telling you to slow down. Switching IPs to get past it ignores that request, and Cloudflare's verified-bot policy expects crawlers to keep reasonable request rates and respect crawl-delay.

Does my scraping API handle rate limits for me?

Only its own. ScraperAPI, Firecrawl and ZenRows cap how fast your account can send requests, and those caps can be far above what a small site can take. The per-site pace is still your call.

Sources

29 sources, accessed .

Official pricing, docs, changelogs and status pages 24

  1. IETF RFC 9309: Robots Exclusion Protocol rfc-editor.org · accessed
  2. IETF RFC 6585: 429 Too Many Requests rfc-editor.org · accessed
  3. IETF RFC 9110: HTTP Semantics (Retry-After, 503) rfc-editor.org · accessed
  4. Google Search Central: how Google interprets robots.txt developers.google.com · accessed
  5. Google Search Central: reduce the Googlebot crawl rate developers.google.com · accessed
  6. Bing Webmaster blog: crawl delay and Bingbot (May 3, 2012) blogs.bing.com · accessed
  7. Yandex Webmaster: robots.txt directives yandex.com · accessed
  8. Yandex Webmaster: crawl rate setting yandex.com · accessed
  9. Anthropic: does Anthropic crawl data from the web? support.claude.com · accessed
  10. Common Crawl: FAQ (CCBot) commoncrawl.org · accessed
  11. Amazon: Amazonbot developer.amazon.com · accessed
  12. Cloudflare: verified bots policy developers.cloudflare.com · accessed
  13. Cloudflare: rate limiting rules developers.cloudflare.com · accessed
  14. Cloudflare: rate limiting rule parameters developers.cloudflare.com · accessed
  15. AWS Architecture Blog: exponential backoff and jitter (Mar 4, 2015) aws.amazon.com · accessed
  16. AWS Builders' Library: timeouts, retries and backoff with jitter (Dec 5, 2019) builder.aws.com · accessed
  17. Google Cloud Storage: retry strategy docs.cloud.google.com · accessed
  18. Scrapy: settings (concurrency, download delay) docs.scrapy.org · accessed
  19. Scrapy: AutoThrottle extension docs.scrapy.org · accessed
  20. Scrapy: downloader middleware (retries) docs.scrapy.org · accessed
  21. Crawlee: scaling crawlers crawlee.dev · accessed
  22. ScraperAPI: JS rendering FAQ (rendering rate limit) docs.scraperapi.com · accessed
  23. Firecrawl: rate limits docs.firecrawl.dev · accessed
  24. ZenRows: pricing and concurrency per plan docs.zenrows.com · accessed

GitHub code, issues and releases 2

  1. Heritrix 3: default crawl profile (politeness settings) raw.githubusercontent.com · accessed
  2. Heritrix 3 wiki: politeness parameters github.com · accessed

Other blogs 3

  1. ZenRows blog: rate limits in web scraping zenrows.com · accessed
  2. Browse AI glossary: rate limiting browse.ai · accessed
  3. Oxylabs blog: rate limiting (updated Mar 3, 2025) oxylabs.io · accessed

What readers say

Used one of the tools in this guide? Tell other readers how it went. I read every review before it goes up, and I never turn one down for being negative. How reviews work.

No reader reviews yet. Yours could be the first.

Write a review
Your rating

What did you use it for, what worked, and what didn't? 30 to 1500 characters. Please don't include links.

Your connection to the company

Your review appears with the name you enter, after I approve it. I don't ask for your email. Privacy.

  1. Explainer

    How to crawl an entire website for your LLM

    Find every URL, keep the crawl in scope and price it first. Settings and costs for Firecrawl, Apify, Crawl4AI, Cloudflare and Spider, checked Nov 2026.

    14 min read