---
title: "Crawl Delay"
slug: crawl-delay
date: 2026-10-04T12:26:11+00:00
modified: 2026-10-04T12:26:11+00:00
permalink: https://brightdata.com/glossary/crawl-delay
type: page
---

# Crawl Delay

**TLDR:** Crawl delay is the waiting period a crawler observes between consecutive requests to the same host. It is declared as an unofficial robots.txt directive and enforced client-side by the crawler.

**Crawl delay** controls request pacing. A crawl delay of 5 means the crawler waits five seconds after completing one request before issuing the next to that host. The directive is the most direct expression of [polite crawling](/glossary/polite-crawling), and because no server can force a client to wait, honoring it is voluntary.

### The robots.txt Directive:

```
User-agent: *
Crawl-delay: 10
Disallow: /admin/

User-agent: MyCrawler
Crawl-delay: 2
Disallow:
```

Delay is declared per user-agent group. A crawler identifying as MyCrawler waits two seconds; anything else waits ten. The directive was never part of the original robots.txt specification and is not covered by RFC 9309, which standardized the protocol in 2022. It remains in wide use regardless.

### Crawler Support:

CrawlerHonors Crawl-delayNotesGooglebotNoRate is set in Search Console and adjusted automaticallyBingbotYesAlso configurable in Bing Webmaster ToolsYandexYesDocumented supportScrapyYesVia DOWNLOAD\_DELAY and ROBOTSTXT\_OBEYCustom crawlersVariesMust be implemented explicitlyGoogle’s omission is deliberate rather than an oversight. Googlebot adjusts its own rate based on observed server response times, slowing when latency rises. The reasoning is that a static declared value cannot reflect current server conditions.

### Choosing a Delay Value:

Target typeSuggested delayRationaleLarge commercial site0.5–1sSubstantial infrastructure, high baseline trafficMid-size site1–3sModerate capacity, crawler load is noticeableSmall or self-hosted site5–10sLimited resources, crawler may be significant loadDeclared in robots.txtUse declared valueOperator has stated a preferenceAfter a 429 responseUse Retry-AfterServer has issued an explicit instruction### Delay and Crawl Duration:

Delay has a direct and often underestimated effect on completion time. Crawling 100,000 pages at a one-second delay takes roughly 28 hours sequentially. At five seconds it takes nearly six days. At ten seconds, close to twelve.

This is why delay cannot be set in isolation from crawl architecture. The way to maintain both politeness and throughput is parallelism across hosts rather than speed against any single host. Ten thousand domains crawled concurrently at a one-second per-host delay yields high aggregate throughput while every individual server sees a single request per second.

### Delay vs. Rate Limiting:

The two mechanisms are often confused. Crawl delay is client-side and voluntary: the crawler chooses to wait. [Rate limiting](/glossary/rate-limiting) is server-side and enforced: the server rejects requests that exceed its threshold, typically returning [429 Too Many Requests](/faqs/proxy-errors/429-error-how-to-avoid).

A crawler can ignore crawl delay but cannot ignore rate limiting, since the requests simply fail. Respecting the declared delay is largely how a crawler avoids encountering the enforced limit at all.

### Implementing Per-Host Delay:

```
from collections import defaultdict
import time

next_allowed = defaultdict(float)

def fetch_with_delay(url, host, delay):
    wait = next_allowed[host] - time.monotonic()
    if wait > 0:
        time.sleep(wait)

    response = requests.get(url, timeout=10)
    next_allowed[host] = time.monotonic() + delay
    return response
```

Tracking the next permitted request time per host, rather than sleeping after every request globally, allows the crawler to work on other hosts while one waits. This is the same principle behind the back-queue layer in a [URL frontier](/glossary/url-frontier).

Bright Data’s [Crawl API](/products/crawl-api) reads and applies declared crawl delays automatically, and adapts pacing when target servers show signs of strain.

Start free trial[Start with Google](#)





 ## Trusted by 50,000+ customers worldwide



























































Welcome to the Scraping Cloud

## Maximum Control and Efficiency

 [Proxies

Premium proxy infrastructure. 100% ethically sourced IPs.

- Automated session management
- Target any city in 195 countries
- Unlimited concurrent sessions





 Proxy Services





 ](/proxy-types)

 [Unlocker

Say goodbye to blocks and CAPTCHAs with a single API.



 Unlocker





 ](/products/web-unlocker)

 [Browser

Scale dynamic scraping with built-in unblocking and proxies



 Scraping Browser





 ](/products/scraping-browser)

 [Functions

Run your scrapers as fully-hosted serverless functions.



 Functions





 ](/products/web-scraper/functions)

 [Scrapers

Extract parsed data on-demand with dedicated scraper APIs.



 Scrapers API





 ](/products/web-scraper)

 [Datasets

Get fresh, validated datasets from any public website.



 Datasets





 ](/products/datasets)

 [SERP API

Quick and easy search engine scraping on demand



 SERP API





 ](/products/serp-api)







### Ready to get started?

Start free trial