---
title: "Polite Crawling"
slug: polite-crawling
date: 2026-10-04T08:25:20+00:00
modified: 2026-10-04T11:35:13+00:00
permalink: https://brightdata.com/glossary/polite-crawling
type: page
---

# Polite Crawling

**TLDR:** Polite crawling means fetching pages at a rate that does not degrade the target server. It combines request pacing, concurrency limits, robots.txt compliance, and honest identification.

**Polite crawling** is the practice of operating a [web crawler](/products/crawl-api) so that it collects data without placing undue load on the sites it visits. A crawler that ignores politeness can consume a target’s server capacity, degrade the experience for human visitors, and trigger blocks that end the crawl. Politeness is both an ethical obligation and a practical requirement for sustained data collection.

### Core Principles of Polite Crawling:

1. **Request pacing**: Leave a deliberate interval between consecutive requests to the same host, commonly governed by [crawl delay](/glossary/crawl-delay).
2. **Concurrency limits**: Cap the number of simultaneous connections opened against a single domain, regardless of total crawler throughput.
3. **robots.txt compliance**: Fetch and respect the target’s robots.txt before crawling, including Disallow rules and any Crawl-delay directive.
4. **Honest identification**: Send a descriptive User-Agent that identifies the crawler and provides a contact method.
5. **Off-peak scheduling**: Run heavy crawls during the target’s low-traffic hours where that information is known.
6. **Adaptive backoff**: Slow down automatically when response times rise or error rates increase, rather than pushing through.

### Reading Server Signals:

A polite crawler treats the target’s responses as instructions rather than obstacles. Rising latency usually means the server is under strain and the crawl should slow before it causes a problem. A [429 Too Many Requests](/faqs/proxy-errors/429-error-how-to-avoid) response is an explicit request to back off, and a Retry-After header states exactly how long to wait. Repeated 503 responses indicate the server is struggling and the crawl should pause entirely rather than retry aggressively.

Ignoring these signals is what converts a crawl into something indistinguishable from an attack, and it is the fastest route to a permanent IP block.

### Implementing Rate Control:

```
import time
import requests
from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()

delay = rp.crawl_delay("my-crawler") or 1.0
headers = {"User-Agent": "my-crawler/1.0 (+https://mysite.com/bot)"}

for url in urls:
    if not rp.can_fetch("my-crawler", url):
        continue

    response = requests.get(url, headers=headers, timeout=10)

    if response.status_code == 429:
        wait = int(response.headers.get("Retry-After", 60))
        time.sleep(wait)
        continue

    process(response)
    time.sleep(delay)
```

The parser reads robots.txt once, honors any declared crawl delay, skips disallowed paths, and respects Retry-After when rate limited.

### Politeness and Proxy Networks:

Distributing requests across many IP addresses increases total throughput, but it does not remove the politeness obligation. The target server experiences aggregate load regardless of how many source addresses it arrives from. A crawler running 500 concurrent requests through [residential proxies](/proxy-types/residential-proxies) places the same burden on the origin as 500 requests from one address.

Properly configured, proxy rotation serves a different purpose: it prevents a single IP from accumulating rate-limit penalties while the crawler as a whole still respects a sane per-domain request rate. Treat per-domain concurrency as the limit that matters, not per-IP.

### Why Politeness Pays:

1. **Sustained access**: Crawls that stay under the radar run for months; aggressive crawls get blocked within hours.
2. **Lower cost**: Blocked requests still consume bandwidth and proxy budget while returning nothing usable.
3. **Cleaner data**: Servers under strain return partial pages, timeouts, and error templates that pollute the dataset.
4. **Reduced legal exposure**: Demonstrable restraint supports any claim that collection was conducted responsibly.
5. **Fewer anti-bot escalations**: Sites that detect load spikes deploy stricter [anti-scraping measures](/blog/web-data/anti-scraping-techniques) that make future collection harder for everyone.

### Politeness at Scale:

The tension in large crawls is that politeness caps per-domain throughput while business requirements demand volume. The resolution is horizontal rather than vertical: crawl many domains slowly and in parallel, instead of one domain quickly. A crawler respecting a one-second delay across 10,000 domains still completes 10,000 requests per second in aggregate while remaining entirely polite to each individual host.

Bright Data’s [Crawl API](/products/crawl-api) enforces per-domain pacing and concurrency automatically, so large crawls distribute load correctly without requiring per-target tuning.

Start free trial[Start with Google](#)





 ## Trusted by 50,000+ customers worldwide



























































Welcome to the Scraping Cloud

## Maximum Control and Efficiency

 [Proxies

Premium proxy infrastructure. 100% ethically sourced IPs.

- Automated session management
- Target any city in 195 countries
- Unlimited concurrent sessions





 Proxy Services





 ](/proxy-types)

 [Unlocker

Say goodbye to blocks and CAPTCHAs with a single API.



 Unlocker





 ](/products/web-unlocker)

 [Browser

Scale dynamic scraping with built-in unblocking and proxies



 Scraping Browser





 ](/products/scraping-browser)

 [Functions

Run your scrapers as fully-hosted serverless functions.



 Functions





 ](/products/web-scraper/functions)

 [Scrapers

Extract parsed data on-demand with dedicated scraper APIs.



 Scrapers API





 ](/products/web-scraper)

 [Datasets

Get fresh, validated datasets from any public website.



 Datasets





 ](/products/datasets)

 [SERP API

Quick and easy search engine scraping on demand



 SERP API





 ](/products/serp-api)







### Ready to get started?

Start free trial