---
title: "Crawl Scope"
slug: crawl-scope
date: 2026-10-04T12:26:09+00:00
modified: 2026-10-04T12:26:09+00:00
permalink: https://brightdata.com/glossary/crawl-scope
type: page
---

# Crawl Scope

**TLDR:** Crawl scope is the set of rules defining which URLs a crawler is allowed to visit. It covers domain boundaries, path patterns, and exclusions, and it is the main defense against a crawl expanding without limit.

**Crawl scope** determines the boundaries of a crawl. Without it, a crawler following links outward from its [seed URLs](/glossary/seed-url) will eventually leave the target site entirely and begin crawling the open web. Scope rules are evaluated against every discovered URL before it enters the [URL frontier](/glossary/url-frontier), so out-of-scope links are discarded before consuming any bandwidth.

### Types of Scope Rules:

1. **Domain scope**: Restricts crawling to one or more hostnames. The strictest form allows only an exact host; looser forms permit subdomains.
2. **Path scope**: Limits crawling to URL paths matching a prefix or pattern, such as only `/blog/` or only `/products/`.
3. **Exclusion patterns**: Explicitly blocks paths that would otherwise be in scope, typically login pages, carts, filters, and print views.
4. **Parameter rules**: Controls whether query-string variants are treated as distinct URLs or collapsed during [URL normalization](/glossary/url-normalization).
5. **Content-type scope**: Restricts which MIME types are fetched, commonly excluding images, video, and binaries.
6. **Depth scope**: Caps how far from the seed a crawler may travel, enforced through [crawl depth](/glossary/crawl-depth).

### Scope Modes Compared:

ModeAllowsRiskTypical useExact hostexample.com onlyMisses content on subdomainsSingle-site crawlsHost and subdomains\*.example.comMay pull in unrelated subdomainsLarge multi-property sitesPath-prefixedexample.com/docs/\*Misses linked content outside the prefixDocumentation, blog sectionsDomain listNamed set of hostsRequires maintenance as sites changeCompetitor monitoringUnrestrictedAny link followedCrawl never terminatesWeb-scale discovery only### Example Scope Configuration:

```
{
  "seeds": ["https://example.com/docs/"],
  "scope": {
    "allowed_domains": ["example.com", "docs.example.com"],
    "include_patterns": ["/docs/*", "/api-reference/*"],
    "exclude_patterns": [
      "/docs/*/print",
      "*?lang=*",
      "*/search*",
      "*.pdf"
    ],
    "max_depth": 5,
    "allowed_content_types": ["text/html"]
  }
}
```

The crawl stays on two hosts, covers only documentation paths, skips print and translated variants, avoids the search endpoint entirely, and fetches HTML only.

### Scope Traps to Avoid:

Some URL patterns generate effectively infinite link space, and a crawler without exclusions will keep fetching them until it exhausts its budget.

**Faceted navigation** on e-commerce sites combines filters into unique URLs. Five filters with ten values each produce a hundred thousand combinations of the same underlying product set.

**Calendar pages** link to the next month indefinitely, so a crawler can walk forward into the year 3000 one page at a time.

**Session identifiers** in URLs make every page appear unique on each visit, defeating deduplication entirely.

**Sort and pagination parameters** multiply the same listing into dozens of variants that return identical content in a different order.

### Setting Scope Deliberately:

Scope that is too narrow silently omits content, and the gap is often invisible until the dataset is analyzed. Scope that is too broad wastes budget and can pull the crawl onto third-party domains the operator never intended to touch.

The practical approach is to start restrictive, run a small sample crawl, inspect which URLs were rejected, then widen deliberately. Logging skipped URLs with the rule that rejected them turns scope tuning from guesswork into a reviewable process.

Bright Data’s [Crawl API](/products/crawl-api) accepts include and exclude patterns alongside domain and depth limits, and reports which rules filtered which URLs so scope can be refined between runs.

Start free trial[Start with Google](#)





 ## Trusted by 50,000+ customers worldwide



























































Welcome to the Scraping Cloud

## Maximum Control and Efficiency

 [Proxies

Premium proxy infrastructure. 100% ethically sourced IPs.

- Automated session management
- Target any city in 195 countries
- Unlimited concurrent sessions





 Proxy Services





 ](/proxy-types)

 [Unlocker

Say goodbye to blocks and CAPTCHAs with a single API.



 Unlocker





 ](/products/web-unlocker)

 [Browser

Scale dynamic scraping with built-in unblocking and proxies



 Scraping Browser





 ](/products/scraping-browser)

 [Functions

Run your scrapers as fully-hosted serverless functions.



 Functions





 ](/products/web-scraper/functions)

 [Scrapers

Extract parsed data on-demand with dedicated scraper APIs.



 Scrapers API





 ](/products/web-scraper)

 [Datasets

Get fresh, validated datasets from any public website.



 Datasets





 ](/products/datasets)

 [SERP API

Quick and easy search engine scraping on demand



 SERP API





 ](/products/serp-api)







### Ready to get started?

Start free trial