Crawl Scope

TLDR: Crawl scope is the set of rules defining which URLs a crawler is allowed to visit. It covers domain boundaries, path patterns, and exclusions, and it is the main defense against a crawl expanding without limit.

Crawl scope determines the boundaries of a crawl. Without it, a crawler following links outward from its seed URLs will eventually leave the target site entirely and begin crawling the open web. Scope rules are evaluated against every discovered URL before it enters the URL frontier, so out-of-scope links are discarded before consuming any bandwidth.

Types of Scope Rules:

Domain scope: Restricts crawling to one or more hostnames. The strictest form allows only an exact host; looser forms permit subdomains.
Path scope: Limits crawling to URL paths matching a prefix or pattern, such as only /blog/ or only /products/.
Exclusion patterns: Explicitly blocks paths that would otherwise be in scope, typically login pages, carts, filters, and print views.
Parameter rules: Controls whether query-string variants are treated as distinct URLs or collapsed during URL normalization.
Content-type scope: Restricts which MIME types are fetched, commonly excluding images, video, and binaries.
Depth scope: Caps how far from the seed a crawler may travel, enforced through crawl depth.

Scope Modes Compared:

Mode	Allows	Risk	Typical use
Exact host	example.com only	Misses content on subdomains	Single-site crawls
Host and subdomains	*.example.com	May pull in unrelated subdomains	Large multi-property sites
Path-prefixed	example.com/docs/*	Misses linked content outside the prefix	Documentation, blog sections
Domain list	Named set of hosts	Requires maintenance as sites change	Competitor monitoring
Unrestricted	Any link followed	Crawl never terminates	Web-scale discovery only

Example Scope Configuration:

{
  "seeds": ["https://example.com/docs/"],
  "scope": {
    "allowed_domains": ["example.com", "docs.example.com"],
    "include_patterns": ["/docs/*", "/api-reference/*"],
    "exclude_patterns": [
      "/docs/*/print",
      "*?lang=*",
      "*/search*",
      "*.pdf"
    ],
    "max_depth": 5,
    "allowed_content_types": ["text/html"]
  }
}

The crawl stays on two hosts, covers only documentation paths, skips print and translated variants, avoids the search endpoint entirely, and fetches HTML only.

Scope Traps to Avoid:

Some URL patterns generate effectively infinite link space, and a crawler without exclusions will keep fetching them until it exhausts its budget.

Faceted navigation on e-commerce sites combines filters into unique URLs. Five filters with ten values each produce a hundred thousand combinations of the same underlying product set.

Calendar pages link to the next month indefinitely, so a crawler can walk forward into the year 3000 one page at a time.

Session identifiers in URLs make every page appear unique on each visit, defeating deduplication entirely.

Sort and pagination parameters multiply the same listing into dozens of variants that return identical content in a different order.

Setting Scope Deliberately:

Scope that is too narrow silently omits content, and the gap is often invisible until the dataset is analyzed. Scope that is too broad wastes budget and can pull the crawl onto third-party domains the operator never intended to touch.

The practical approach is to start restrictive, run a small sample crawl, inspect which URLs were rejected, then widen deliberately. Logging skipped URLs with the rule that rejected them turns scope tuning from guesswork into a reviewable process.

Bright Data’s Crawl API accepts include and exclude patterns alongside domain and depth limits, and reports which rules filtered which URLs so scope can be refined between runs.

Start free trial Start with Google