---
title: "Breadth First vs. Depth First Crawling"
slug: breadth-first-vs-depth-first-crawling
date: 2026-10-07T08:56:58+00:00
modified: 2026-10-07T08:56:58+00:00
permalink: https://brightdata.com/glossary/breadth-first-vs-depth-first-crawling
type: page
---

# Breadth First vs. Depth First Crawling

**TLDR:** Breadth first crawling visits every page at one link level before going deeper. Depth first follows a single path as far as it goes before backtracking. The choice determines which pages a crawl reaches first and what it covers if stopped early.

Both strategies describe the order in which a crawler removes URLs from its [URL frontier](/glossary/url-frontier). The mechanical difference is small: breadth first uses a FIFO queue, depth first uses a LIFO stack. The practical consequences for coverage, memory, and politeness are substantial.

### How Each Traverses:

Starting from a [seed URL](/glossary/seed-url) that links to pages A, B, and C, where A links to A1 and A2:

**Breadth first order:** seed, A, B, C, A1, A2, B1, B2, C1, C2. All of depth 1 completes before any depth 2 page is touched.

**Depth first order:** seed, A, A1, A1a, A1b, A2, B, B1. The crawler descends as far as it can through A before it ever looks at B.

### Direct Comparison:

PropertyBreadth firstDepth firstQueue structureFIFOLIFOMemory useHigh, holds entire levelLow, holds one pathCoverage if stopped earlyComplete at shallow levelsOne branch deep, rest untouchedHost distributionSpread across many hostsConcentrated on one hostPolitenessNatural, requests spread outPoor, hammers a single hostTrap vulnerabilityLower, traps compete with other URLsHigh, can descend indefinitelyReaches important pagesEarly, they tend to be shallowUnpredictable### Implementation:

```
from collections import deque

def crawl(seed, max_depth, strategy="bfs"):
    queue = deque([(seed, 0)])
    seen = {seed}

    while queue:
        url, depth = queue.popleft() if strategy == "bfs" else queue.pop()

        if depth >= max_depth:
            continue

        for link in extract_links(fetch(url), url):
            if link not in seen:
                seen.add(link)
                queue.append((link, depth + 1))
```

One method call separates the two strategies. Using `popleft` takes the oldest queued URL and produces breadth first order. Using `pop` takes the most recently added URL and produces depth first.

### Why Breadth First Dominates:

Almost every production web crawler uses breadth first traversal or a priority variant of it, for several reinforcing reasons.

**Important pages are shallow.** Sites link to their most valuable content from the homepage and primary navigation. Breadth first reaches that content immediately, while depth first may spend thousands of requests in an archive before touching a main category.

**Partial crawls stay useful.** A breadth first crawl halted at any point has complete coverage up to some depth. A depth first crawl halted early has one deeply explored branch and nothing else, which is rarely a usable dataset.

**Politeness falls out naturally.** Because a level typically spans many hosts, consecutive requests tend to hit different servers, which supports [polite crawling](/glossary/polite-crawling) without extra scheduling. Depth first does the opposite, concentrating requests on one host.

**Traps are contained.** A calendar generating infinite pages pulls a depth first crawler in forever. Under breadth first, those URLs sit in a queue alongside everything else at that level and consume only their share of the budget.

### When Depth First Fits:

Depth first is the better choice in a narrow set of cases, usually where the target structure is known in advance.

Extracting a single documentation section, following a linear chain such as paginated results, or operating under tight memory constraints all favor depth first. Its memory profile is genuinely better: breadth first on a large site can hold millions of URLs at a single level, while depth first holds only the current path plus siblings.

For general data collection, that memory advantage rarely outweighs the coverage and politeness costs.

### What Production Crawlers Actually Do:

Large scale crawlers rarely implement either strategy in pure form. They use priority queues in which breadth first is the default ordering, modified by signals such as page importance, change frequency, domain quotas, and business value.

The result behaves like breadth first with exceptions: a high value product page discovered at depth 4 can be fetched before a low value tag page at depth 2. The underlying commitment to level by level progress remains, which is what keeps coverage predictable and [crawl depth](/glossary/crawl-depth) limits meaningful.

Start free trial[Start with Google](#)





 ## Trusted by 50,000+ customers worldwide



























































Welcome to the Scraping Cloud

## Maximum Control and Efficiency

 [Proxies

Premium proxy infrastructure. 100% ethically sourced IPs.

- Automated session management
- Target any city in 195 countries
- Unlimited concurrent sessions





 Proxy Services





 ](/proxy-types)

 [Unlocker

Say goodbye to blocks and CAPTCHAs with a single API.



 Unlocker





 ](/products/web-unlocker)

 [Browser

Scale dynamic scraping with built-in unblocking and proxies



 Scraping Browser





 ](/products/scraping-browser)

 [Functions

Run your scrapers as fully-hosted serverless functions.



 Functions





 ](/products/web-scraper/functions)

 [Scrapers

Extract parsed data on-demand with dedicated scraper APIs.



 Scrapers API





 ](/products/web-scraper)

 [Datasets

Get fresh, validated datasets from any public website.



 Datasets





 ](/products/datasets)

 [SERP API

Quick and easy search engine scraping on demand



 SERP API





 ](/products/serp-api)







### Ready to get started?

Start free trial