--- title: "URL Frontier" slug: url-frontier date: 2026-10-04T11:36:49+00:00 modified: 2026-10-04T12:21:52+00:00 permalink: https://brightdata.com/glossary/url-frontier type: page --- # URL Frontier **TLDR:** The URL frontier is the data structure holding every URL a crawler has discovered but not yet fetched. It decides what gets crawled next and enforces politeness between requests to the same host. The **URL frontier** is the queue at the center of any [web crawler](/products/crawl-api). Discovered URLs enter it, the crawler pulls URLs out of it to fetch, and newly found links are fed back in. In a trivial crawler the frontier is a simple list. In a production crawler it is a sophisticated component that must handle prioritization, politeness, deduplication, and persistence across billions of URLs. ### What the Frontier Must Handle: 1. **Ordering**: Deciding which URL to fetch next, whether by discovery order, priority score, or freshness requirement. 2. **Politeness enforcement**: Ensuring consecutive requests to the same host are spaced apart, even when thousands of that host’s URLs are queued. 3. **Deduplication**: Rejecting URLs already crawled or already queued, after [URL normalization](/glossary/url-normalization) collapses equivalent variants. 4. **Persistence**: Surviving crawler restarts without losing queued work, which matters when a crawl runs for days. 5. **Scale**: Holding far more URLs than fit in memory, which requires spilling to disk or a distributed store. ### The Two-Queue Design: The classic frontier architecture, described in Stanford’s *Introduction to Information Retrieval*, separates the problem into two queue layers. **Front queues** handle prioritization. Each front queue holds URLs of a given priority level, and a selector pulls from higher-priority queues more often. Priority might reflect page importance, update frequency, or business value. **Back queues** handle politeness. Each back queue holds URLs for exactly one host. A heap tracks the earliest time each host may next be contacted. The crawler always fetches from whichever host is due next, guaranteeing that [crawl delay](/glossary/crawl-delay) is respected without stalling the whole crawler while one host waits. This separation is what lets a crawler be simultaneously fast overall and polite to each individual server. ### Frontier States: ``` DISCOVERED → URL extracted from a page, not yet validated QUEUED → passed scope and dedup checks, waiting in frontier IN_FLIGHT → currently being fetched COMPLETED → fetched successfully, links extracted FAILED → fetch failed, may be retried with backoff SKIPPED → rejected by scope rules, robots.txt, or depth limit ``` Tracking state explicitly prevents the two most common frontier bugs: refetching URLs already processed, and losing URLs that failed mid-flight when a worker crashed. ### Deduplication at Scale: The frontier must answer “have I seen this URL before?” for every discovered link, often billions of times per crawl. Storing full URLs in a hash set becomes impractical past a certain size. Production crawlers typically hash each normalized URL to a fixed-length fingerprint and store only fingerprints. Bloom filters are common where a small false-positive rate is acceptable, since they use dramatically less memory than exact sets. The tradeoff is that a false positive silently drops a URL that was never actually crawled, so systems requiring complete coverage pair a Bloom filter with an exact backing store. ### Distributed Frontiers: When a crawl spans multiple machines, the frontier must be shared or partitioned. Partitioning by host is the standard approach: a hash of the hostname determines which worker owns that host’s URLs. This keeps all politeness logic for a given host on a single machine, which is what makes per-host rate limiting enforceable. The alternative, a centrally shared frontier, is simpler to reason about but becomes a throughput bottleneck and a single point of failure. Most large-scale systems partition and accept the added complexity of routing discovered URLs to the correct owner. ### Frontier and Crawl Outcomes: Frontier design determines what a crawl actually produces. A purely FIFO frontier performs breadth-first traversal and surfaces shallow pages first. A priority frontier can reach high-value pages early, which matters when a crawl may be cut short. A frontier without freshness scheduling will recrawl stale and fresh pages at the same rate, wasting requests on content that has not changed since [incremental crawling](/glossary/incremental-crawling) last visited it. Bright Data’s [Crawl API](/products/crawl-api) manages frontier state, deduplication, and per-host scheduling internally, so crawls scale without requiring a custom queueing layer. Start free trial[Start with Google](#) ## Trusted by 50,000+ customers worldwide Welcome to the Scraping Cloud ## Maximum Control and Efficiency [Proxies Premium proxy infrastructure. 100% ethically sourced IPs. - Automated session management - Target any city in 195 countries - Unlimited concurrent sessions Proxy Services ](/proxy-types) [Unlocker Say goodbye to blocks and CAPTCHAs with a single API. Unlocker ](/products/web-unlocker) [Browser Scale dynamic scraping with built-in unblocking and proxies Scraping Browser ](/products/scraping-browser) [Functions Run your scrapers as fully-hosted serverless functions. Functions ](/products/web-scraper/functions) [Scrapers Extract parsed data on-demand with dedicated scraper APIs. Scrapers API ](/products/web-scraper) [Datasets Get fresh, validated datasets from any public website. Datasets ](/products/datasets) [SERP API Quick and easy search engine scraping on demand SERP API ](/products/serp-api) ### Ready to get started? Start free trial