Link Extraction

TLDR: Link extraction is the step where a crawler pulls URLs out of a fetched page so they can be queued and visited. Everything a crawler discovers beyond its seeds comes from this step.

Link extraction is what turns a single page fetch into a crawl. After retrieving a page, the crawler parses it, identifies every URL it contains, resolves those URLs into absolute form, and passes them to the URL frontier. A crawler that extracts links poorly will miss large sections of a site no matter how well the rest of its configuration is tuned.

Where Links Live:

1. Anchor elements: The href attribute on anchor tags accounts for most discoverable links.
2. Link elements: The link element carries canonical URLs, pagination hints, alternate language versions, and feed locations.
3. Frames and iframes: The src attribute points at embedded documents that may contain further links.
4. Forms: The action attribute reveals endpoints, though following them requires submitting data rather than a plain GET.
5. Sitemaps: XML sitemaps provide URL lists directly, without any parsing of page markup.
6. Embedded JSON: API responses and state objects inside script tags frequently contain URLs that appear nowhere in the visible markup.

Resolving Relative URLs:

Most links in real HTML are relative and must be resolved against the page they appeared on before they can be fetched.

A base element, if present, overrides the page URL as the resolution base. Ignoring it produces URLs that look plausible and return 404 across the entire site.

Extraction in Practice:

from bs4 import BeautifulSoup
from urllib.parse import urljoin

def extract_links(html, page_url):
    soup = BeautifulSoup(html, "lxml")

    base_tag = soup.find("base", href=True)
    base = urljoin(page_url, base_tag["href"]) if base_tag else page_url

    links = set()

    for tag in soup.find_all("a", href=True):
        href = tag["href"].strip()
        if href.startswith(("mailto:", "tel:", "javascript:", "#")):
            continue
        links.add(urljoin(base, href))

    for tag in soup.find_all("link", href=True):
        if tag.get("rel") and "next" in tag.get("rel"):
            links.add(urljoin(base, tag["href"]))

    return links

Non HTTP schemes and pure fragments are discarded, the base element is respected, and results are returned as a set so repeated links on the same page collapse immediately.

Links a Parser Will Miss:

Static HTML parsing only sees what the server sent. A growing share of links exist only after the browser runs JavaScript.

Client side routing. Single page applications often render navigation from a JavaScript router, so the initial HTML contains no anchor tags for most routes.

Infinite scroll. Additional results load through background requests triggered by scrolling, and never appear in the first response.

Click handlers. Elements that navigate through an onclick handler rather than an href are invisible to a parser looking for anchors.

Obfuscated links. Some sites assemble URLs in JavaScript specifically to keep them out of static parsers.

The answer in each case is to render the page before extracting, which is what JavaScript rendering provides. Bright Data’s Browser API executes the page in a real browser so post render links become visible to extraction.

Filtering What You Extract:

Not every discovered link should be queued. Extraction typically feeds into several checks before a URL reaches the frontier.

1. Scheme check: Discard anything that is not HTTP or HTTPS.
2. Scope check: Apply crawl scope rules for domain and path boundaries.
3. Normalization: Canonicalize through URL normalization before any comparison.
4. Deduplication: Drop URLs already crawled or already queued.
5. robots.txt check: Confirm the path is permitted for this user agent.

Performing these checks at extraction time rather than at fetch time keeps the frontier smaller and avoids spending bandwidth discovering that a URL should never have been queued.

Nofollow and Crawler Behavior:

The rel="nofollow" attribute was introduced as a signal to search engines, telling them not to pass ranking credit through a link. It was never a crawl directive, and search engines still fetch nofollow URLs in many circumstances.

For data collection crawlers the attribute carries no obligation, though it is often a useful hint. Links marked nofollow are frequently user generated, paid placements, or low value pagination, so treating them as lower priority rather than excluding them outright is usually the right balance.