---
title: "Boilerplate Removal"
slug: boilerplate-removal
date: 2026-10-18T14:19:52+00:00
modified: 2026-10-18T14:22:16+00:00
permalink: https://brightdata.com/glossary/boilerplate-removal
type: page
---

# Boilerplate Removal

**TLDR:** Boilerplate removal strips the repeated elements that surround a page’s main content, such as navigation, footers, advertisements, and related article panels, leaving only the content unique to that page.

**Boilerplate removal**, also called content extraction or main content detection, addresses a structural problem with web pages: most of what a page contains is not what the page is about. A news article of 800 words typically ships inside a document containing a navigation menu, a subscription prompt, a sidebar of trending stories, a comment section, and a footer sitemap. The article may represent under 10 percent of the visible text.

### What Counts as Boilerplate:

1. **Navigation**: Headers, menus, breadcrumbs, and category links that appear identically on every page.
2. **Footers**: Sitemaps, legal links, contact details, and social icons.
3. **Advertisements**: Display units, sponsored blocks, and promotional banners.
4. **Related content**: Trending panels, you might also like modules, and tag clouds.
5. **Interface chrome**: Cookie notices, newsletter prompts, paywall overlays, and share buttons.
6. **User generated sections**: Comments, which are sometimes boilerplate and sometimes the most valuable part of the page, depending entirely on the use case.

### Detection Methods:

MethodHow it decidesStrengthWeaknessSemantic tagsTrusts article and main elementsAccurate when presentMany sites do not use themText densityFavors blocks with high text to markup ratioWorks without semantic markupFails on short or list heavy pagesLink densityRejects blocks that are mostly linksReliably removes navigationCan discard reference sectionsCross page comparisonRemoves content repeated site wideVery accurateNeeds multiple pages from the siteSite specific rulesHand written selectors per siteHighest precisionBreaks on redesign, does not scaleModel basedClassifies blocks by learned featuresGeneralizes across sitesSlower, higher cost per page
### Text Density in Practice:

The most widely used heuristic rests on a simple observation. Navigation elements pack many links into little text. Article paragraphs contain long runs of text with few links.

Scoring each block by its ratio of text characters to markup characters, then penalizing blocks whose text is mostly inside links, separates the two reliably on a large share of pages. Readability, the algorithm behind Firefox Reader View and many extraction libraries, is built on this principle.

It fails in predictable ways. Pages that are genuinely lists, such as product category pages or search results, look exactly like navigation by these measures and get stripped entirely. Short pages provide too little signal. Pages with heavy inline linking, including reference material, can score as boilerplate despite being the main content.

### Extraction with a Fallback Chain:

```
from bs4 import BeautifulSoup
from readability import Document

STRIP = ["nav", "header", "footer", "aside",
         "script", "style", "form", "iframe"]

def main_content(html):
    soup = BeautifulSoup(html, "lxml")

    for tag in soup.find_all(STRIP):
        tag.decompose()

    # Prefer explicit semantic containers
    for selector in ["article", "main", "[role=main]"]:
        node = soup.select_one(selector)
        if node and len(node.get_text(strip=True)) > 200:
            return node

    # Fall back to density based extraction
    return BeautifulSoup(Document(html).summary(), "lxml")
```

Semantic containers are trusted first because a site that marks its article element is telling you where the content is. The length check guards against empty wrappers. Density based extraction covers everything else.

### The Cost of Getting It Wrong:

Both error directions are damaging, in different ways.

**Removing too much** deletes real content. An over aggressive link density rule can strip a reference list, a specifications table, or the entire body of a list based page. The failure is silent, since the output still looks like a clean extraction.

**Removing too little** leaves navigation and footers in the output. In a retrieval system this is worse than it sounds: every document from the same site now shares large blocks of identical text, which pollutes embeddings and causes near duplicate chunks to crowd out genuine matches.

For [document chunking](/glossary/document-chunking) specifically, residual boilerplate is corrosive. A chunk consisting mostly of a cookie banner is retrievable, meaningless, and occupies a slot that a real answer could have filled.

### Boilerplate Removal and Token Cost:

The economic case is straightforward. A page carrying 800 words of article inside 6,000 words of surrounding furniture costs roughly eight times more to process than the article alone, and the extra tokens contribute nothing.

Across a corpus of millions of pages, boilerplate removal is frequently the single largest cost reduction available in an AI pipeline, and it improves output quality at the same time.

Bright Data’s [Crawl API](/products/crawl-api) performs content isolation before returning results, so pages arrive as clean Markdown without a separate extraction stage.

Start free trial[Start with Google](#)





 ## Trusted by 50,000+ customers worldwide



























































Welcome to the Scraping Cloud

## Maximum Control and Efficiency

 [Proxies

Premium proxy infrastructure. 100% ethically sourced IPs.

- Automated session management
- Target any city in 195 countries
- Unlimited concurrent sessions





 Proxy Services





 ](/proxy-types)

 [Unlocker

Say goodbye to blocks and CAPTCHAs with a single API.



 Unlocker





 ](/products/web-unlocker)

 [Browser

Scale dynamic scraping with built-in unblocking and proxies



 Scraping Browser





 ](/products/scraping-browser)

 [Functions

Run your scrapers as fully-hosted serverless functions.



 Functions





 ](/products/web-scraper/functions)

 [Scrapers

Extract parsed data on-demand with dedicated scraper APIs.



 Scrapers API





 ](/products/web-scraper)

 [Datasets

Get fresh, validated datasets from any public website.



 Datasets





 ](/products/datasets)

 [SERP API

Quick and easy search engine scraping on demand



 SERP API





 ](/products/serp-api)







### Ready to get started?

Start free trial
