---
title: "Document Chunking"
slug: document-chunking
date: 2026-11-04T16:45:29+00:00
modified: 2026-11-04T17:04:05+00:00
permalink: https://brightdata.com/glossary/document-chunking
type: page
---

# Document Chunking

**TLDR:** Document chunking splits long documents into smaller segments that can be embedded, indexed, and retrieved individually. Chunk boundaries determine what a retrieval system is able to return.

**Document chunking** is a preparation step in retrieval systems. Embedding models accept a limited amount of text, and retrieval works best when each unit returned is coherent and self contained. A 40 page manual embedded as one vector produces a single result that is mostly irrelevant to any specific question. The same manual split into sections produces results that answer questions directly.

### Why Chunk Size Matters:

1. **Retrieval precision**: Smaller chunks match queries more exactly, because less unrelated text dilutes the embedding.
2. **Context completeness**: Larger chunks retain surrounding information that a fragment would lose.
3. **Token cost**: Retrieved chunks are inserted into the prompt, so chunk size directly affects cost per query.
4. **Answer quality**: A chunk containing the answer but missing its qualifying conditions can produce a confidently wrong response.

The tension is permanent. Precision pushes toward smaller chunks, completeness pushes toward larger ones, and no single size resolves it for all content.

### Chunking Strategies:

StrategySplits onStrengthWeaknessFixed lengthCharacter or token countSimple, predictable sizesCuts mid sentence and mid ideaSentenceSentence boundariesNever breaks a sentenceChunks vary widely in sizeParagraphBlank linesMatches author intentFails on poorly formatted textStructuralHeadings and sectionsSemantically coherent unitsRequires structured inputRecursiveHierarchy of separatorsDegrades gracefullyMore complex to configureSemanticEmbedding similarity shiftsFollows topic changesSlow and costly to compute### Structural Chunking:

Where documents have headings, splitting on them usually outperforms size based methods. A heading marks a topic boundary that the author placed deliberately, and the text beneath it is already a coherent unit.

This is a practical reason to convert pages through [HTML to Markdown](/glossary/html-to-markdown) before chunking. Markdown preserves heading hierarchy in a form that is trivial to split on, while raw HTML buries it in markup and plain text extraction discards it entirely.

The limitation is that many documents have no usable structure. Transcripts, scanned text, and single block pages provide nothing to split on, and those fall back to recursive or sentence based methods.

### Recursive Chunking:

```
SEPARATORS = ["n## ", "n### ", "nn", "n", ". ", " "]

def chunk(text, max_size, separators=SEPARATORS):
    if len(text) <= max_size:
        return [text]

    if not separators:
        # No separators left: split at the size limit
        return [text[i:i + max_size]
                for i in range(0, len(text), max_size)]

    sep, rest = separators[0], separators[1:]
    parts = text.split(sep)

    chunks = []
    for part in parts:
        if len(part) <= max_size:
            chunks.append(part)
        else:
            chunks.extend(chunk(part, max_size, rest))

    return [c for c in chunks if c.strip()]
```

The separator list is ordered from strongest boundary to weakest. Headings are tried first, then blank lines, then line breaks, then sentences, then individual words. Splitting only descends to a weaker boundary when the stronger one leaves a section still too large.

### Overlap:

Overlap repeats a portion of each chunk at the start of the next, so that information near a boundary appears in both.

Without overlap, a sentence split across two chunks is fully present in neither, and a query matching that sentence may retrieve a fragment that makes no sense alone. With overlap, the boundary region survives intact in at least one chunk.

The cost is storage and duplication. Twenty percent overlap means twenty percent more vectors to store and search, and it means the same passage can be returned twice in a single result set, which deduplication has to handle.

### Typical Starting Points:

*Figures below are common starting configurations rather than measured optima. Tune against your own content and evaluation set.*

Content typeChunk sizeOverlapTechnical documentationSection basedMinimalLong form articles500 to 1000 tokens10 to 20 percentProduct descriptionsOne record per chunkNoneTranscripts200 to 400 tokens20 to 30 percentLegal and policy textClause basedModerate### Metadata and Context Preservation:

A chunk retrieved in isolation loses everything around it. Attaching metadata restores part of that context without inflating the embedded text.

Useful fields include the source URL, document title, the heading path the chunk sits under, its position in the document, and a collection timestamp. Retrieval can then present “Installation, Requirements, step 3” rather than an unattributed paragraph, and filters can restrict search to a document set or date range before ranking begins.

Some systems also prepend the heading path to the chunk text before embedding, so topical context influences the vector itself. This improves matching on short chunks at the cost of slight repetition across the index.

### Chunking and Content Quality:

Chunking amplifies whatever it is given. If pages still contain navigation menus and footers, those fragments become retrievable chunks that match queries and return nothing useful. Running [boilerplate removal](/glossary/boilerplate-removal) before chunking is what prevents an index from filling with cookie banners.

Bright Data’s [Crawl API](/products/crawl-api) returns clean Markdown with boilerplate stripped and heading structure intact, which is the input format structural chunking requires.

Start free trial[Start with Google](#)





 ## Trusted by 50,000+ customers worldwide



























































Welcome to the Scraping Cloud

## Maximum Control and Efficiency

 [Proxies

Premium proxy infrastructure. 100% ethically sourced IPs.

- Automated session management
- Target any city in 195 countries
- Unlimited concurrent sessions





 Proxy Services





 ](/proxy-types)

 [Unlocker

Say goodbye to blocks and CAPTCHAs with a single API.



 Unlocker





 ](/products/web-unlocker)

 [Browser

Scale dynamic scraping with built-in unblocking and proxies



 Scraping Browser





 ](/products/scraping-browser)

 [Functions

Run your scrapers as fully-hosted serverless functions.



 Functions





 ](/products/web-scraper/functions)

 [Scrapers

Extract parsed data on-demand with dedicated scraper APIs.



 Scrapers API





 ](/products/web-scraper)

 [Datasets

Get fresh, validated datasets from any public website.



 Datasets





 ](/products/datasets)

 [SERP API

Quick and easy search engine scraping on demand



 SERP API





 ](/products/serp-api)







### Ready to get started?

Start free trial
