Chunking Strategies for RAG

Optimizing text chunk sizes and boundaries to preserve semantic context in vector search.

🟡 intermediate5 min readragpractical

Chunking Strategies split raw documents into smaller text passages for vector embedding and retrieval in RAG pipelines. Choosing chunk size involves trade-offs: small chunks preserve precise search specificity but lose global context, while large chunks capture broad context but dilute vector retrieval precision. Strategies range from Fixed Size Chunking with Overlap to Document Structure Aware Chunking, Semantic Chunking based on embedding distance shifts, and Parent Child Hierarchical Chunking.

Why Chunking Strategy Matters

Before text can be embedded into a Vector Database, raw documents (PDFs, Markdown files, HTML pages) must be split into Chunks.

If you chunk poorly:

Too Small (e.g. 50 tokens): Chunks lose surrounding context, leaving the LLM confused.
Too Large (e.g. 2000 tokens): Chunks contain multiple unrelated topics, diluting embedding similarity and wasting context window tokens.

┌──────────────────────────┬──────────────────────────┬──────────────────────────┐
│ 1. FIXED-SIZE CHUNKING   │ 2. STRUCTURE-AWARE       │ 3. SEMANTIC CHUNKING     │
├──────────────────────────┼──────────────────────────┼──────────────────────────┤
│ Character / token count  │ Splits by Markdown H2/H3 │ Splits when sentence     │
│ with sliding overlap.    │ headers, code blocks, or │ embedding similarity     │
│ Fast, crude baseline.    │ HTML DOM elements.       │ drops below threshold.   │
└──────────────────────────┴──────────────────────────┴──────────────────────────┘

1. Fixed-Size Chunking with Overlap

Splits text every $N$ characters or tokens (e.g. 512 tokens) with a sliding Overlap of $M$ tokens (e.g. 50 tokens).

  [ Sentence 1 ... Sentence 2 ... Sentence 3 ] ──► Chunk 1 (Tokens 1 to 512)
                          [ Sentence 3 ... Sentence 4 ... Sentence 5 ] ──► Chunk 2 (Tokens 462 to 974)
                          (50-Token Overlap Area!)

Why Overlap is Mandatory

Without overlap, a critical sentence split in half across boundary $N$ will lose its meaning in both Chunk 1 and Chunk 2.

Overlap ensures sentences near boundaries stay complete in at least one chunk.

2. Document Structure-Aware Chunking

Leverages inherent document formatting (Markdown headers ##, HTML tags <article>, or Python function boundaries def).

Respecting natural document boundaries prevents splitting a single code function or table across separate chunks.

3. Semantic Chunking (Embedding Distance Shifts)

Instead of counting characters, Semantic Chunking splits text based on semantic topic shifts:

  Sentence 1 ──► [ Embed ] ──┐
                             ├──► Cosine Similarity High ──► Keep in SAME Chunk!
  Sentence 2 ──► [ Embed ] ──┘
                             ├──► Cosine Similarity DROPS LOW ──► SPLIT NEW CHUNK HERE!
  Sentence 3 ──► [ Embed ] ──┘

Pass consecutive sentences through an embedding model.
Measure cosine distance between adjacent sentence embeddings.
Insert a chunk split whenever semantic distance exceeds a threshold.

4. Parent-Child (Hierarchical) Chunking

Resolves the fundamental trade-off between Retrieval Precision and Generation Context:

                              PARENT CHUNK (1,024 Tokens - Rich Context!)
            ┌──────────────────────────────────────┼──────────────────────────────────────┐
            ▼                                      ▼                                      ▼
  CHILD CHUNK 1 (128 Tokens)            CHILD CHUNK 2 (128 Tokens)            CHILD CHUNK 3 (128 Tokens)
  [ Vector Index Search Here! ]         [ Vector Index Search Here! ]         [ Vector Index Search Here! ]

Search index uses Small Child Chunks (128 tokens) for ultra-precise vector matching.
When Child Chunk 2 is matched, retrieve its Parent Chunk (1024 tokens) and pass the parent chunk into the LLM prompt context!

Say this out loud

Chunking Strategy splits documents for vector indexing. Fixed size chunking with overlap prevents splitting sentences across boundaries. Document structure aware chunking splits by headers or code blocks. Semantic chunking splits when sentence embedding similarity drops. Parent Child chunking searches on small child vectors for high retrieval precision while returning larger parent chunks for rich LLM context.

Followups to expect

What is Hypothetical Document Embeddings (HyDE)? Using an LLM to generate a synthetic hypothetical answer for a user query first, then embedding the hypothetical answer to search for similar real document chunks.
How do tables impact chunking in RAG? Converting complex PDF tables into Markdown or HTML tables before chunking, or summarizing each table into a text paragraph for vector indexing.

Check yourself

Question 1 of 3

Why is Chunk Overlap (e.g. 512 token chunk size with 50 token overlap) applied during document splitting for RAG?

More in LLMs & GenAI

See all →

Pretraining → SFT → RLHF5 min Fine-Tune vs RAG vs Prompt: Choosing5 min Retrieval-Augmented Generation5 min