Lotook
To chunk documents for RAG effectively, split files along existing document structure first, apply a token-length cap with sentence-level overlap to oversized sections, and attach source metadata to every passage. Choosing between token, semantic, and structure-aware chunking depends on whether your corpus consists of uniform prose, shifting topics, or formatted tables and headings.
Why Does RAG Chunking Determine Retrieval Quality?
Retrieval-augmented generation (RAG) splits source documents into smaller passages before embedding and storing those passages in a search index. At query time, the system retrieves the most relevant chunks and passes them to a language model with the user’s question. Chunking decides the unit of retrieval because a full document is usually too broad to match a specific question well and too large to fit alongside other candidates in the model’s context window.
When Patrick Lewis and colleagues introduced the framework in their 2020 NeurIPS paper (Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks), their experiment split each Wikipedia article into non-overlapping chunks of about 100 words, producing about 21 million passages. That 100-word split was a 2020 experimental setting for Wikipedia rather than a production default for 2026. Modern production RAG as of November 2026 usually runs as a modular engineering pipeline (chunk, embed, index, retrieve, and generate) rather than the original end-to-end differentiable DPR-and-BART setup.
What Are the Three Main RAG Chunking Methods?
RAG chunking methods fall into three families: token-based splitting, semantic splitting, and structure-aware splitting. Each family decides where a passage begins and ends using length, meaning, or document markup.
| Chunking Method | Splitting Signal | Best Fit | Primary Risk | Indexing Cost |
|---|---|---|---|---|
| Token (or length) chunking | Fixed token or character window with sliding overlap | Uniform prose, plain text, or a length cap after structure splits | Cuts sentences, table rows, code blocks, or definitions in half | Low |
| Semantic chunking | Drop in embedding similarity between neighboring sentences | Unformatted prose where topics shift mid-document without headings | Slower, costlier indexing; produces uneven chunk sizes | High |
| Structure-aware chunking | Headings, paragraphs, lists, tables, code blocks, or pages | Markdown, HTML, API docs, legal clauses, and structured manuals | Weak structure in scanned PDFs; single sections can still be huge | Low to Medium |
1. Token and Character Chunking
Token chunking divides text into fixed-length windows measured in tokens or characters, often paired with a sliding overlap. Measuring in tokens aligns with how models work: embedding models and LLMs have strict token limits, API bills are priced in tokens, and the same paragraph can tokenize differently across model families. A character count is only a rough proxy.
Teams usually start from a window size that fits inside their embedding model’s limit (tutorials often start at a few hundred tokens) and adjust after checking retrieval failures. The primary risk of a fixed window is that it ignores syntax and layout, slicing through a sentence, table row, code block, or definition and dropping the clause that answers the question.
2. Semantic Chunking
Semantic chunking splits text when meaning shifts, such as when embedding similarity between neighboring sentences drops or a discourse boundary appears, rather than at a rigid token count. Keeping a complete idea in one chunk helps retrieval when topics change mid-document without headings.
However, semantic chunking is slower and more expensive to index because the split itself may require embedding calls. A noisy similarity threshold can also over-split or under-split text into uneven sizes, and sentence similarity is not automatically better for tables, forms, or code.
3. Structure-Aware Chunking
Structure-aware chunking uses existing document structure as the first cut: Markdown or HTML headings, paragraphs, lists, sections, pages, fenced code blocks, or table boundaries. As noted in AWS Prescriptive Guidance on RAG, preserving document structure keeps logically related context intact. A chunk starting at a Refund Policy heading that contains the full section is easier to retrieve and cite than a mid-sentence window.
Because many PDFs and scans lack clean headings (and a single heading can still span 20 pages), production pipelines usually combine methods:
- Split hierarchically by structure first: Divide the document by section or heading, then paragraph or block, keeping code blocks, tables, and lists intact.
- Apply a token-length cap second: If a section remains too large, split that section with a token window.
- Use fallback splits for unstructured text: For scanned PDFs or long unstructured sections, fall back to page splits, token windows, or semantic boundaries.
How Should You Configure Chunk Overlap and RAG Metadata?
Chunk overlap and passage metadata prevent lost boundary clauses and connect retrieved snippets back to their source documents.
Configuring Chunk Overlap
Chunk overlap repeats the end of one chunk at the start of the next so a sentence or definition sitting on a boundary appears in full in at least one retrieved passage. It acts as a simple hedge when fixed windows or fallback splits cut mid-thought.
Size your overlap to cover a typical sentence or two in your corpus while keeping it small enough that you do not store almost the same chunk twice. Overlap increases the tokens you embed, store, and retrieve, raising index size and adding near-duplicate neighbors to results. If retrieval frequently returns near-duplicate chunks, reduce your overlap or chunk size, or add a post-retrieval deduplication step.
Attaching RAG Metadata
RAG metadata consists of structured fields stored alongside each chunk, including:
- Source document ID or URL and title
- Section or heading path and page number
- Created or updated date and document type (policy, ticket, API reference)
- Access and permission labels, language, and a stable chunk ID
Teams can keep metadata out of the embedding when raw IDs or timestamps would pollute similarity, or prepend selected fields (such as a section title) into the embedded text when they improve retrieval. Metadata supports three production needs:
- Citations: Your UI or model can cite a file, heading, and page number rather than a bare passage.
- Permission and version filters: Retrieve only the latest policy version, a specific product, or documents the user is allowed to see. Permissions belong in index filters rather than only in the prompt, since placing private documents into an unfiltered RAG index exposes them to anyone who can query the system.
- Debugging and updates: When a source document changes, re-chunk and re-embed that document and retire old chunk IDs so stale facts do not cause factual errors.
How Do You Know if Your Chunks Are Too Big or Too Small?
Inspecting retrieved chunks on real user questions reveals whether your passages are oversized or fragmented.
Signs Your Chunks Are Too Large
- Buried answers: Retrieval returns a long section that mentions the query term but buries the actual answer, so the model has to hunt or misses it.
- Crowded prompts: Only a few chunks fit in the prompt, so one mediocre match crowds out better ones.
- Higher cost and latency: Sending many irrelevant tokens increases spend and response time.
- Worse distraction: Extra noise worsens context rot, where the model has the right fact in a large prompt yet leans on training memory or an adjacent, wrong sentence.
Signs Your Chunks Are Too Small
- Dangling fragments: A retrieved chunk cannot be understood without the previous sentence (such as a sentence stating an exception for enterprise plans without naming the rule).
- Split multi-step answers: A definition and its exception, a function and its parameters, or a policy and its effective date land in separate chunks, and top-k returns only one piece.
- Near-identical fragments: You retrieve many repetitive snippets and still lack a complete thought, while citations become noisy because every sentence is its own source.
To tune sizes practically, review real questions, inspect the retrieved chunks, and ask whether a human given only those passages could answer. If the needed sentence is in an unretrieved neighboring chunk, adjust overlap, structure rules, or size. If the needed sentence is inside the retrieved chunk surrounded by unrelated text, split smaller or retrieve fewer, tighter chunks.
How Does RAG Chunking Affect Context Rot?
RAG chunking is a direct lever against context rot, which is the drop in a model’s reasoning quality, instruction following, and recall as a prompt gets longer or more cluttered, even well below the advertised context limit.
RAG exists so you do not dump the whole corpus into the prompt. However, retrieving oversized chunks, or raising top-k to be safe and stuffing many vaguely similar chunks into the prompt, recreates a long, noisy context. More context is not automatically more accurate. Keeping chunks focused reduces prompt clutter, though good chunking alone cannot fix a bad embedding model or eliminate AI hallucinations when a model ignores retrieved text.
FAQ
Should I measure RAG chunk size in tokens or characters?
Measure RAG chunk size in tokens whenever possible. Embedding models and language models enforce context limits in tokens, and API usage is billed per token. Character counts are only a rough proxy because the same paragraph can tokenize into different token counts across languages, code blocks, and model families. Using your embedding model’s tokenizer prevents unexpected truncation during indexing.
Does semantic chunking always beat token chunking?
Semantic chunking does not automatically beat token or structure-aware chunking. Splitting when sentence similarity drops helps keep ideas together in unstructured prose, but it requires extra embedding work during indexing, creates uneven chunk sizes, and can over-split or under-split on a noisy threshold. For formatted documents, splitting along headings, tables, and code blocks is faster and follows author-intended boundaries.
How should I chunk tables and code blocks for RAG?
Tables, invoices, legal clauses, and code require structure-aware rules that keep blocks intact. A naive character or token split through a Markdown table or function body produces unusable chunks that separate values from column headers or break syntax. Keep code blocks, tables, and lists whole when possible, and apply structure-aware fallbacks only when a single block exceeds model limits.
Should metadata be included in the chunk embedding?
Including metadata inside the embedded text is a design choice that depends on the field. Prepending a document title or section heading path into the chunk text often helps retrieval when paragraphs rely on the heading for context. Raw identifiers, timestamps, and access labels should stay in separate filterable metadata fields so they do not pollute embedding similarity.
What happens when a source document changes?
When a source document changes, your ingestion pipeline must re-chunk and re-embed that document and retire the old chunk IDs from the index. Leaving stale chunks active alongside updated chunks causes the retriever to surface outdated policies or conflicting passages, turning an index maintenance oversight into a factual error in the final answer.
