RAG Document Chunking Best Practices: Size, Overlap, and Structure

Your RAG is bad because your chunks are bad. That is the uncomfortable truth nobody publishing "RAG best practices" blog posts wants to type out loud, because chunking is unglamorous, hard to benchmark, and impossible to fix with a clever prompt. Reranking gets the headlines. Embeddings get the model comparisons. But if your chunks are wrong, no reranker on earth saves you — you are just sorting garbage faster.

After tuning RAG systems across legal contracts, support tickets, codebases, and 300-page financial PDFs, the pattern is consistent: chunking is roughly 60% of retrieval quality. Embedding model choice is maybe 20%. Reranking is 15%. Everything else — query rewriting, hybrid search weighting, prompt engineering — fights for the remaining 5%. This post is the honest version of how to chunk documents: what works, what the marketing pages won't tell you, and how to evaluate without lying to yourself.

Why chunk size dominates retrieval quality

Retrieval is a similarity search between a query vector and a corpus of chunk vectors. The chunk you embed becomes the unit of meaning the model can reason about. Make it too small and you lose context — "the company" means nothing without knowing which company. Make it too large and the embedding becomes a smeared average across multiple topics, which means nothing matches well and everything matches mediocre.

Here is what we measured across a 12,000-document support knowledge base, varying chunk size and holding everything else constant (same embedding model, same reranker, same eval set of 500 graded queries):

Notice that 512 wins on accuracy by 11 points over 128 and 11 points over 2048. The shape is not linear and not monotonic — it is an inverted U. Most teams pick 1000 tokens because it is the LangChain default. The default is wrong for most corpora.

The right size depends on the semantic density of your content. Dense academic text packs meaning into shorter spans, so 256-384 tokens often wins. Conversational support content is verbose and needs 512-768. Legal contracts have long clause structures that demand 768-1024. There is no universal answer, which is exactly why the "just use 1000" advice is lazy.

Fixed-size chunking: when it still wins

Fixed-size chunking (split every N tokens, optionally on sentence boundaries) gets a bad reputation in 2026 because semantic chunking sounds more sophisticated. But fixed-size is the right answer more often than the discourse suggests. Specifically:

• When your corpus is large and homogeneous. A million support tickets all follow similar shape — fixed-size chunking gives predictable, uniform embeddings that the retriever loves.
• When you need reproducibility. Semantic chunkers depend on embedding models that drift. Fixed-size is deterministic. Re-running the pipeline gives the same chunks every time. That matters when you are debugging.
• When latency and cost matter. Semantic chunking requires embedding every sentence twice (once to find boundaries, once to index). At scale this doubles your embedding bill and pipeline time.
• When you have good document structure already. If documents have clean headings, fixed-size chunking within sections is often better than running a semantic splitter on top of an already-structured document.

The fixed-size approach that consistently performs well: split by tokens (not characters), respect sentence boundaries, use a chunk size calibrated to your corpus, and add modest overlap. That is it. No magic.

Semantic chunking and structure-aware splits

Semantic chunking groups text by meaning rather than length. The naive implementation: embed every sentence, compute cosine similarity between consecutive sentences, split where similarity drops below a threshold. Sounds elegant. In practice it is finicky and the threshold is corpus-dependent.

Where semantic chunking actually earns its keep: documents with mixed topics and no clean structure. Long blog posts, meeting transcripts, RFPs that weave between sections. Anywhere a fixed-size split would tear a coherent thought in half. We covered the head-to-head numbers in semantic vs fixed chunking, but the short version: semantic chunking wins by 4-7 points of recall on heterogeneous corpora, ties or loses on homogeneous ones, and always costs more to build.

Structure-aware chunking is the underrated middle ground. Instead of running a semantic model, you use the document's own structure: markdown headings, HTML sections, PDF outline nodes, code function boundaries. The split is determined by the document, not by an embedding. This is fast, deterministic, and respects what the author already encoded as meaning. For any document that has structure — and most do — this beats both fixed-size and naive semantic on quality-to-cost.

The rule we follow: structure first, semantic only if structure is missing. If a document has H1/H2/H3 hierarchy, use it. If it is wall-of-text transcript output, run semantic splitting. If it is structured XML or HTML, parse it as a tree and chunk at appropriate nodes.

Overlap: how much, and when it hurts

Chunk overlap exists to solve one problem: a sentence that sits right on a chunk boundary loses its context. If chunk A ends mid-sentence and chunk B starts mid-sentence, neither chunk is fully retrievable for a query about that idea. Overlap duplicates a token window between adjacent chunks so the boundary content lives in both.

The default LangChain overlap is 200 tokens on 1000-token chunks — a 20% overlap. We have measured this and it is too high for most corpora. Here is what overlap looked like on the same 12,000-document benchmark:

Translation: 10-15% overlap is almost always the right answer. Beyond that you are inflating your index, doubling your embedding bill, and — worst of all — causing the same content to occupy multiple top-K slots, which crowds out genuinely different chunks. We have seen production RAG systems where overlap was so aggressive that the top 5 retrieved chunks were three overlapping versions of the same paragraph. The generator hallucinated because it had no diversity in context.

Overlap also interacts with reranking. If your reranker is good, you actually want less overlap, because the reranker can handle minor boundary mismatches. If you have no reranker, slightly more overlap (15-20%) protects you. Tune based on whether you have a reranker stage downstream.

Document-type playbook: PDFs, code, tables, transcripts

One chunking strategy will not serve all document types. Here is the playbook we use for each common shape:

PDFs (reports, contracts, papers)

Extract first, chunk second. Use a structure-aware PDF extractor that preserves headings, paragraphs, lists, and tables as distinct elements. Then chunk by section, not by arbitrary token count across the whole document. Target 512-1024 tokens with 10% overlap. Keep figure captions and table titles attached to their adjacent content — splitting a caption from its table is a classic retrieval killer.

Code (repositories, technical docs)

Never chunk code by raw token count. You will cut a function in half, and the embedding of half a function is useless. Chunk by syntactic unit: function, class, method, or module. Use a parser appropriate to the language (tree-sitter is the standard). Include the surrounding imports and class signature as context. For long functions, split at logical boundaries — between try/catch blocks, between distinct phases — never mid-statement.

Tables and structured data

Tables embedded as flat text become noise. Either extract them as structured records and embed each row with its column headers as a self-contained mini-document, or convert the table into a markdown representation with the headers repeated. For long tables, chunk by row groups (10-30 rows each) and always include the header row in every chunk. This is one of the highest-leverage fixes for any RAG over financial, scientific, or operational documents.

Transcripts (calls, meetings, podcasts)

Transcripts are the worst-case for naive chunking. They are conversational, full of pronouns referring to earlier turns, and have no document structure. This is where semantic chunking earns its cost. Run a topic-shift detector or use sentence embedding similarity to find logical breakpoints. Target 400-600 tokens per chunk. Include speaker labels in the chunk text so the model knows who is talking. Parent document retrieval is especially valuable here — retrieve on small chunks, return the larger parent segment to the LLM for context.

Q&A pairs and FAQs

These are pre-chunked by design. One question plus its answer is one chunk. Do not re-chunk them. The biggest mistake we see: people run a generic 500-token splitter over a clean FAQ and end up with question 7 stapled to the answer for question 8. Respect the source format.

Evaluating chunking quality without guessing

Most teams ship chunking changes by vibe. They tweak the splitter, run three example queries, and declare victory. This is how you ship regressions. Real evaluation is not optional. Here is the minimum viable harness:

1. Build a graded eval set. 100-500 queries representative of real user questions, each labeled with the correct document(s) and ideally the correct chunk(s). Yes, this is annotation work. There is no shortcut.
2. Measure recall@K and MRR. Recall@10 tells you whether the right content made it into the top 10 retrieved chunks. MRR (Mean Reciprocal Rank) tells you how high the right answer was ranked. Both matter, for different reasons.
3. Measure end-to-end answer accuracy. Retrieval quality is not the same as answer quality. Use an LLM judge (with a clear rubric) or human raters to score whether the final answer is correct, partially correct, or wrong.
4. A/B chunk strategies on the same eval set. Change one variable at a time — chunk size, then overlap, then strategy. If you change three things at once, you learn nothing.
5. Log retrieval traces in production. Sample 5% of real queries, store the retrieved chunks, periodically audit them. Production drift is real — a chunking strategy that worked on launch data may degrade as the corpus grows.

One more eval trick that nobody talks about: look at the chunks themselves. Just read 50 random chunks from your index. Do they look like coherent units of meaning, or do they start mid-sentence and end mid-thought? Are tables intact? Are code blocks unbroken? Do headings appear with their content or orphaned? This 20-minute manual review will surface more problems than any automated metric.

The chunking decisions that matter most

If you only fix three things this week, fix these:

• Calibrate chunk size to your corpus. Run the inverted-U experiment on 50-100 queries. 256, 512, 1024 tokens. Pick the winner. Stop using defaults.
• Drop overlap to 10-15%. If you are at 20%+, you are wasting index space and crowding out diversity in top-K. Test 10% on your eval set.
• Use structure when it exists. Headings, code boundaries, table rows, FAQ pairs — respect them. Generic splitters destroy structured content.

Chunking is not the glamorous part of building a RAG system. It is the part where you grind. But because it sits at the foundation of retrieval, every other component — embeddings, reranking, generation — inherits your chunking decisions. Fix the foundation. The rest gets easier.

The teams shipping good RAG in 2026 are not the ones with the fanciest embedding model. They are the ones who measured their chunks, calibrated to their corpus, and stopped using the defaults. That is the whole game.