How RAG Chunking Works

Before a document can be embedded and retrieved, it's split into smaller pieces called chunks. Chunking exists because embedding models and retrieval systems work better over focused passages than entire documents — a whole 50-page PDF embedded as one vector loses most of its specific meaning.

Chunk size

Chunk size is the target length of each piece, usually measured in tokens. Smaller chunks retrieve more precisely but produce more chunks (and more embedding cost, plus more vectors to search). Larger chunks reduce chunk count but can dilute relevance by mixing multiple topics into one vector.

Chunk overlap

Overlap repeats a portion of the previous chunk at the start of the next one, so ideas that span a chunk boundary aren't split awkwardly in half. The "effective step" — how far each chunk advances past the last — is chunk size minus overlap:

If overlap is greater than or equal to chunk size, the effective step becomes zero or negative and chunking can never progress through the document — that combination is always invalid.

Picking values for your documents

The right chunk size and overlap depend on your documents, your embedding model, and how your retrieval step uses the results — see Choosing Chunk Size and Overlap for practical starting points.

Use the RAG Chunk Calculator to see how a given chunk size and overlap translate into chunk count for your document size.