Text Splitting and Chunking Strategies

Text splitting means cutting a long document into smaller pieces called chunks. In a RAG (Retrieval-Augmented Generation) app, you embed and search chunks, not whole files. How you cut the text decides whether the right facts are found later, so chunking is one of the most important choices in a RAG pipeline.

Overview: why we split and how it works

There are three reasons to split documents:

• Context limits. An LLM can only read a limited number of tokens at once. A 300-page PDF will not fit.
• Retrieval quality. An embedding turns a text into one vector. If a chunk covers ten topics, its vector is a blurry average and matches nothing well. A focused chunk gives a sharper vector.
• Cost and speed. Sending only the three most relevant chunks to the model is cheaper and faster than sending everything.

Chunks that are too small lose context (a sentence like "It was fixed in version 2" is useless alone). Chunks that are too large dilute meaning and waste prompt space. Good chunking is a balance, and you tune it with real questions.

The recursive idea

The most-used splitter in LangChain is RecursiveCharacterTextSplitter. It takes an ordered list of separators, by default ["\n\n", "\n", " ", ""]. Internally it works like this:

1. Try to split the text on the first separator (paragraph breaks).
2. Pieces that are already smaller than chunk_size are merged back together, as many as fit.
3. Any piece still too big is split again, using the next separator (line breaks), and so on down to single words and finally single characters.

The result is that paragraphs stay together when possible, then sentences or lines, then words. Meaning is broken only when there is no other choice.

Chunk size and overlap

chunk_size is the maximum length of a chunk. By default length is counted in characters. chunk_overlap is how much text the end of one chunk shares with the start of the next. Overlap helps when an important sentence sits exactly on a boundary: it appears in both chunks, so neither loses it. Too much overlap creates duplicate content and a bigger index. A common starting point is an overlap of about 10 to 20 percent of the chunk size.

Syntax and key concepts

Splitters live in the separate package langchain-text-splitters. They need no API key and work fully offline.

pip install langchain-text-splitters

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=['\n\n', '\n', ' ', ''],
    add_start_index=True,
)
chunks = splitter.split_text(text)          # list of str
docs = splitter.split_documents(documents)  # list of Document

Examples

Example 1: Recursive splitting by paragraph

A small text with three paragraphs and a chunk size of 60 characters.

from langchain_text_splitters import RecursiveCharacterTextSplitter

text = 'LangChain splits text.\n\nEach chunk stays small.\n\nOverlap keeps context.'
splitter = RecursiveCharacterTextSplitter(chunk_size=60, chunk_overlap=0)
for chunk in splitter.split_text(text):
    print(len(chunk), repr(chunk))

Output:

47 'LangChain splits text.\n\nEach chunk stays small.'
22 'Overlap keeps context.'

The paragraphs are 22, 23 and 22 characters long. The first two fit together (22 + 2 + 23 = 47). Adding the third would make 71, which is over 60, so a new chunk starts. The splitter never cut inside a paragraph because it did not need to.

Example 2: Seeing overlap clearly

Here we use CharacterTextSplitter, which splits on one separator only. Splitting on spaces makes the overlap easy to see.

from langchain_text_splitters import CharacterTextSplitter

text = 'one two three four five six seven eight nine ten'
splitter = CharacterTextSplitter(separator=' ', chunk_size=20, chunk_overlap=8)
for i, chunk in enumerate(splitter.split_text(text)):
    print(i, chunk)

Output:

0 one two three four
1 four five six seven
2 seven eight nine ten

The word four ends chunk 0 and starts chunk 1; seven does the same for chunks 1 and 2. The splitter keeps whole words from the end of the previous chunk while their total length is at most the overlap (8 characters). Overlap is therefore approximate: it is made of whole pieces, not an exact character count.

Example 3: Splitting Documents and keeping metadata

In real pipelines you split Document objects from a loader. Metadata is copied to every chunk, and add_start_index records where each chunk began.

from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter

text = 'Alpha beta gamma delta.\n\nEpsilon zeta eta theta iota.'
doc = Document(page_content=text, metadata={'source': 'demo.txt'})

splitter = RecursiveCharacterTextSplitter(
    chunk_size=50, chunk_overlap=0, add_start_index=True
)
for d in splitter.split_documents([doc]):
    print(d.metadata, repr(d.page_content))

Output:

{'source': 'demo.txt', 'start_index': 0} 'Alpha beta gamma delta.'
{'source': 'demo.txt', 'start_index': 25} 'Epsilon zeta eta theta iota.'

The two paragraphs are 23 and 28 characters, which together with the break would be 53, over the limit of 50. The second chunk begins at character 25 of the original text (23 characters plus the two newlines). Metadata like source lets you show citations later.

Example 4: Structure-aware splitting for Markdown

For Markdown, splitting on headers keeps each section together and stores the headings as metadata.

from langchain_text_splitters import MarkdownHeaderTextSplitter

md = '# Guide\nIntro line.\n## Install\nRun pip install.\n## Usage\nCall the function.'
splitter = MarkdownHeaderTextSplitter(headers_to_split_on=[('#', 'h1'), ('##', 'h2')])
for doc in splitter.split_text(md):
    print(doc.metadata, repr(doc.page_content))

Output:

{'h1': 'Guide'} 'Intro line.'
{'h1': 'Guide', 'h2': 'Install'} 'Run pip install.'
{'h1': 'Guide', 'h2': 'Usage'} 'Call the function.'

The header text is moved into metadata, so the retriever can filter by section or show it to the user. A common pattern is to run this splitter first, then run a RecursiveCharacterTextSplitter on the results to cap very long sections.

Under the hood: choosing a length function

Models count tokens, not characters. A token is a piece of a word; English averages roughly four characters per token, but this varies by language and tokenizer. If you must respect a token limit (for example, an embedding model's input limit), measure chunks in tokens. The from_tiktoken_encoder constructor does this and needs the tiktoken package.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    chunk_size=256, chunk_overlap=32
)
chunks = splitter.split_text('Your long document text goes here. ' * 100)
print(len(chunks))

Here chunk_size=256 means 256 tokens. The number printed depends on the tokenizer, so it is not shown. Other splitters exist for code (RecursiveCharacterTextSplitter.from_language), HTML, JSON and sentence-based splitting. There are also experimental semantic splitters that cut where the meaning changes. Check the current LangChain documentation for the full list.

Common Mistakes

1. Overlap larger than chunk size

from langchain_text_splitters import RecursiveCharacterTextSplitter

# Wrong: overlap is bigger than the chunk
splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=150)
# ValueError: Got a larger chunk overlap (150) than chunk size (100), should be smaller.

Overlap is a part of a chunk, so it cannot be bigger than the chunk. Fix: use something like chunk_size=100, chunk_overlap=20.

2. Using split_text and losing metadata

pieces = splitter.split_text(doc.page_content)   # plain strings, source is lost
docs = splitter.split_documents([doc])           # Documents keep the metadata

With split_text you get bare strings. Later you cannot say which file or page an answer came from. Prefer split_documents (or create_documents with a metadatas list) when you index real files.

3. Choosing sizes without testing

Copying chunk_size=1000 from a tutorial and never checking results is a very common error. Different content needs different sizes: FAQ answers want small chunks, legal contracts want larger ones. Always inspect a sample of chunks and test with real questions.

4. Counting characters when the limit is in tokens

A 2000-character chunk may exceed a model's token limit for non-English text or code. Use a token-based length function when the limit matters.

Best Practices

• Start with RecursiveCharacterTextSplitter; it is a strong default for plain prose.
• Begin with a moderate size (a few hundred to about a thousand characters) and an overlap of 10 to 20 percent, then tune using retrieval tests.
• Use structure-aware splitters (Markdown, HTML, code) when the format has natural boundaries.
• Turn on add_start_index and keep rich metadata: source, page, section title.
• Print and read a sample of chunks before embedding thousands of them. Cutting mid-sentence is easy to spot by eye.
• Keep the splitting settings in one config so you can re-index consistently when you change them. Changing chunking means you must re-embed.
• Do not over-engineer: measure answer quality first, then try fancier strategies only if needed.

Practice Exercises

1. Split a text of about five paragraphs with chunk_size=80 and chunk_overlap=0. Print each chunk's length and confirm none is above 80. Hint: use len(chunk).
2. Take the word-splitting example and change the overlap to 0, then to 12. Predict the chunks before running, then compare.
3. Write a Markdown string with three ## sections. Split it with MarkdownHeaderTextSplitter, then pass the result through RecursiveCharacterTextSplitter.split_documents with a small chunk_size. Check that the header metadata is kept on the smaller chunks.

Summary

• Chunking makes documents fit model limits and makes retrieval more precise.
• RecursiveCharacterTextSplitter tries paragraphs, then lines, then words, then characters, keeping meaning together.
• chunk_size is the maximum size; chunk_overlap shares text across boundaries and must not exceed the size.
• Use split_documents to keep metadata, and add_start_index to track position.
• Use Markdown or code splitters for structured content, and token-based lengths when token limits matter.
• Tune chunking with real questions, and re-index whenever you change it.