Chunking turns source documents into retrieval units that an embedding model, vector index, and generator can use reliably. The outcome is not merely smaller text. A good preparation pipeline produces chunks that each carry a coherent idea, a traceable source location, and the same access rules as the original document.
In the Embeddings and Retrieval section, chunking sits between raw content ingestion and embedding generation. If this step is careless, the retriever may return fragments that omit the answer, merge unrelated policies, lose permissions, or cite the wrong source. If this step is deliberate, retrieval-augmented generation can provide evidence that is specific enough for the model to use and small enough to fit in context.
Purpose And Outcome
The practical purpose is to convert documents such as PDFs, web pages, tickets, transcripts, runbooks, and policy pages into records shaped like chunk_id, text, metadata, and sometimes embedding. The expected outcome is a collection where each chunk can stand on its own for retrieval while still pointing back to its parent document, section, page, author, timestamp, and access-control list.
A retrieval system does not understand a document as a human reader does. It compares query vectors against chunk vectors. That means chunk boundaries define what can be found. A chunk that is too small may match the query but lack the answer. A chunk that is too large may contain the answer but dilute the embedding with unrelated material. Preparation is the engineering work of choosing a useful middle ground.
How Chunking Works Internally
A typical pipeline has five stages. First, extraction converts each source into text plus structure: headings, pages, tables, list items, timestamps, and links. Second, normalization removes repeated headers, broken hyphenation, navigation chrome, or scanner artifacts without destroying meaningful wording. Third, segmentation finds candidate boundaries such as paragraphs, Markdown headings, sentences, transcript turns, or table rows. Fourth, packing combines candidates until a size budget is reached. Fifth, enrichment attaches metadata and creates stable identifiers before embeddings are computed.
The size budget is usually described in tokens, because model context is token-based. Character counts are cheaper to compute and useful for demonstrations, but production systems often use the tokenizer for the embedding and generation models. The chunk budget should leave room for retrieved neighbors, user instructions, citations, and the final answer.
Overlap is the deliberate repetition of text between adjacent chunks. It protects facts that straddle a boundary: for example, a heading in one paragraph and the rule in the next. Too little overlap loses context; too much overlap increases storage, indexing cost, duplicate retrieval, and prompt bloat. Many systems use overlap by sentences, paragraphs, or tokens rather than by raw characters because semantic units matter.
Record Anatomy
A prepared chunk normally includes a stable chunk_id, the chunk text, a source_id, an ordinal index, offsets or page numbers, and security metadata. The chunk_id should not depend on accidental ingestion order alone. A common pattern combines a source identifier, section path, and chunk index. For documents that change often, store a source revision or content hash so old embeddings can be invalidated cleanly.
Metadata is not decoration. It controls filtering, citation rendering, debugging, and deletion. Access metadata must be copied from the source or computed from the source policy before indexing. A retriever should filter by permissions before results are sent to a generator. Adding security labels after retrieval is too late, because unauthorized text may already have entered the prompt.
Example 1: Sentence Windows
This first example chunks a short support document into two-sentence windows with one-sentence overlap. The overlap keeps the connecting sentence available in both neighboring chunks.
import re
from dataclasses import dataclass
@dataclass(frozen=True)
class DocChunk:
chunk_id: str
text: str
start_sentence: int
end_sentence: int
def sentences(text: str) -> list[str]:
return [part.strip() for part in re.split(r'(?<=[.!?])\s+', text.strip()) if part.strip()]
def chunk_by_sentence_window(doc_id: str, text: str, window: int = 2, overlap: int = 1) -> list[DocChunk]:
if window < 1 or overlap >= window:
raise ValueError('window must be positive and overlap smaller than window')
parts = sentences(text)
chunks = []
step = window - overlap
for start in range(0, len(parts), step):
selected = parts[start:start + window]
if not selected:
break
chunks.append(DocChunk(f'{doc_id}:{len(chunks)}', ' '.join(selected), start, start + len(selected) - 1))
if start + window >= len(parts):
break
return chunks
document = 'Project Atlas stores customer support tickets. Tickets are exported daily. Each ticket has an account_id and visibility label. Password reset tickets must stay visible only to support_staff.'
for chunk in chunk_by_sentence_window('atlas', document):
print(f'{chunk.chunk_id} [{chunk.start_sentence}-{chunk.end_sentence}] {chunk.text}')The deterministic output is three chunks. Sentence 1 appears in chunks 0 and 1, and sentence 2 appears in chunks 1 and 2. That is useful when a query mentions exports and account identifiers, because those details are not separated by a hard boundary.
Example 2: Metadata Preservation
The second example shows the part many prototypes skip: every chunk inherits source metadata and access labels. Without this, citations and permission filters become unreliable.
import re
def sentence_chunks(doc: dict[str, str], window: int = 2) -> list[dict[str, object]]:
parts = [p.strip() for p in re.split(r'(?<=[.!?])\s+', doc['text']) if p.strip()]
records = []
for index in range(0, len(parts), window):
records.append({
'chunk_id': '%s:%s' % (doc['doc_id'], index // window),
'text': ' '.join(parts[index:index + window]),
'metadata': {
'source_id': doc['doc_id'],
'title': doc['title'],
'acl': tuple(doc['acl'].split(',')),
'start_sentence': index,
},
})
return records
doc = {
'doc_id': 'hr-17',
'title': 'Leave Policy',
'acl': 'employees,hr',
'text': 'Full-time employees accrue leave monthly. Contractors do not accrue paid leave. HR may approve exceptions in writing.'
}
for record in sentence_chunks(doc):
meta = record['metadata']
print(record['chunk_id'], meta['title'], meta['acl'], meta['start_sentence'])The output shows two records, both labeled with the same title and access-control tuple. A retriever can now filter results to users in employees or hr, and a citation renderer can display Leave Policy instead of an anonymous text fragment.
Example 3: Diagnosing Bad Chunks
The third example checks prepared chunks before indexing. This is cheaper than discovering problems after embeddings have been generated and search quality has already degraded.
def diagnose_chunks(chunks: list[dict[str, object]], max_chars: int) -> list[str]:
problems = []
seen = set()
for chunk in chunks:
chunk_id = str(chunk.get('chunk_id', '<missing>'))
text = str(chunk.get('text', ''))
metadata = chunk.get('metadata')
if chunk_id in seen:
problems.append(f'duplicate id: {chunk_id}')
seen.add(chunk_id)
if not text.strip():
problems.append(f'empty text: {chunk_id}')
if len(text) > max_chars:
problems.append(f'oversized text: {chunk_id}')
if not isinstance(metadata, dict) or 'acl' not in metadata:
problems.append(f'missing acl: {chunk_id}')
return problems
sample = [
{'chunk_id': 'a:0', 'text': 'Short allowed text.', 'metadata': {'acl': ('team-a',)}},
{'chunk_id': 'a:1', 'text': '', 'metadata': {'source_id': 'a'}},
{'chunk_id': 'a:1', 'text': 'This chunk is too long for the configured limit.', 'metadata': {'acl': ('team-a',)}},
]
for problem in diagnose_chunks(sample, max_chars=32):
print(problem)The expected output is empty text: a:1, missing acl: a:1, duplicate id: a:1, and oversized text: a:1. These checks catch four different failure classes: extraction gaps, metadata loss, unstable identity, and size-budget violations.
Design Choices And Trade-Offs
Fixed-size chunking is simple and predictable, but it can cut through sentences, tables, or code blocks. Sentence or paragraph chunking preserves readability, but it may produce uneven sizes. Heading-aware chunking works well for manuals and policies because headings name the local topic, but it needs reliable structure from the extractor. Table-aware chunking should keep headers with rows; otherwise retrieved rows may be meaningless.
Chunk size is a recall and precision trade-off. Smaller chunks are more precise and cheaper to place in prompts, but may omit qualifying details. Larger chunks preserve context, but their embeddings can represent several topics at once. Overlap improves recall near boundaries, but creates duplicate search hits and higher indexing cost. A practical approach is to start with structure-aware chunks, measure retrieval quality on real questions, then adjust size and overlap based on missed answers and noisy results.
Failure Modes And Troubleshooting
If users see answers that quote the right document but miss the relevant exception, inspect boundary placement. The cause is often a rule split from its heading or exception paragraph. Reproduce the query, view the top retrieved chunks, and check whether the missing condition is in a neighboring chunk. Correct it by adding sentence or paragraph overlap, or by packing headings with their child content.
If search returns many nearly identical results, overlap is probably too large or duplicate source versions were indexed. Diagnose by grouping top results by source_id and text hash. Correct it by reducing overlap, deduplicating at ingestion, or collapsing adjacent hits before prompt construction.
If unauthorized content appears in answers, the likely cause is missing or late permission filtering. Check whether chunks carry acl metadata and whether the vector query includes a permission filter before results are returned. Correct it by propagating access labels during preparation, reindexing affected documents, and blocking retrieval when labels are absent.
If citations point to the wrong page, offsets were lost or changed during normalization. Compare extracted text, normalized text, and stored offsets. Correct it by storing both normalized text for embedding and source coordinates for citation, or by maintaining a mapping between normalized spans and original spans.
Security, Performance, And Reliability
Security depends on preserving document-level and section-level permissions throughout preparation. Some documents contain mixed-access sections, so a single inherited label may be insufficient. In that case, split at the permission boundary first, then chunk inside each region.
Performance depends on controlling chunk count. More chunks mean more embedding calls, larger indexes, slower ingestion, and more candidate results to rank. Reliability depends on deterministic preparation. Given the same source revision and configuration, the pipeline should produce the same chunk identifiers and text so deletions, updates, and audits are possible.
Hands-On Lab
Prerequisites: Python 3, a terminal, and one plain text document of at least six sentences. No external package is required.
- Create a temporary working directory and place the document in
source.txt. - Adapt Example 1 so it reads
source.txtinstead of the inline string. - Run it with a two-sentence window and one-sentence overlap.
- Add metadata fields for
source_id,title,acl, andstart_sentenceusing Example 2 as the pattern. - Run the diagnostic function from Example 3 with a size limit appropriate for your sample, such as 800 characters.
- Inspect the printed chunks and confirm each chunk has coherent text, a stable identifier, inherited access labels, and enough neighboring context.
Verification is concrete: no diagnostic problems should print, every chunk should have non-empty text, and the first repeated sentence should appear at the end of one chunk and the beginning of the next. Cleanup is simply removing the temporary directory. If you generated embeddings during an extended version of the lab, delete the temporary collection or namespace as well.
Assessment Exercises
- A policy page has a heading
Refund Exceptionsfollowed by five short bullets. Design chunk boundaries that keep each bullet findable without losing the heading context. - A transcript chunker returns excellent recall but fills prompts with repeated text. Explain which configuration you would inspect first and how you would measure improvement.
- A retriever returns an HR-only paragraph to a general employee. Identify the preparation metadata that should have prevented this and where filtering should occur.
- Given a document that changes daily, describe how stable chunk identifiers and source revisions help update the vector index without leaving stale results.
- Compare character-based, token-based, and structure-aware chunking for a mixed document containing prose, tables, and code examples.
Summary
Chunking and document preparation define what retrieval can find and what the generator can cite. Effective chunks are coherent, bounded, traceable, and permission-aware. The best configuration is not universal: it depends on source structure, expected questions, model context limits, security requirements, and measured retrieval quality.
