RAG Architecture
Retrieval-augmented generation, or RAG, gives a language model access to external evidence at answer time. Instead of asking the model to rely only on parameters learned during training, the application retrieves relevant passages from a controlled corpus and places those passages into the prompt. The outcome is an answer that can use current, private, or domain-specific information while exposing which source passages supported it.
In this course, RAG is the bridge between model behavior and application data architecture. A useful RAG system is not just a vector database next to a chat model. It is a sequence of stages: ingestion, parsing, chunking, metadata enrichment, embedding, indexing, query transformation, retrieval, optional reranking, context assembly, generation, citation rendering, and evaluation. Each stage changes what evidence the model can see, so each stage affects answer quality.
Pipeline Mechanics
Ingestion starts with source documents such as PDFs, web pages, tickets, policies, API docs, or product records. The ingestion job records source identity, version, permissions, timestamps, and document structure. Parsing then converts files into text and layout hints. A poor parser can silently remove table headers, footnotes, or section titles; the retriever may later find the right paragraph but lose the meaning attached to it.
Chunking divides parsed content into retrievable units. Chunks should be large enough to preserve meaning and small enough to fit into prompts with other evidence. Common strategies include fixed token windows with overlap, heading-aware chunks, semantic chunks, and record-level chunks for structured data. Overlap helps when an answer spans a boundary, but excessive overlap increases storage, duplicate retrieval, and prompt cost.
Embedding maps each chunk into a vector so similar text has nearby coordinates. The index stores vectors plus metadata such as source id, page, heading, tenant id, document version, and access labels. At query time, the user question is embedded, nearest neighbors are fetched, filters remove unauthorized or irrelevant records, and a reranker may reorder candidates using a stronger but slower model. The context assembler then chooses passages, trims them to a budget, preserves source labels, and formats the evidence for the generator.
The generator receives instructions, the user question, and assembled evidence. A RAG prompt should tell the model how to use evidence, how to cite it, and what to do when evidence is insufficient. The renderer maps citation markers back to source metadata so users can inspect the underlying document. The evaluator checks retrieval recall, grounding, answer correctness, citation accuracy, latency, and cost.
API Anatomy
Most RAG systems expose the same internal objects even when libraries name them differently. A Document is the durable source. A Chunk is the indexed evidence unit. An Embedding is the vector representation of a chunk or query. A Retriever returns candidate chunks. A Reranker improves ordering. A ContextBuilder turns candidates into prompt evidence. A Generator produces the answer. Keeping these roles separate makes the pipeline easier to test because retrieval quality can be measured without calling the generator.
from dataclasses import dataclass
@dataclass(frozen=True)
class Chunk:
chunk_id: str
source: str
text: str
heading: str
def chunk_by_sentence(source: str, heading: str, text: str, size: int = 2) -> list[Chunk]:
sentences = [part.strip() for part in text.split(".") if part.strip()]
chunks: list[Chunk] = []
for start in range(0, len(sentences), size):
body = ". ".join(sentences[start:start + size]) + "."
chunks.append(Chunk(f"{source}:{start // size}", source, body, heading))
return chunks
chunks = chunk_by_sentence("handbook", "Refunds", "Refunds take five days. Premium plans include priority review. Trials are not refundable.")
print(chunks[0].text)This first example performs deterministic heading-aware chunking. The expected output is Refunds take five days. Premium plans include priority review.. Notice that the source and heading travel with the text. Without that metadata, a later citation could mention the answer but not the location that supports it.
Progressive Retrieval Examples
The second example uses a tiny lexical retriever so the mechanics are visible. Production systems often use embedding similarity, hybrid search, or both, but the core idea is the same: score candidate chunks against the query, return the best evidence, and preserve identifiers for citation.
from dataclasses import dataclass
@dataclass(frozen=True)
class Chunk:
chunk_id: str
source: str
text: str
heading: str
def retrieve(query: str, chunks: list[Chunk], top_k: int = 2) -> list[Chunk]:
query_terms = set(query.lower().split())
scored = []
for chunk in chunks:
text_terms = set(chunk.text.lower().replace(".", "").split())
score = len(query_terms & text_terms)
scored.append((score, chunk.chunk_id, chunk))
scored.sort(reverse=True)
return [chunk for score, _chunk_id, chunk in scored[:top_k] if score > 0]
corpus = [
Chunk("a", "handbook", "Refunds take five business days after approval.", "Refunds"),
Chunk("b", "handbook", "Enterprise customers receive quarterly security reviews.", "Security"),
Chunk("c", "pricing", "Premium plans include priority refund review.", "Plans"),
]
for item in retrieve("premium refund", corpus):
print(item.chunk_id, item.heading)The expected output is c Plans followed by a Refunds. The premium plan chunk scores highest because it matches both query terms after simple normalization. The refund policy chunk still appears because it supplies timing. This example also shows why retrieval alone is not answering: it finds evidence but does not yet compose a response.
The third example assembles a grounded prompt and renders a deterministic answer from retrieved chunks. A real generator would be a model call, but this small function makes the contract explicit: answer only from evidence and include citation ids.
from dataclasses import dataclass
@dataclass(frozen=True)
class Chunk:
chunk_id: str
source: str
text: str
heading: str
def build_context(chunks: list[Chunk], max_chars: int = 180) -> str:
lines: list[str] = []
used = 0
for chunk in chunks:
line = f"[{chunk.chunk_id}] {chunk.heading}: {chunk.text}"
if used + len(line) > max_chars:
continue
lines.append(line)
used += len(line)
return "\n".join(lines)
def answer_from_context(question: str, context: str) -> str:
if not context:
return "I do not have enough retrieved evidence to answer."
if "five business days" in context.lower() and "priority refund review" in context.lower():
return "Premium refunds receive priority review, and approved refunds take five business days. [a]"
return "The retrieved evidence is insufficient for a complete answer."
chunks = [
Chunk("c", "pricing", "Premium plans include priority refund review.", "Plans"),
Chunk("a", "handbook", "Refunds take five business days after approval.", "Refunds"),
]
context = build_context(chunks)
print(answer_from_context("How long do premium refunds take?", context))The expected output is Premium refunds receive priority review, and approved refunds take five business days. [a]. The important behavior is not the string template; it is the separation between evidence selection and answer generation. If no evidence arrives, the system declines instead of inventing a policy.
Design Choices
Chunk size is the first major trade-off. Small chunks improve pinpoint retrieval and reduce prompt waste, but they can omit definitions, exceptions, or table context. Large chunks preserve context but may crowd out other evidence and confuse the generator with unrelated details. A practical starting point is to chunk by document structure, then measure retrieval recall on real questions.
Search strategy is another trade-off. Vector search handles paraphrase well: a query for reimbursement may find refund policies. Keyword search handles exact identifiers, error codes, SKUs, and names. Hybrid search combines both and is often stronger for enterprise corpora. Reranking adds latency and cost but can improve precision when the first-stage retriever returns many plausible chunks.
Context assembly decides what the model actually sees. Sorting only by similarity can put duplicates ahead of complementary evidence. Sorting by source diversity, recency, authority, or section priority may produce better answers. The assembler should also remove chunks the user cannot access, deduplicate near-identical text, and retain enough metadata for citations.
Failure Modes And Troubleshooting
Symptom: answers cite the right document but state the wrong exception. Cause: chunking separated a rule from its exception or table header. Diagnostic steps: inspect the retrieved chunk text, source page, neighboring chunks, and parser output. Correction: use heading-aware or table-aware chunking, include sibling chunks, or add overlap only around sections where rules span boundaries.
Symptom: answers are fluent but uncited. Cause: the prompt allows general knowledge, the context builder dropped source ids, or citation rendering cannot map markers to metadata. Diagnostic steps: log the assembled prompt without secrets, verify each context block has a stable id, and test an answer where only one source supports the claim. Correction: require citation markers for factual claims and reject or regenerate answers that contain unsupported assertions.
Symptom: relevant documents are never retrieved. Cause: stale embeddings, a missing tenant filter, poor OCR, language mismatch, or an embedding model change without reindexing. Diagnostic steps: search by exact source id, compare raw parsed text with the original document, run keyword search, and check index timestamps. Correction: reprocess affected documents, add hybrid search, fix metadata filters, or maintain separate indexes for incompatible embedding spaces.
Security, Performance, And Reliability
RAG changes the security model because retrieved text becomes prompt input. Documents may contain prompt injection such as instructions to ignore policy or reveal secrets. The system should treat retrieved text as evidence, not instruction. Put developer instructions outside the evidence block, label evidence clearly, and avoid giving the model access to tools unless the tool layer independently checks authorization.
Performance depends on ingestion latency, index freshness, retrieval latency, reranking cost, prompt size, and generation length. Caching embeddings and retrieval results can help, but cache keys must include document version, user permissions, and query transformation settings. Reliability depends on graceful degradation: if reranking fails, the system may fall back to first-stage retrieval; if retrieval returns no evidence, it should explain that the corpus did not contain support.
Hands-On Lab
Prerequisites: Python 3, a terminal, and no external services. The lab builds a miniature RAG pipeline with local chunks, lexical retrieval, context assembly, and grounded answering. Step 1: create a temporary file named mini_rag.py. Step 2: paste the code below. Step 3: run python mini_rag.py. Step 4: verify that the first answer cites sources and the second answer declines because the corpus lacks evidence. Cleanup is simply deleting the temporary file.
from dataclasses import dataclass
@dataclass(frozen=True)
class Chunk:
chunk_id: str
text: str
def retrieve(query: str, corpus: list[Chunk]) -> list[Chunk]:
terms = set(query.lower().replace("?", "").split())
ranked = []
for chunk in corpus:
words = set(chunk.text.lower().replace(".", "").split())
ranked.append((len(terms & words), chunk))
ranked.sort(key=lambda item: item[0], reverse=True)
return [chunk for score, chunk in ranked if score > 0][:2]
def answer(question: str, evidence: list[Chunk]) -> str:
joined = " ".join(chunk.text for chunk in evidence).lower()
if "password resets expire after 30 minutes" in joined:
ids = " ".join(f"[{chunk.chunk_id}]" for chunk in evidence)
return f"Password reset links expire after 30 minutes. {ids}"
return "I do not have enough retrieved evidence to answer."
corpus = [
Chunk("policy-1", "Password resets expire after 30 minutes."),
Chunk("policy-2", "Administrators can revoke active sessions."),
]
print(answer("When do password resets expire?", retrieve("When do password resets expire?", corpus)))
print(answer("What is the vacation policy?", retrieve("What is the vacation policy?", corpus)))Verification output should be Password reset links expire after 30 minutes. [policy-1] and then I do not have enough retrieved evidence to answer.. If the first answer declines, inspect token normalization and confirm the corpus sentence is unchanged. If the second answer invents a vacation policy, the answer function is using knowledge outside retrieved evidence and must be tightened.
Assessment
- A support bot retrieves a chunk saying refunds take five days and another saying chargebacks are never refundable. What context assembly rule would help the generator avoid merging these into one incorrect policy?
- Your vector search misses product codes such as
XR-17B, but keyword search finds them. Design a hybrid retrieval strategy and explain how you would rank combined results. - A user receives an answer citing a document they are not allowed to open. Which metadata should have been checked, and at what stages?
- How would you evaluate whether smaller chunks improved the system? Name the dataset, metric, and one failure case you would inspect manually.
- When should a RAG answer decline rather than answer, and how would you test that behavior automatically?
Summary
RAG architecture is an evidence pipeline wrapped around a generator. Ingestion and chunking determine what can be found. Embeddings, filters, retrieval, and reranking determine what evidence is selected. Context assembly determines what the model can use. Generation and citation rendering determine whether users can trust and inspect the result. The best RAG systems are engineered stage by stage, measured with representative questions, and designed to decline when the retrieved evidence is not enough.
