Building a Basic RAG Chain

Retrieval-Augmented Generation (RAG) lets a language model answer questions using your own documents instead of only what it learned in training. The idea is simple: first retrieve the most relevant pieces of text, then paste them into the prompt so the model can answer from them. In this lesson you will wire these steps into one LangChain chain using LCEL (LangChain Expression Language), the | pipe syntax.

Overview: How a RAG chain works

A basic RAG chain has four parts connected in a line:

1. Retriever: takes the user’s question, searches a vector store, and returns the top k matching Document objects.
2. Formatter: turns that list of documents into one plain string (the “context”).
3. Prompt: a template with two slots, {context} and {question}, that tells the model to answer only from the context.
4. Model + parser: the chat model writes the answer, and StrOutputParser converts the model’s message into a plain Python string.

The key trick is that the question must travel along two paths. One path goes through the retriever to find context. The other path carries the original question unchanged to the prompt. LangChain handles this with a dictionary at the start of the chain. A dictionary of runnables is automatically converted into a RunnableParallel: it sends the same input to every value and collects the results under the same keys.

So when you call chain.invoke("What is the refund policy?"), this happens:

• The string goes to retriever | format_docs and to RunnablePassthrough() at the same time.
• The result is a dict: {"context": "...text...", "question": "What is the refund policy?"}.
• That dict is the input to the prompt, which fills both slots and produces chat messages.
• The model receives the messages and returns an AIMessage; the parser extracts its text.

Every piece is a Runnable, so the whole chain also supports .batch(), .stream() and .ainvoke() for free.

Syntax and key concepts

chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

Example 1: A fully offline RAG chain

You do not need an API key to learn the plumbing. This example uses DeterministicFakeEmbedding (it turns text into repeatable pseudo-random vectors), InMemoryVectorStore and FakeListChatModel (it returns canned replies). Because the fake embeddings carry no real meaning, the retrieved documents are not truly the most relevant ones here. The goal is to see the structure working.

from langchain_core.documents import Document
from langchain_core.embeddings import DeterministicFakeEmbedding
from langchain_core.language_models.fake_chat_models import FakeListChatModel
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.vectorstores import InMemoryVectorStore

docs = [
    Document(page_content="Refunds are accepted within 30 days of purchase.", metadata={"source": "policy.txt"}),
    Document(page_content="Standard shipping takes 3 to 5 business days.", metadata={"source": "shipping.txt"}),
    Document(page_content="Support is available Monday to Friday, 9am to 6pm IST.", metadata={"source": "support.txt"}),
]

store = InMemoryVectorStore(DeterministicFakeEmbedding(size=256))
store.add_documents(docs)
retriever = store.as_retriever(search_kwargs={"k": 2})


def format_docs(documents):
    return "\n\n".join(d.page_content for d in documents)


prompt = ChatPromptTemplate.from_template(
    "Answer the question using only the context below.\n"
    "If the answer is not in the context, say you do not know.\n\n"
    "Context:\n{context}\n\n"
    "Question: {question}"
)

llm = FakeListChatModel(responses=["Refunds are accepted within 30 days."])

chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

print(chain.invoke("What is the refund policy?"))

Output:

Refunds are accepted within 30 days.

The printed answer is the fake model’s canned reply, but everything before it ran for real: the question was embedded, the store was searched, two documents were joined into a string, and the prompt was filled. The same chain shape works unchanged when you swap in real components.

Example 2: The same chain with a real provider

To get real answers, replace the fake embeddings and model with provider classes. Here we use the langchain-openai partner package. The API key is read from the OPENAI_API_KEY environment variable; never hard-code it. Model names change often, so check your provider’s documentation for the current names.

import os

from langchain_core.documents import Document
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

assert os.environ.get("OPENAI_API_KEY"), "Set OPENAI_API_KEY first"

docs = [
    Document(page_content="Refunds are accepted within 30 days of purchase."),
    Document(page_content="Standard shipping takes 3 to 5 business days."),
    Document(page_content="Support is available Monday to Friday, 9am to 6pm IST."),
]

store = InMemoryVectorStore.from_documents(docs, OpenAIEmbeddings(model="text-embedding-3-small"))
retriever = store.as_retriever(search_kwargs={"k": 2})

prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer only from the context. If it is missing, say you do not know."),
    ("human", "Context:\n{context}\n\nQuestion: {question}"),
])
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)


def format_docs(documents):
    return "\n\n".join(d.page_content for d in documents)


chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

print(chain.invoke("How long does shipping take?"))

Output (illustrative; wording varies by model):

Standard shipping takes 3 to 5 business days.

Two details matter. First, temperature=0 makes the model more deterministic, which suits factual question answering. Second, the system message sets the rule “answer only from the context”. This is your main defence against hallucination (the model inventing facts).

Example 3: Returning the sources with the answer

Users trust answers more when they can see where they came from. A plain chain returns only a string. To keep the retrieved documents, build a RunnableParallel first and then use .assign() to add the answer as an extra key.

from langchain_core.documents import Document
from langchain_core.embeddings import DeterministicFakeEmbedding
from langchain_core.language_models.fake_chat_models import FakeListChatModel
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnableParallel, RunnablePassthrough
from langchain_core.vectorstores import InMemoryVectorStore

docs = [
    Document(page_content="Refunds are accepted within 30 days of purchase.", metadata={"source": "policy.txt"}),
    Document(page_content="Standard shipping takes 3 to 5 business days.", metadata={"source": "shipping.txt"}),
    Document(page_content="Support is available Monday to Friday, 9am to 6pm IST.", metadata={"source": "support.txt"}),
]
store = InMemoryVectorStore(DeterministicFakeEmbedding(size=256))
store.add_documents(docs)
retriever = store.as_retriever(search_kwargs={"k": 2})

prompt = ChatPromptTemplate.from_template("Context:\n{context}\n\nQuestion: {question}")
llm = FakeListChatModel(responses=["Refunds are accepted within 30 days."])


def to_prompt_input(state):
    text = "\n\n".join(d.page_content for d in state["context"])
    return {"context": text, "question": state["question"]}


answer_chain = to_prompt_input | prompt | llm | StrOutputParser()

rag = RunnableParallel(context=retriever, question=RunnablePassthrough()).assign(
    answer=answer_chain
)

result = rag.invoke("What is the refund policy?")
print(sorted(result.keys()))
print(len(result["context"]))
print(result["answer"])

Output:

['answer', 'context', 'question']
2
Refunds are accepted within 30 days.

Now result["context"] holds the list of Document objects, so you can show metadata["source"] as citations in your UI. Notice that formatting moved into to_prompt_input: the retriever’s raw documents are kept in state, and only the prompt input gets the joined string.

Under the hood: step by step

1. chain.invoke(question) receives a string.
2. The RunnableParallel (built from your dict) runs both branches with the same string. The retriever calls the embedding model on the question, then runs a similarity search in the vector store.
3. The store returns the k nearest documents. format_docs joins their text.
4. The dict {"context": ..., "question": ...} reaches prompt.invoke(), which returns a prompt value (messages).
5. The chat model sends the messages to the LLM and returns an AIMessage.
6. StrOutputParser returns message.content as a string.

You can debug any stage on its own. For example, retriever.invoke("refund") shows what was retrieved, and prompt.invoke({...}) shows the exact final prompt. When an answer is wrong, check the retrieval first: most RAG failures are retrieval failures, not model failures.

Common Mistakes

1. Forgetting to format the documents

chain = (
    {"context": retriever, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

This runs without an error, but {context} becomes the Python representation of a list, like [Document(id=..., metadata={...}, page_content='...')]. The model sees noisy metadata text instead of clean passages, and answers get worse. Fix: pipe the retriever into a formatter, retriever | format_docs.

2. Building the index inside the request path

def answer(question):
    store = InMemoryVectorStore.from_documents(docs, embeddings)
    return build_chain(store).invoke(question)

This re-embeds every document on every question. It is slow and, with a paid embedding API, expensive. Fix: build (or load) the vector store once at startup, create the chain once, and call chain.invoke() per question.

3. A prompt that does not forbid guessing

If the prompt only says “Answer the question”, the model may fall back on its own memory when the context lacks the answer. Always include an instruction such as “If the answer is not in the context, say you do not know”, and test it with a question your documents cannot answer.

Best Practices

• Create the vector store and chain once, and reuse them across requests.
• Tell the model to use only the supplied context and to admit when it does not know.
• Use temperature=0 (or a low value) for factual answers.
• Return source documents alongside the answer so users can verify claims.
• Tune k: too small misses facts, too large adds noise and cost. Start with 3 to 5 and measure.
• Put separators between retrieved passages, and consider including metadata["source"] in the formatted text.
• Keep API keys in environment variables, never in source code.
• Test each stage (retriever, prompt, model) separately before blaming the whole chain.
• Use the offline fake classes in unit tests so your tests are fast, free and repeatable.

Practice Exercises

1. Add a citation line. Change format_docs in Example 1 so each passage is followed by (source: policy.txt), using metadata["source"]. Print the formatted context with retriever.invoke(...) to confirm.
2. Change k. In Example 3, set k to 3 and check that len(result["context"]) becomes 3. What happens if k is larger than the number of stored documents?
3. Handle unknown questions. Using a real model, ask a question your documents cannot answer. Adjust the system prompt until the chain reliably replies that it does not know. Hint: be explicit about the exact phrase to use.

Summary

• A basic RAG chain is retriever, formatter, prompt, model and parser joined with |.
• A dict at the start of the chain becomes a RunnableParallel, sending the question down two paths.
• RunnablePassthrough() keeps the original question for the prompt.
• Retrieved Document lists must be formatted into a string before the prompt.
• Use RunnableParallel(...).assign(...) to return sources with the answer.
• Fake embeddings and fake chat models let you build and test the structure offline.
• Build the index once, instruct the model to stay within the context, and debug retrieval first.