Conversational RAG with Chat History

A basic RAG chain answers one question at a time. Real users do not talk like that: they ask “What is Chroma?” and then “Is it free?”. The word “it” means nothing to a vector store. Conversational RAG adds chat history and a question-rewriting step so that follow-up questions work as people expect.

Overview: how conversational RAG works

RAG means Retrieval-Augmented Generation: you search your documents for relevant text and give that text to a language model (LLM) so it can answer using your data. In a single-turn chain the flow is: question, retriever, prompt, LLM, answer.

When you add a conversation, two problems appear.

1. Retrieval problem. The retriever sees only the text you give it. A follow-up such as “Is it free?” has no topic words. The embedding of that sentence is close to many unrelated chunks, so retrieval returns poor results.
2. Generation problem. The LLM is stateless. It does not remember the previous turn unless you send the earlier messages again in the prompt.

The standard solution has three parts:

• Chat history: a list of messages (human and AI) saved per conversation.
• Condense step (also called question rewriting): if there is history, an LLM turns the latest question plus the history into a standalone question. “Is it free?” becomes “Is Chroma free?”.
• Answer step: retrieve with the standalone question, then build a prompt containing the retrieved context, the history, and the original question.

Notice that the history is used twice: once to rewrite the query, and once to give the answering model conversational context. Retrieval itself never sees the history directly; it only sees the rewritten query.

Key concepts and building blocks

The general shape of the chain with LCEL (LangChain Expression Language, the | operator) is:

input: {question, chat_history}
  -> standalone = condense(question, chat_history)   # skip if history is empty
  -> context    = retriever(standalone)
  -> prompt(context, chat_history, question) -> LLM -> text

Each arrow adds a key to the dictionary that flows through the chain, so later steps can use earlier results.

Examples

Example 1: A chat prompt with a history slot

First we look at how history enters a prompt. This needs no API key because we only format the prompt and do not call a model.

from langchain_core.messages import AIMessage, HumanMessage
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

prompt = ChatPromptTemplate.from_messages([
    ('system', 'Answer using only this context: {context}'),
    MessagesPlaceholder('chat_history'),
    ('human', '{question}'),
])

history = [
    HumanMessage(content='What is Chroma?'),
    AIMessage(content='Chroma is an open-source vector database.'),
]

messages = prompt.invoke({
    'context': 'Chroma stores embeddings.',
    'chat_history': history,
    'question': 'Is it free?',
}).to_messages()

for m in messages:
    print(f'{m.type}: {m.content}')

Output:

system: Answer using only this context: Chroma stores embeddings.
human: What is Chroma?
ai: Chroma is an open-source vector database.
human: Is it free?

The placeholder expanded into the two earlier messages, in order, between the system message and the new question. The LLM now sees the whole conversation. Note that the message types print as human and ai.

Example 2: A complete offline conversational RAG flow

To learn the logic without an API key, we replace the LLM with plain Python functions and the vector store with a keyword-overlap retriever. The structure (condense, retrieve, answer) is identical to a real chain, and the output is fully deterministic.

from langchain_core.documents import Document
from langchain_core.messages import AIMessage, HumanMessage
from langchain_core.runnables import RunnableBranch, RunnableLambda, RunnablePassthrough

DOCS = [
    Document(page_content='Chroma is an open-source vector database for embeddings.'),
    Document(page_content='FAISS is a library for fast similarity search.'),
    Document(page_content='Chroma is free to run locally inside your Python process.'),
]

def words(text):
    return {w.lower().strip('?.,') for w in text.split()}

def retrieve(query):
    q = words(query)
    scored = [(len(q & words(d.page_content)), d) for d in DOCS]
    scored.sort(key=lambda pair: pair[0], reverse=True)
    return [d for score, d in scored[:1] if score > 0]

def rewrite(inputs):
    last = [m for m in inputs['chat_history'] if isinstance(m, HumanMessage)][-1]
    topic = last.content.rstrip('?').split()[-1]
    tokens = inputs['question'].rstrip('?').split()
    return ' '.join(topic if t.lower() == 'it' else t for t in tokens) + '?'

condense = RunnableBranch(
    (lambda x: not x['chat_history'], lambda x: x['question']),
    RunnableLambda(rewrite),
)

chain = (
    RunnablePassthrough.assign(standalone=condense)
    | RunnablePassthrough.assign(
        context=lambda x: ' '.join(d.page_content for d in retrieve(x['standalone']))
    )
    | RunnableLambda(
        lambda x: {'standalone': x['standalone'], 'answer': 'Based on the docs: ' + x['context']}
    )
)

history = []
for q in ['What is Chroma?', 'Is it free?']:
    result = chain.invoke({'question': q, 'chat_history': history})
    print('Standalone:', result['standalone'])
    print('Answer:', result['answer'])
    history.extend([HumanMessage(content=q), AIMessage(content=result['answer'])])

Output:

Standalone: What is Chroma?
Answer: Based on the docs: Chroma is an open-source vector database for embeddings.
Standalone: Is Chroma free?
Answer: Based on the docs: Chroma is free to run locally inside your Python process.

On turn one the history is empty, so RunnableBranch passes the question through unchanged. On turn two the history exists, so rewrite replaces “it” with the topic of the last human message. The retriever then receives “Is Chroma free?” and picks the right chunk. If we had searched with “Is it free?” the best match would have been the FAISS chunk, because only the word “is” overlaps. In a real system an LLM does the rewriting, which handles far more cases than a word swap.

Example 3: The real thing with an LLM and managed history

This version uses a real chat model and embeddings. It needs a provider package and an API key read from the environment, so the output below is illustrative. For an offline run you can swap in DeterministicFakeEmbedding from langchain_core.embeddings, but retrieval quality will then be meaningless.

import os
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.runnables import RunnableLambda, RunnablePassthrough
from langchain_core.runnables.history import RunnableWithMessageHistory
from langchain_core.vectorstores import InMemoryVectorStore

# Reads OPENAI_API_KEY from the environment; CHAT_MODEL is a model name you choose.
llm = ChatOpenAI(model=os.environ['CHAT_MODEL'])
store = InMemoryVectorStore.from_texts(
    ['Chroma is an open-source vector database.', 'FAISS is a similarity search library.'],
    OpenAIEmbeddings(),
)
retriever = store.as_retriever(search_kwargs={'k': 2})

condense_prompt = ChatPromptTemplate.from_messages([
    ('system', 'Rewrite the latest question as a standalone question. Return only the question.'),
    MessagesPlaceholder('chat_history'),
    ('human', '{question}'),
])
answer_prompt = ChatPromptTemplate.from_messages([
    ('system', 'Answer using only the context. If it is missing, say you do not know. Context: {context}'),
    MessagesPlaceholder('chat_history'),
    ('human', '{question}'),
])
condense_chain = condense_prompt | llm | StrOutputParser()

def standalone(x):
    if not x['chat_history']:
        return x['question']
    return condense_chain.invoke(x)

def format_docs(docs):
    return ' '.join(d.page_content for d in docs)

rag = (
    RunnablePassthrough.assign(standalone=RunnableLambda(standalone))
    | RunnablePassthrough.assign(context=lambda x: format_docs(retriever.invoke(x['standalone'])))
    | answer_prompt
    | llm
    | StrOutputParser()
)

sessions = {}

def get_history(session_id):
    if session_id not in sessions:
        sessions[session_id] = InMemoryChatMessageHistory()
    return sessions[session_id]

chat = RunnableWithMessageHistory(
    rag,
    get_history,
    input_messages_key='question',
    history_messages_key='chat_history',
)

cfg = {'configurable': {'session_id': 'user-1'}}
print(chat.invoke({'question': 'What is Chroma?'}, config=cfg))
print(chat.invoke({'question': 'Is it open source?'}, config=cfg))

Output (illustrative):

Chroma is an open-source vector database.
Yes, Chroma is open source.

You never pass chat_history yourself. RunnableWithMessageHistory loads the messages for the given session_id, injects them under the key chat_history, runs the chain, and then saves the new human question and the AI answer back to the store.

Under the hood: one turn, step by step

1. You call chat.invoke({'question': ...}, config=...) with a session id.
2. The wrapper calls get_history(session_id) and reads the stored messages.
3. It builds the dictionary {question, chat_history} and runs the inner chain.
4. The standalone step checks whether history is empty. If it is, no LLM call is spent. If not, the condense chain makes one extra LLM call.
5. The retriever embeds the standalone question and returns the top-k chunks from the vector store.
6. The answer prompt receives context, history and the original question; the LLM produces the answer.
7. The wrapper appends the human question and the AI answer to the history, ready for the next turn.

So each follow-up turn costs up to two LLM calls (rewrite and answer) plus one embedding call. Keep this in mind for latency and cost.

Why the original question still goes to the answer prompt

The standalone question is only for retrieval. The answer prompt keeps the user’s real words and the history, so the tone and references of the conversation are preserved.

Agents as an alternative

Another design lets an agent decide when to retrieve, by exposing the retriever as a tool. With LangGraph, a checkpointer saves the conversation state per thread. This costs flexibility in return for less predictable behaviour; the chain above always retrieves exactly once per turn, which is easier to test and debug. Check the current LangGraph documentation for the exact agent and checkpointer APIs.

Common Mistakes

Mistake 1: Retrieving with the raw follow-up question

docs = retriever.invoke(x['question'])   # 'Is it open source?'

Why it is wrong: the sentence has no subject, so the embedding does not point to Chroma. The retriever returns weak or random chunks, and the LLM either hallucinates or says it does not know.

docs = retriever.invoke(x['standalone'])   # 'Is Chroma open source?'

Always search with the rewritten question.

Mistake 2: Sending unlimited history

history = get_history('user-1').messages   # every message, forever
answer_prompt.invoke({'context': ctx, 'chat_history': history, 'question': q})

Why it is wrong: a long chat will eventually exceed the model’s context window, and cost and latency grow with each turn. Trim the history first:

from langchain_core.messages import trim_messages

recent = trim_messages(
    history,
    max_tokens=6,
    token_counter=len,
    strategy='last',
    start_on='human',
)

Here token_counter=len counts messages instead of tokens, so this keeps roughly the last six messages. For production you can pass a real token counter such as the chat model itself.

Mistake 3: One shared history for all users

If get_history ignores session_id and returns one global object, users will see each other’s context. Always key the store by session or user id.

Best Practices

• Skip the condense LLM call when history is empty; it saves cost and removes one failure point.
• Give the condense prompt a strict instruction: “Return only the question; do not answer it.” Use a low temperature for it.
• Log the standalone question during development. Most conversational RAG bugs are visible there.
• Trim or summarise old turns, and store history in a database (not process memory) for real deployments.
• Use one session id per conversation, and never mix users in one history.
• Tell the answer model to rely on the retrieved context and admit when it is missing, so history cannot replace evidence.
• Consider a smaller, cheaper model for the rewrite step. Check your provider for current model names and prices.
• Test with multi-turn scripts that include pronouns, topic switches and unrelated questions.

Practice Exercises

1. Extend the offline chain. Make rewrite also replace “its” and “they” with the topic. Test it with the question “What are its limits?”. Hint: compare lowercase tokens against a set.
2. Add a window. In Example 2, pass only the last two messages of history into the chain on every turn. Does the rewrite still work after five questions?
3. Topic switch. Design a rule (or a prompt) so that a new, unrelated question such as “What is FAISS?” is not rewritten using the previous topic. Write two test conversations that show the difference.

Summary

• Follow-up questions fail in plain RAG because the retriever and the LLM have no memory.
• Conversational RAG adds a condense step that rewrites the question into a standalone query using chat history.
• Retrieve with the standalone question; answer with the context, the history and the original question.
• MessagesPlaceholder inserts history into prompts, and RunnableWithMessageHistory loads and saves it per session.
• Each follow-up turn can cost an extra LLM call, so skip rewriting on the first turn and use a cheap model for it.
• Trim history, key it by session, and persist it outside process memory in production.