Project: Documentation Chatbot

In this project you will build a chatbot that answers questions about a set of documentation pages. It looks up the most relevant passages first, then asks a language model to answer using only those passages. This pattern is called Retrieval-Augmented Generation (RAG). It is one of the most common real-world uses of LangChain, and the project joins together everything from the earlier lessons.

Overview / How the project works

A documentation chatbot has two phases. The ingestion phase runs once (or whenever the docs change). The question phase runs for every user message.

1. Load the documentation pages as Document objects. Each has text (page_content) and metadata, such as the source file name.
2. Split every page into small chunks. Small chunks give precise search results and fit easily in the prompt.
3. Embed each chunk. An embedding is a list of numbers that represents the meaning of the text. Similar meanings give similar numbers.
4. Store the vectors in a vector store.
5. Retrieve: at question time, embed the question and fetch the k nearest chunks.
6. Generate: put the chunks and the chat history into a prompt, call the chat model, and parse the reply to text.

We will use three small files: ingest.py (steps 1 to 4), chain.py (steps 5 and 6) and app.py (the chat loop). The project works offline. If the environment variable OPENAI_API_KEY is not set, it uses a fake embedding model and a fake chat model, so you can run and study the whole pipeline without an account. When the key is set, it switches to real OpenAI models through the langchain-openai package.

Setup

pip install langchain-core langchain-text-splitters
# optional, only for real models:
pip install langchain-openai
export OPENAI_API_KEY="your-key-here"
python app.py

Never write a real key inside your source files. Always read it from the environment.

Step 1: Ingestion (ingest.py)

To keep the project self-contained, the documentation is written inside the file as four short pages. In a real project you would read .md files from a folder and create one Document per file, in the same way.

import os

from langchain_core.documents import Document
from langchain_core.embeddings import DeterministicFakeEmbedding
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_text_splitters import RecursiveCharacterTextSplitter

DOCS = [
    Document(
        page_content='Installation. Install the SDK with pip install acme-sdk. Python 3.9 or newer is required.',
        metadata={'source': 'install.md'},
    ),
    Document(
        page_content='Authentication. Create an API token in the dashboard and set it in the ACME_TOKEN environment variable. Never commit tokens to git.',
        metadata={'source': 'auth.md'},
    ),
    Document(
        page_content='Rate limits. Each token may send 60 requests per minute. When you receive HTTP 429, wait and retry with exponential backoff.',
        metadata={'source': 'limits.md'},
    ),
    Document(
        page_content='Errors. HTTP 401 means the token is missing or invalid. HTTP 404 means the resource does not exist. HTTP 500 means a server problem.',
        metadata={'source': 'errors.md'},
    ),
]


def make_embeddings():
    if os.environ.get('OPENAI_API_KEY'):
        from langchain_openai import OpenAIEmbeddings

        return OpenAIEmbeddings(model=os.environ.get('EMBED_MODEL', 'text-embedding-3-small'))
    return DeterministicFakeEmbedding(size=256)


def build_store():
    splitter = RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=30)
    chunks = splitter.split_documents(DOCS)
    store = InMemoryVectorStore(make_embeddings())
    store.add_documents(chunks)
    return store

RecursiveCharacterTextSplitter tries to cut at paragraph breaks first, then lines, then spaces, so chunks stay readable. split_documents copies the metadata to every chunk, so each chunk still knows its source. chunk_overlap repeats a little text between neighbouring chunks, so a sentence cut at a boundary is not lost. InMemoryVectorStore keeps vectors in Python memory. It is perfect for learning and tests; use a persistent store such as Chroma, FAISS or pgvector in production.

Important: DeterministicFakeEmbedding makes a repeatable vector from the text hash. It has no real meaning, so offline search results are not truly relevant. It only lets you test the plumbing. Use real embeddings to judge answer quality.

Step 2: The RAG chain (chain.py)

import itertools
import os

from langchain_core.language_models.fake_chat_models import GenericFakeChatModel
from langchain_core.messages import AIMessage
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.runnables import RunnableLambda, RunnablePassthrough

SYSTEM = (
    'You are a helpful assistant for the Acme SDK documentation. '
    'Answer using ONLY the context below. '
    'If the answer is not in the context, say that you do not know. '
    'Mention the source file names you used.'
    '\n\nContext:\n{context}'
)


def make_llm():
    if os.environ.get('OPENAI_API_KEY'):
        from langchain_openai import ChatOpenAI

        return ChatOpenAI(model=os.environ.get('CHAT_MODEL', 'gpt-4o-mini'), temperature=0)
    reply = AIMessage(content='(offline demo) Install it with: pip install acme-sdk')
    return GenericFakeChatModel(messages=itertools.cycle([reply]))


def format_docs(docs):
    return '\n\n'.join(f'[{d.metadata["source"]}] {d.page_content}' for d in docs)


def build_chain(store, llm):
    retriever = store.as_retriever(search_kwargs={'k': 3})
    prompt = ChatPromptTemplate.from_messages([
        ('system', SYSTEM),
        MessagesPlaceholder('history'),
        ('human', '{question}'),
    ])
    retrieve = RunnablePassthrough.assign(
        docs=RunnableLambda(lambda x: retriever.invoke(x['question']))
    )
    add_context = RunnablePassthrough.assign(
        context=RunnableLambda(lambda x: format_docs(x['docs']))
    )
    generate = RunnablePassthrough.assign(answer=prompt | llm | StrOutputParser())
    return retrieve | add_context | generate

The chain uses the LCEL pipe operator |. The input is a dictionary with question and history. RunnablePassthrough.assign keeps the existing keys and adds a new one, so the dictionary grows as it moves along: first docs, then context, then answer. We keep docs in the final result so the app can show sources. The system prompt gives the model a strict rule: use only the context and admit when it does not know. This rule is what makes the bot trustworthy.

In offline mode the fake chat model always returns the same canned message. That is enough to prove the chain runs from start to finish.

Step 3: The chat loop (app.py)

from langchain_core.messages import AIMessage, HumanMessage

from chain import build_chain, make_llm
from ingest import build_store


def ask(chain, history, question):
    result = chain.invoke({'question': question, 'history': history})
    history.extend([HumanMessage(content=question), AIMessage(content=result['answer'])])
    sources = sorted({d.metadata['source'] for d in result['docs']})
    return result['answer'], sources


def main():
    chain = build_chain(build_store(), make_llm())
    history = []
    for question in ['How do I install the SDK?', 'And how do I authenticate?']:
        answer, sources = ask(chain, history, question)
        print('Q:', question)
        print('A:', answer)
        print('Sources found:', len(sources) > 0)
    print('History messages:', len(history))


if __name__ == '__main__':
    main()

Output (offline mode):

Q: How do I install the SDK?
A: (offline demo) Install it with: pip install acme-sdk
Sources found: True
Q: And how do I authenticate?
A: (offline demo) Install it with: pip install acme-sdk
Sources found: True
History messages: 4

Each turn adds two messages (the human question and the AI answer) to history, so after two turns there are four. The history list lives outside the chain. That makes the chain stateless and easy to test. With a real model, the answers will be different and grounded in the retrieved text; the output then is illustrative and its wording will vary.

Under the hood: one question, step by step

1. You call chain.invoke with the question and the history.
2. The retriever embeds the question with the same embedding model used at ingestion. This must match; vectors from different models cannot be compared.
3. The vector store computes similarity against every stored chunk and returns the top 3 as Document objects.
4. format_docs joins them into one text block, each labelled with its source file.
5. The prompt template fills {context}, inserts the history messages at the placeholder and appends the new question.
6. The chat model produces a message and StrOutputParser turns it into a plain string.

Common Mistakes

1. A prompt that does not ground the model

prompt = ChatPromptTemplate.from_messages([
    ('system', 'You are a helpful assistant.'),
    ('human', '{question}'),
])
splitter = RecursiveCharacterTextSplitter(chunk_size=5000, chunk_overlap=0)

This is wrong for two reasons. The prompt has no {context}, so retrieved text never reaches the model, and the bot will invent answers from its general knowledge. Also, 5000-character chunks mix many topics, so the retriever returns noisy, unfocused passages. The fix is the system prompt from Step 2 and chunks of a few hundred to about a thousand characters, tuned by testing.

2. Mixing embedding models

If you build the store with one embedding model and then query with another (for example, you set the API key only for one run), the results are meaningless or the dimensions do not match. Rebuild the store whenever you change the embedding model.

3. Forgetting that follow-up questions need context

A question like “And how do I authenticate?” works here because the topic words are in the question. A question like “What about that?” gives the retriever nothing to search for. The history helps the model, but not the retriever. See the extension ideas for a fix.

Best Practices

• Tell the model clearly to answer only from the context and to say “I do not know” otherwise.
• Always keep source metadata on chunks and show it to the user, so answers can be checked.
• Tune chunk_size, chunk_overlap and k by testing real questions, not by guessing.
• Keep history short (for example the last 6 to 10 messages) to control cost and prompt length.
• Read keys and model names from environment variables; model names change often, so check your provider’s documentation for current ones.
• Use a persistent vector store in production, and re-ingest only the pages that changed.
• Write a small test set of questions with known answers and run it after each change.

Practice Exercises

1. Add a fifth documentation page about “Webhooks” and ask a question about it. Hint: add one more Document with a new source name.
2. Change k from 3 to 1 and then to 4. What changes in the context text sent to the model? Hint: temporarily print result['context'].
3. Trim history so only the last 6 messages are passed to the chain. Hint: use slicing, history[-6:], in ask.

Extension ideas

• Load real .md files from a folder with pathlib.
• Add a question-rewriting step that turns a follow-up into a standalone question before retrieval.
• Stream the answer token by token with chain.stream.
• Swap in a persistent store (Chroma or FAISS) and add a web UI.
• Move to LangGraph if you want routing, retries or tool calls around the retriever.

Summary

• A documentation chatbot is load, split, embed, store, retrieve and generate.
• Chunks keep their source metadata, which lets you cite sources.
• LCEL with RunnablePassthrough.assign builds up a result dictionary step by step.
• A strict grounding prompt reduces made-up answers.
• Chat history is a plain list of messages kept outside the chain.
• Fake models let you test the pipeline offline; real models are needed to judge quality.