What RAG Is and When to Use It

Retrieval-Augmented Generation (RAG) is a pattern where an application first finds relevant information in your own data, and then asks a large language model (LLM) to write an answer using that information. It matters because an LLM only knows what it saw during training. It does not know your company documents, yesterday’s news or your private database. RAG fixes this without retraining the model.

Overview: How RAG Works

An LLM is a text predictor. You give it a prompt, and it continues the text. Two limits cause most real-world problems:

• Knowledge cut-off: the model was trained up to some date. Anything newer is unknown to it.
• No private data: your internal wikis, contracts, tickets and product manuals were never in its training set.

When a model is asked about something it does not know, it often still produces a fluent answer. This is called a hallucination: confident text that is not true. RAG reduces hallucinations by placing the correct facts directly inside the prompt, so the model reads them instead of guessing.

The word retrieval means looking up documents. The word augmented means the prompt is extended with those documents. The word generation means the LLM writes the final answer. The core idea fits in one line: retrieve, then generate.

The two phases

Every RAG system has two phases:

1. Indexing (offline): you load your documents, split them into small pieces called chunks, convert each chunk to a list of numbers called an embedding, and store them in a vector store (a database that finds similar items quickly). You do this once, and again whenever the data changes.
2. Query (online): when a user asks a question, you embed the question, search the vector store for the most similar chunks, put those chunks into a prompt, and call the LLM.

The later lessons in this course cover each step in depth. In this lesson you only need the big picture, so the examples below use a very simple keyword search instead of embeddings.

What the LLM actually sees

The model never connects to your database. It only receives text. A RAG prompt usually has three parts: an instruction (for example, answer only from the context), the retrieved context, and the user question. The quality of the final answer depends on both steps: if retrieval returns the wrong chunk, even the best model will answer badly.

Syntax and Key Concepts

RAG is a pattern, not a single function. In LangChain it is usually written as a chain using the | operator (LangChain Expression Language, or LCEL). The general shape is:

question -> retriever -> context
(context + question) -> prompt -> LLM -> parser -> answer

The key terms you will meet throughout this course:

To follow the LangChain example below, install the core package. It needs no API key.

pip install langchain-core

Examples

Example 1: RAG in plain Python

First, let us build the idea with no libraries. We store three facts, score each by how many words it shares with the question, and place the best one in a prompt.

import re

DOCS = [
    'Refunds are accepted within 30 days of purchase with a receipt.',
    'Standard shipping takes 5 to 7 business days within India.',
    'Support is available Monday to Friday from 9 am to 6 pm IST.',
]


def tokens(text):
    return set(re.findall(r'[a-z0-9]+', text.lower()))


def retrieve(question, docs, k=1):
    q = tokens(question)
    scored = [(len(q & tokens(d)), d) for d in docs]
    scored.sort(key=lambda pair: pair[0], reverse=True)
    return [d for score, d in scored[:k]]


question = 'How many days does shipping take?'
context = retrieve(question, DOCS)

prompt = f'''Answer using only this context:
{context[0]}

Question: {question}'''
print(prompt)

Output:

Answer using only this context:
Standard shipping takes 5 to 7 business days within India.

Question: How many days does shipping take?

The question shares only the word days with the refund text (score 1), but shares days and shipping with the shipping text (score 2), so that one wins. The word take does not match takes, which shows why real systems use embeddings that match meaning, not exact words. The printed prompt is what you would send to an LLM.

Example 2: The same flow as a LangChain chain

Now we express the flow with LCEL. To keep it runnable offline, the "LLM" is a fake step that just echoes what it received. In a real app you would replace it with a chat model from a partner package (for example langchain_openai.ChatOpenAI, reading its key from an environment variable).

from langchain_core.documents import Document
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnableLambda, RunnablePassthrough

docs = [
    Document(
        page_content='Refunds are accepted within 30 days of purchase.',
        metadata={'source': 'refund-policy.md'},
    ),
    Document(
        page_content='Standard shipping takes 5 to 7 business days.',
        metadata={'source': 'shipping.md'},
    ),
]


def words(text):
    return set(text.lower().replace('?', '').replace('.', '').split())


def retrieve(question):
    q = words(question)
    best = max(docs, key=lambda d: len(q & words(d.page_content)))
    return best.page_content + ' (source: ' + best.metadata['source'] + ')'


prompt = ChatPromptTemplate.from_messages([
    ('system', 'Answer only from the context.'),
    ('human', 'Context: {context} | Question: {question}'),
])


def fake_llm(prompt_value):
    human_text = prompt_value.to_messages()[-1].content
    return 'FAKE LLM saw: ' + human_text


chain = (
    {'context': RunnableLambda(retrieve), 'question': RunnablePassthrough()}
    | prompt
    | RunnableLambda(fake_llm)
    | StrOutputParser()
)

print(chain.invoke('How long does shipping take?'))

Output:

FAKE LLM saw: Context: Standard shipping takes 5 to 7 business days. (source: shipping.md) | Question: How long does shipping take?

The dictionary at the start of the chain runs both entries in parallel on the same input: retrieve builds the context, and RunnablePassthrough forwards the original question. The prompt template fills in both values, the fake model reads the final human message, and StrOutputParser returns a plain string. Swapping in a real chat model changes only one line.

Example 3: Admitting "I do not know"

A good RAG system should refuse when nothing relevant is found, instead of guessing. We add a minimum score.

import re

DOCS = [
    'Refunds are accepted within 30 days of purchase with a receipt.',
    'Standard shipping takes 5 to 7 business days within India.',
    'Support is available Monday to Friday from 9 am to 6 pm IST.',
]


def tokens(text):
    return set(re.findall(r'[a-z0-9]+', text.lower()))


def answer(question, docs, min_score=1):
    q = tokens(question)
    best_score, best_doc = max(
        ((len(q & tokens(d)), d) for d in docs),
        key=lambda pair: pair[0],
    )
    if best_score < min_score:
        return 'I do not know based on the provided documents.'
    return 'Based on the documents: ' + best_doc


print(answer('Is support open on Sunday?', DOCS))
print(answer('Do you sell gift cards?', DOCS))

Output:

Based on the documents: Support is available Monday to Friday from 9 am to 6 pm IST.
I do not know based on the provided documents.

The first question matches is and support. The second matches nothing, so the system declines. In production the same idea is done with a similarity-score threshold and an instruction in the prompt telling the model to say it does not know.

Step by Step: One Real Query

1. The user asks: "What is the refund window?"
2. The app embeds the question into a vector.
3. The vector store returns the top few chunks closest in meaning.
4. The app builds a prompt: instruction + chunks + question.
5. The LLM reads the prompt and writes an answer from the chunks.
6. The app returns the answer, ideally with the source names so the user can verify it.

When to Use RAG (and When Not To)

RAG is one of several ways to give a model knowledge. Compare them:

Choose RAG when: your data is large or changes often, answers must be traceable to a source, different users may be allowed to see different documents, or you want to update knowledge by simply re-indexing.

Do not choose RAG when: the whole knowledge fits comfortably in one prompt, the task is about tone or format rather than facts, or you need exact answers from structured data (use a SQL query or an API call instead). RAG and fine-tuning are also not rivals: many systems fine-tune for style and use RAG for facts.

Common Mistakes

Mistake 1: Asking the model without any context

answer = llm.invoke('What is our refund policy?')

This is wrong because the model has never seen your policy. It will invent a plausible one. The fix is to retrieve first and pass the text in:

context = retrieve('What is our refund policy?')
answer = llm.invoke('Use only this context: ' + context + ' Question: What is our refund policy?')

Mistake 2: Stuffing every document into the prompt

Sending all your documents each time looks simple, but it is slow, costly and can exceed the context window. Even when it fits, models can miss details buried in a long prompt. Retrieve only the few most relevant chunks instead.

Mistake 3: Trusting the answer without checking retrieval

When an answer is wrong, beginners often blame the LLM. Very often the retriever returned the wrong chunks. Always print and inspect the retrieved text first.

Best Practices

• Tell the model to answer only from the context and to say when the context is not enough.
• Return source names or links with every answer so users can verify it.
• Evaluate retrieval separately from generation, with a small set of real questions and expected sources.
• Keep the index fresh: re-index when documents change, and store metadata such as source and date.
• Start simple (one retriever, one prompt) and add complexity only when measurements show a problem.
• Never put real API keys in code; read them from environment variables. Use fake models or embeddings for tests so they run offline.
• Check your provider’s documentation for current model names, limits and prices, because they change often.

Practice Exercises

1. Add a fourth document to Example 1 about a warranty period. Write a question that retrieves it and print the resulting prompt.
2. Change retrieve in Example 1 to return the top two documents (k=2) and build a prompt that lists both. Hint: join them with a space or a line break.
3. For each situation, decide between RAG, fine-tuning, long-context prompting or a database query: (a) a chatbot over 50,000 support articles, (b) making replies sound like your brand, (c) "how many orders did we ship last week?". Write one sentence of reasoning for each.

Summary

• RAG means retrieve relevant data first, then let the LLM generate an answer from it.
• It solves the knowledge cut-off and private-data problems and reduces hallucinations.
• There are two phases: indexing (offline) and querying (online).
• In LangChain, RAG is typically an LCEL chain: retriever, prompt, model, output parser.
• Choose RAG for large, changing or private knowledge that needs sources; choose other tools for style, small data or structured queries.
• Retrieval quality decides answer quality, so inspect and test it.