Evaluating RAG Quality

Evaluating a RAG system means measuring, with numbers, how good its answers are and why. Without evaluation you are guessing: a new chunk size or a new prompt may feel better, but you cannot prove it. In this lesson you will learn to evaluate the retriever and the generator separately, build a small golden dataset, compute retrieval metrics in plain Python, and score answer faithfulness, including with an LLM acting as a judge.

Overview: why RAG needs two kinds of evaluation

A RAG pipeline has two stages: retrieval (find the right chunks) and generation (write an answer from those chunks). A wrong final answer can come from either stage. If the retriever never fetched the right chunk, the best model in the world cannot answer correctly. If the retriever did fetch it but the model ignored it or invented extra facts, the fault is in generation. So you must measure each stage on its own. Otherwise you will tune the wrong part.

Retrieval metrics

These compare the document IDs your retriever returned with the IDs you know are correct. They are cheap, fast and need no LLM.

• Hit rate (hit@k): the share of questions where at least one correct document appears in the top k results.
• MRR (Mean Reciprocal Rank): for each question, take 1 divided by the rank of the first correct document (1, 1/2, 1/3 …; 0 if missing), then average. It rewards putting the right chunk first.
• Recall@k: of all documents that are relevant, what fraction appears in the top k.
• Precision@k: of the k returned documents, what fraction is relevant. Low precision means noise in the prompt.

Generation metrics

These judge the written answer. They are harder because language is flexible.

• Faithfulness (groundedness): is every claim in the answer supported by the retrieved context? This detects hallucination.
• Answer relevance: does the answer actually address the question?
• Correctness: does the answer agree with a reference answer you wrote?
• Context precision and context recall: how much of the retrieved context was useful, and whether it contained everything needed. Libraries such as RAGAS and platforms such as LangSmith offer ready-made versions of these. Their exact names and APIs change between releases, so check their current documentation.

The golden dataset

Everything starts with a golden dataset: a list of realistic questions, each with the IDs of the chunks that answer it and ideally a short reference answer. Even 30 to 50 carefully chosen questions are far better than none. Take questions from real users or support tickets, include easy and hard ones, and include questions the corpus cannot answer (the correct behaviour is to say “I don’t know”).

Syntax and key concepts

A golden record is plain data. Store it as JSON or in a LangSmith dataset so it can be versioned and rerun.

[
  {
    "question": "How many days do I have to ask for a refund?",
    "relevant_ids": ["d1"],
    "reference_answer": "You can request a refund within 30 days of purchase."
  },
  {
    "question": "Can I pay with cryptocurrency?",
    "relevant_ids": [],
    "reference_answer": "The documents do not say."
  }
]

• question: the user input sent to the pipeline.
• relevant_ids: the document or chunk IDs that contain the answer. An empty list marks an unanswerable question.
• reference_answer: a short ground truth for correctness checks.

To make IDs available at evaluation time, put them in Document.metadata when you index, for example metadata={"id": "d1"}. Retrievers return Document objects, so you can read the IDs back.

Examples

Example 1: retrieval metrics in pure Python

The metrics are simple enough to write yourself, which helps you understand them. Each case holds the IDs the retriever returned (in order) and the set of correct IDs.

def first_relevant_rank(retrieved, relevant):
    for position, doc_id in enumerate(retrieved, start=1):
        if doc_id in relevant:
            return position
    return None


def recall_at_k(retrieved, relevant, k):
    found = set(retrieved[:k]) & relevant
    return len(found) / len(relevant)


cases = [
    (['d1', 'd4', 'd7'], {'d1'}),
    (['d9', 'd2', 'd5'], {'d2', 'd3'}),
    (['d6', 'd8', 'd4'], {'d3'}),
]

ranks = [first_relevant_rank(r, rel) for r, rel in cases]
hit_rate = sum(rank is not None for rank in ranks) / len(cases)
mrr = sum(1 / rank for rank in ranks if rank) / len(cases)
recall = sum(recall_at_k(r, rel, 3) for r, rel in cases) / len(cases)

print('hit_rate=%.3f' % hit_rate)
print('mrr=%.3f' % mrr)
print('recall@3=%.3f' % recall)

Output:

hit_rate=0.667
mrr=0.500
recall@3=0.500

Case 1 finds the right document at rank 1 (reciprocal rank 1.0). Case 2 finds one of two relevant documents at rank 2 (reciprocal rank 0.5, recall 0.5). Case 3 misses completely (0 for everything). Averaging gives hit rate 2/3, MRR (1 + 0.5 + 0) / 3 = 0.5, and recall@3 of 0.5.

Example 2: evaluating a real LangChain retriever offline

Here we evaluate an actual BaseRetriever. To avoid API keys, it scores documents by word overlap instead of embeddings. In your own project you would swap in vectorstore.as_retriever() and keep the evaluation loop unchanged.

import re

from langchain_core.callbacks import CallbackManagerForRetrieverRun
from langchain_core.documents import Document
from langchain_core.retrievers import BaseRetriever


def tokens(text):
    return set(re.findall(r'[a-z0-9]+', text.lower()))


class OverlapRetriever(BaseRetriever):
    docs: list[Document]
    k: int = 2

    def _get_relevant_documents(
        self, query: str, *, run_manager: CallbackManagerForRetrieverRun
    ) -> list[Document]:
        q = tokens(query)
        scored = [(len(q & tokens(d.page_content)), d) for d in self.docs]
        scored = [s for s in scored if s[0] > 0]
        scored.sort(key=lambda s: s[0], reverse=True)
        return [d for _, d in scored[: self.k]]


texts = {
    'd1': 'Refunds are available within 30 days of purchase.',
    'd2': 'Shipping takes 5 to 7 business days within India.',
    'd3': 'Premium support is available by email and chat.',
    'd4': 'Passwords can be reset from the account settings page.',
}
docs = [Document(page_content=t, metadata={'id': i}) for i, t in texts.items()]
retriever = OverlapRetriever(docs=docs, k=2)

golden = [
    {'question': 'How many days for refunds?', 'relevant': {'d1'}},
    {'question': 'How long does shipping take in India?', 'relevant': {'d2'}},
    {'question': 'How do I reset my password?', 'relevant': {'d4'}},
    {'question': 'Can I talk to a human on chat?', 'relevant': {'d3'}},
]

reciprocal_ranks = []
for item in golden:
    got = [d.metadata['id'] for d in retriever.invoke(item['question'])]
    rank = next((i for i, x in enumerate(got, 1) if x in item['relevant']), None)
    rr = 1 / rank if rank else 0.0
    reciprocal_ranks.append(rr)
    print('%r -> %s rr=%.2f' % (item['question'], got, rr))

hits = sum(1 for rr in reciprocal_ranks if rr > 0) / len(golden)
print('hit_rate=%.3f mrr=%.3f' % (hits, sum(reciprocal_ranks) / len(golden)))

Output:

'How many days for refunds?' -> ['d1', 'd2'] rr=1.00
'How long does shipping take in India?' -> ['d2'] rr=1.00
'How do I reset my password?' -> ['d4'] rr=1.00
'Can I talk to a human on chat?' -> ['d2', 'd3'] rr=0.50
hit_rate=1.000 mrr=0.875

All four questions are found in the top 2, so hit rate is perfect. But MRR is 0.875 because the last question ranks the wrong document first (it ties on common words like “to”). Hit rate alone would hide this weakness. That is why you should track several metrics together.

Example 3: scoring faithfulness

A faithfulness check asks: is each claim in the answer supported by the context? The simplest offline version counts how many meaningful answer words appear in the context. It is crude, but it shows the shape of an evaluator: a function from {answer, context} to a score. We wrap it in RunnableLambda so it behaves like any other LCEL component.

import re

from langchain_core.runnables import RunnableLambda

STOP = {'the', 'a', 'an', 'is', 'are', 'of', 'to', 'in', 'within',
        'for', 'by', 'and', 'from', 'can', 'be'}


def words(text):
    return re.findall(r'[a-z0-9]+', text.lower())


def groundedness(inputs):
    context = set(words(inputs['context']))
    claims = [w for w in words(inputs['answer']) if w not in STOP]
    supported = [w for w in claims if w in context]
    return len(supported) / len(claims)


judge = RunnableLambda(groundedness)
context = 'Refunds are available within 30 days of purchase.'

good = 'Refunds are available within 30 days.'
bad = 'Refunds are available for 90 days and shipping is free.'

print('grounded answer: %.2f' % judge.invoke({'answer': good, 'context': context}))
print('hallucinated answer: %.2f' % judge.invoke({'answer': bad, 'context': context}))

Output:

grounded answer: 1.00
hallucinated answer: 0.50

The first answer uses four meaningful words and all four are in the context. The second uses six, and only “refunds”, “available” and “days” are supported; “90”, “shipping” and “free” are invented. Word overlap cannot understand meaning, so real systems use an LLM judge instead.

Under the hood: LLM-as-judge

An LLM judge is a second model call that grades the first. You give it the question, the retrieved context and the answer, plus a strict rubric, and ask for a structured verdict. Because the judge is an LCEL chain, you can run it over a whole dataset with .batch().

from pydantic import BaseModel, Field
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI


class Verdict(BaseModel):
    faithful: bool = Field(description='True only if every claim is supported by the context')
    reason: str = Field(description='One short sentence explaining the verdict')


prompt = ChatPromptTemplate.from_messages([
    ('system', 'You are a strict grader. Judge only from the context given.'),
    ('human', 'Context:\n{context}\n\nAnswer:\n{answer}\n\nIs the answer fully supported by the context?'),
])

# Reads OPENAI_API_KEY from the environment. Model names change often; check the provider docs.
judge_llm = ChatOpenAI(model='gpt-4o-mini', temperature=0)
chain = prompt | judge_llm.with_structured_output(Verdict)

# verdict = chain.invoke({'context': '...', 'answer': '...'})
# print(verdict.faithful, verdict.reason)

The output is illustrative: a Verdict object such as faithful=False with a reason naming the unsupported claim. Step by step: the prompt fills in the two variables, the model reads the rubric, and with_structured_output forces the reply into the Verdict schema so your code gets a boolean, not free text to parse.

Judges are useful but not perfect. They can prefer long answers, favour their own style, and give different scores on reruns. Keep the temperature at 0, use a clear rubric, and periodically check judge scores against a human-labelled sample.

Common Mistakes

Mistake 1: looking only at the final answer

Wrong: "Accuracy dropped to 60%, so let's change the prompt."

Why it is wrong: the drop may come from retrieval. Check hit rate first. If the right chunk is not retrieved, no prompt change will help. Corrected approach: measure retrieval metrics, then generation metrics, and fix the stage that regressed.

Mistake 2: evaluating on the data you tuned on

Wrong: tune chunk size on 20 questions, then report the score on the same 20.

Why it is wrong: you overfit to those questions and the score is too optimistic. Corrected approach: keep a held-out test set that you never look at while tuning.

Mistake 3: trusting one number

A single “quality score” hides trade-offs. In Example 2, hit rate was 1.0 while MRR showed a ranking weakness. Report retrieval and generation metrics separately, and read a handful of failing examples by hand every time.

Mistake 4: no unanswerable questions

If every golden question has an answer, you never test whether the system refuses politely when the context is missing. Add some out-of-scope questions and check that the answer says it does not know.

Best Practices

• Build the golden dataset early, store it in version control, and grow it from real failures.
• Put stable IDs in document metadata so retrieval can be scored automatically.
• Evaluate retrieval and generation separately, then end-to-end.
• Run the evaluation on every change to chunking, embeddings, retriever settings, prompts or models, and compare against the previous run.
• Use cheap deterministic metrics (hit rate, MRR) in CI, and LLM judges on a scheduled or pre-release run to control cost.
• Fix the judge model, temperature and rubric, and spot-check judge verdicts against human labels.
• Track latency and cost next to quality; a small accuracy gain is not always worth double the cost.
• Log real production questions (respecting privacy) and add the failing ones to the dataset.
• Check current docs for RAGAS and LangSmith evaluators before adopting them, as their APIs evolve.

Practice Exercises

1. Add precision@k. Extend Example 1 with a precision_at_k function and print the mean precision@3. Hint: divide the number of relevant IDs in the top k by k.
2. Improve the retriever. In Example 2, make tokens ignore common words such as “to”, “a” and “on”. Rerun and check that MRR rises from 0.875. Hint: reuse a stop-word set like the one in Example 3.
3. Test unanswerable questions. Add a golden item whose relevant set is empty (for example, “Do you offer student discounts?”). Write code that counts it as correct only when the retriever returns no documents.

Summary

• RAG has two stages, so evaluate retrieval and generation separately.
• A golden dataset of questions, relevant IDs and reference answers is the foundation.
• Hit rate, MRR, recall@k and precision@k measure the retriever quickly and cheaply.
• Faithfulness, answer relevance and correctness measure the generated answer.
• LLM judges scale answer grading but need fixed settings and human spot-checks.
• Keep a held-out test set, include unanswerable questions, and rerun evaluation after every change.