Citations and Source Attribution

A RAG (Retrieval-Augmented Generation) app answers questions using documents you retrieved. A citation tells the reader which document, and which part of it, supports each claim. Without citations, users must trust the model blindly. With them, users can check the answer, and you can debug wrong answers quickly.

In this lesson you will learn how to carry source metadata through your pipeline, number sources in the prompt, get the model to return structured citations, and validate those citations in code.

Overview: how citations work

An LLM does not “know” where its words came from. It only sees the text you put in the prompt. So citation in RAG is a design you build, not a feature you switch on. The idea has four steps:

1. Keep metadata. Every chunk in your vector store is a Document with page_content and a metadata dictionary. Put things like source (file name or URL), page or a section title in that dictionary when you ingest the data.
2. Label the context. When you build the prompt, give every retrieved chunk a short number such as [1], [2]. The model can easily copy a small number. It cannot reliably copy a long URL.
3. Ask for citations. Tell the model to answer only from the numbered sources and to say which numbers it used. The best way is structured output: the model returns an object with an answer field and a citations list.
4. Resolve and verify. Your code maps each number back to the real Document and its metadata. It also checks that the numbers are valid. Then you show the real source names to the user.

Notice that the model only outputs numbers. Your code turns them into file names, pages and links. This is safer, because a model can invent a plausible-looking URL but it cannot change what is in your metadata.

Two levels of attribution

• Document-level: “This answer used policy.pdf and shipping.md.” Easy and usually enough.
• Claim-level (inline): “Refunds take 7 days [1]. Premium shipping is free [2].” Each sentence points to its own source. This is more useful but needs a good prompt and validation.

Why it matters

• Trust: users can open the source and check.
• Debugging: if the cited chunk does not support the answer, you know whether retrieval or generation failed.
• Compliance: legal, medical and finance apps often must show sources.

Syntax and key concepts

The general shape of a citation-aware chain in LCEL (LangChain Expression Language) is:

input {question, docs}
  -> format_docs(docs)         # adds [1], [2] labels
  -> prompt                    # tells the model to cite
  -> model.with_structured_output(CitedAnswer)
  -> resolve(result, docs)     # numbers -> real sources, validated

Install the packages used here:

pip install langchain-core langchain-openai pydantic

Examples

Example 1: Number the sources

This helper turns a list of documents into a prompt-ready context string. It needs no API key.

from langchain_core.documents import Document

docs = [
    Document(
        page_content='Refunds are processed within 7 business days.',
        metadata={'source': 'policy.pdf', 'page': 3},
    ),
    Document(
        page_content='Premium members get free shipping on all orders.',
        metadata={'source': 'shipping.md', 'page': 1},
    ),
]


def format_docs(docs):
    blocks = []
    for i, d in enumerate(docs, start=1):
        src = d.metadata['source']
        page = d.metadata['page']
        blocks.append(f'[{i}] (source: {src}, page {page})\n{d.page_content}')
    return '\n\n'.join(blocks)


print(format_docs(docs))

Output:

[1] (source: policy.pdf, page 3)
Refunds are processed within 7 business days.

[2] (source: shipping.md, page 1)
Premium members get free shipping on all orders.

enumerate(docs, start=1) makes the labels start at 1, which is natural for readers. The order of labels is the order of the list, so the same list must be used later when resolving citations.

Example 2: A chain that returns structured citations

Here we build the full LCEL chain. To keep it runnable offline, the model is replaced by a small function that returns a fixed CitedAnswer. The rest of the chain is exactly what you would use with a real model.

from pydantic import BaseModel, Field
from langchain_core.documents import Document
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnableLambda, RunnablePassthrough


class CitedAnswer(BaseModel):
    answer: str = Field(description='Answer using only the numbered sources.')
    citations: list[int] = Field(description='Numbers of the sources used.')


docs = [
    Document(
        page_content='Refunds are processed within 7 business days.',
        metadata={'source': 'policy.pdf', 'page': 3},
    ),
    Document(
        page_content='Premium members get free shipping on all orders.',
        metadata={'source': 'shipping.md', 'page': 1},
    ),
]


def format_docs(docs):
    blocks = []
    for i, d in enumerate(docs, start=1):
        src = d.metadata['source']
        blocks.append(f'[{i}] (source: {src})\n{d.page_content}')
    return '\n\n'.join(blocks)


prompt = ChatPromptTemplate.from_messages([
    ('system',
     'Answer only from the numbered sources. Cite source numbers like [1]. '
     'If the sources do not contain the answer, say you do not know.\n\n{context}'),
    ('human', '{question}'),
])


def fake_model(prompt_value):
    # Stands in for llm.with_structured_output(CitedAnswer)
    return CitedAnswer(
        answer='Refunds are processed within 7 business days [1].',
        citations=[1],
    )


chain = (
    RunnablePassthrough.assign(context=lambda x: format_docs(x['docs']))
    | prompt
    | RunnableLambda(fake_model)
)

result = chain.invoke({'question': 'How long do refunds take?', 'docs': docs})
print(result.answer)
print(result.citations)

Output:

Refunds are processed within 7 business days [1].
[1]

RunnablePassthrough.assign keeps the input dictionary and adds a new key, context. The prompt uses context and question; the extra docs key is simply ignored by the prompt. The chain returns a CitedAnswer object, not a raw string.

To use a real model, replace the fake step. This needs an API key in an environment variable. Check your provider’s documentation for current model names.

import os
from langchain_openai import ChatOpenAI

assert os.environ.get('OPENAI_API_KEY'), 'Set OPENAI_API_KEY first'

llm = ChatOpenAI(model='gpt-4o-mini', temperature=0)
structured_llm = llm.with_structured_output(CitedAnswer)

chain = (
    RunnablePassthrough.assign(context=lambda x: format_docs(x['docs']))
    | prompt
    | structured_llm
)

The real answer text will vary between runs, but its shape is always a CitedAnswer with an answer string and a list of integers.

Example 3: Resolve and validate citations

Never show model citations to users unchecked. This resolver maps numbers to metadata and collects invalid numbers separately.

from types import SimpleNamespace
from langchain_core.documents import Document

docs = [
    Document(page_content='Refunds take 7 business days.',
             metadata={'source': 'policy.pdf', 'page': 3}),
    Document(page_content='Premium members ship free.',
             metadata={'source': 'shipping.md', 'page': 1}),
]


def resolve(result, docs):
    valid_range = range(1, len(docs) + 1)
    sources = []
    invalid = []
    seen = set()
    for n in result.citations:
        if n not in valid_range:
            invalid.append(n)
        elif n not in seen:
            seen.add(n)
            meta = docs[n - 1].metadata
            sources.append({'id': n, 'source': meta['source'], 'page': meta['page']})
    return sources, invalid


result = SimpleNamespace(answer='Refunds take 7 days [1][5].', citations=[1, 5, 1])
sources, invalid = resolve(result, docs)
print(sources)
print(invalid)

Output:

[{'id': 1, 'source': 'policy.pdf', 'page': 3}]
[5]

The model cited 1, 5 and 1 again. Only two sources exist, so 5 is rejected. The duplicate 1 is removed with a seen set. In production you could log invalid IDs, retry the call, or refuse to show the answer.

Under the hood: step by step

1. The retriever returns, for example, 4 Document objects. Their metadata was stored at ingestion time and comes back unchanged.
2. format_docs builds one string with [1] to [4] labels. The order is fixed here.
3. The prompt sends the labelled text and the question to the model.
4. with_structured_output gives the model your schema (as a tool or JSON schema, depending on the provider). The provider returns JSON, and LangChain parses it into a CitedAnswer.
5. Your resolver uses the same docs list to translate numbers back to metadata.

The key rule: the list used to build the labels must be the same list used to resolve them. If you re-run retrieval between the two steps, the order may change and your citations will point to the wrong documents.

Preserving metadata when splitting

Text splitters such as RecursiveCharacterTextSplitter copy a parent document’s metadata onto each chunk when you call split_documents. That is why you should set source and page on the loaded document first. Many loaders (for example PDF loaders) already add source and page for you. Print docs[0].metadata once to see what your loader provides.

Common Mistakes

Mistake 1: Sending context without IDs

context = '\n'.join(d.page_content for d in docs)

Why it is wrong: the model has no label to cite, so it will either skip citations or invent file names from memory. Fix: use format_docs so every chunk has a number, and resolve numbers in code.

Mistake 2: Trusting citations blindly

for n in result.citations:
    print(docs[n - 1].metadata['source'])

Why it is wrong: if the model returns 0 or 9, you get a wrong source (index -1 is the last item in Python!) or an IndexError. Fix: check n is in range(1, len(docs) + 1), as in Example 3.

Mistake 3: Believing a citation means the claim is true

A valid number only proves the source was in the prompt. The model may still cite chunk 1 for a claim that chunk 1 does not support. Citations reduce risk; they do not remove it. For high-stakes apps, add a second check (an LLM-as-judge or a text-overlap check) that the cited chunk really supports the sentence.

Best Practices

• Add useful metadata at ingestion: file name, page, section title, URL, and a document ID. Retrofitting is painful.
• Use short numeric labels in the prompt and map them to rich metadata in code.
• Use structured output (a Pydantic schema) instead of parsing [1] from free text with regular expressions.
• Validate every citation. Log invalid ones; they are a good quality signal.
• Tell the model what to do when the sources lack the answer (“say you do not know”). This cuts made-up answers and made-up citations.
• Show users the source name, page and a short quote or link so they can check quickly.
• Keep the retrieved list immutable between prompt building and resolving.
• Limit the number of chunks (for example 3 to 6). Too many sources make citations noisy and increase cost.
• Never log or display secrets or private file paths in citations; show friendly names.

Practice Exercises

1. Add a quote field. Extend CitedAnswer so each citation has an id and a short quote string. Hint: create a second Pydantic class Citation and use list[Citation]. Then verify in code that each quote text actually appears inside the cited chunk.
2. Stable IDs. Change format_docs to use metadata['chunk_id'] instead of the list position. Update the resolver to look up documents by that ID in a dictionary. Think about why this survives re-ordering.
3. Footnote renderer. Write a function that takes the answer text and resolved sources and prints a “Sources:” list such as [1] policy.pdf, page 3. Skip invalid IDs and show a warning line if any exist.

Summary

• Citations are a pipeline design: keep metadata, label chunks, ask for citations, resolve and verify.
• Give the model short numbers; let your code convert them to real sources.
• Use with_structured_output with a Pydantic schema to get answer and citations as clean fields.
• Always validate citation IDs, and remove duplicates.
• The list used for labelling must be the same list used for resolving.
• A valid citation does not guarantee a supported claim; add support checks for high-stakes use.