Retrievers and Similarity Search

A retriever is the part of a RAG app that takes a question and returns the most useful pieces of your data. Similarity search is the most common way it does this: it compares the meaning of the question with the meaning of stored text chunks. If retrieval returns the wrong chunks, the language model cannot answer well, so this step decides much of your app’s quality.

Overview: how retrieval works

Before search, you split documents into chunks. Each chunk is turned into an embedding: a list of numbers (a vector) that represents its meaning. The vectors are saved in a vector store, together with the chunk text and its metadata (source, page, topic and so on).

At question time, the same embedding model turns the user’s question into a vector. The store then finds the stored vectors that are closest to it. “Closest” is measured with a metric. The most common is cosine similarity, which looks at the angle between two vectors. A score near 1 means they point in almost the same direction (similar meaning). A score near 0 means they are unrelated.

The vector store can do this search itself. A retriever is a thinner, more general interface: in LangChain it is a Runnable that takes a string and returns a list of Document objects. Because it is a Runnable, you can use it with invoke, batch and the | operator of LCEL (LangChain Expression Language). A retriever does not have to use vectors at all. It can wrap keyword search, a SQL query, a web API, or a combination.

Vector store versus retriever

• A vector store offers methods such as similarity_search and similarity_search_with_score. It stores and searches.
• A retriever has one job: invoke(query) returns documents. You usually get one with store.as_retriever(...).
• Using the retriever interface lets you swap the store later (in-memory, Chroma, FAISS, a cloud database) without changing the rest of your chain.

Search types

Syntax and key concepts

retriever = vector_store.as_retriever(
    search_type='similarity',
    search_kwargs={'k': 4, 'filter': ...},
)
docs = retriever.invoke('your question')

• k: how many chunks to return. Small k may miss the answer. Large k adds noise and costs more tokens.
• filter: restricts the search using metadata. Its format depends on the store. Some stores take a dictionary; the in-memory store used below takes a Python function.
• fetch_k and lambda_mult: used by MMR. lambda_mult near 1 favours relevance; near 0 favours diversity.
• invoke: the modern way to call a retriever. Older method names such as get_relevant_documents are deprecated.

Examples

Example 1: similarity search with scores

Real embedding models need an API key or a download. To follow along offline, we write a tiny embedding class that counts words from a fixed vocabulary. It is not smart, but it is deterministic, so the output is exactly predictable. Any class that implements embed_documents and embed_query can be used by a vector store.

import re

from langchain_core.documents import Document
from langchain_core.embeddings import Embeddings
from langchain_core.vectorstores import InMemoryVectorStore

VOCAB = ['python', 'code', 'snake', 'reptile', 'dog', 'cat', 'pet', 'vector', 'search']


class BagOfWordsEmbeddings(Embeddings):
    def _embed(self, text: str) -> list[float]:
        words = re.findall(r'[a-z]+', text.lower())
        return [float(words.count(w)) for w in VOCAB]

    def embed_documents(self, texts: list[str]) -> list[list[float]]:
        return [self._embed(t) for t in texts]

    def embed_query(self, text: str) -> list[float]:
        return self._embed(text)


docs = [
    Document(page_content='Python code is easy to read', metadata={'topic': 'programming'}),
    Document(page_content='A python is a snake and a reptile', metadata={'topic': 'animals'}),
    Document(page_content='A dog is a friendly pet', metadata={'topic': 'animals'}),
    Document(page_content='A cat is also a pet', metadata={'topic': 'animals'}),
]

store = InMemoryVectorStore.from_documents(docs, BagOfWordsEmbeddings())

for doc, score in store.similarity_search_with_score('python code', k=2):
    print(f'{score:.3f} {doc.page_content}')

Output:

1.000 Python code is easy to read
0.408 A python is a snake and a reptile

The query vector has 1 for “python” and 1 for “code”. The first chunk has exactly the same vector, so cosine similarity is 1. The snake chunk shares only the word “python”, giving 1 / (√2 × √3) ≈ 0.408. The dog and cat chunks share nothing, so they score 0 and are not in the top 2. With a real embedding model the numbers would be less tidy, but the idea is identical.

Example 2: a retriever, a metadata filter and a chain

Continue in the same file. Here we turn the store into retrievers, filter by metadata, and plug a retriever into an LCEL chain.

retriever = store.as_retriever(search_kwargs={'k': 1})
print(retriever.invoke('python')[0].page_content)

animals_only = store.as_retriever(
    search_kwargs={'k': 1, 'filter': lambda d: d.metadata['topic'] == 'animals'}
)
print(animals_only.invoke('python')[0].page_content)


def format_docs(found):
    return '\n\n'.join(d.page_content for d in found)


pets = store.as_retriever(search_kwargs={'k': 2})
context_chain = pets | format_docs
print(context_chain.invoke('dog pet'))

Output:

Python code is easy to read
A python is a snake and a reptile
A dog is a friendly pet

A cat is also a pet

For the query “python”, the programming chunk scores about 0.707 and the snake chunk about 0.577, so the programming chunk wins. When we filter to topic == 'animals', only animal chunks are searched, so the snake chunk is returned. The last part shows why retrievers are Runnables: pets | format_docs creates a chain that returns one block of context text, ready to be placed in a prompt.

Example 3: a custom retriever with a score threshold

Similarity search always returns k results, even when nothing is truly relevant. A threshold lets you return fewer, or none. To write your own retriever, subclass BaseRetriever and implement _get_relevant_documents.

from langchain_core.callbacks import CallbackManagerForRetrieverRun
from langchain_core.retrievers import BaseRetriever


class ThresholdRetriever(BaseRetriever):
    store: InMemoryVectorStore
    k: int = 4
    min_score: float = 0.5

    def _get_relevant_documents(
        self, query: str, *, run_manager: CallbackManagerForRetrieverRun
    ) -> list[Document]:
        pairs = self.store.similarity_search_with_score(query, k=self.k)
        return [doc for doc, score in pairs if score >= self.min_score]


strict = ThresholdRetriever(store=store, min_score=0.5)
print([d.page_content for d in strict.invoke('python code')])
print([d.page_content for d in strict.invoke('vector search')])

Output:

['Python code is easy to read']
[]

The first query keeps only the chunk with score 1.0; the snake chunk (0.408) is dropped. The second query matches nothing in our tiny data, so the retriever returns an empty list. Your app can then say “I do not know” instead of inventing an answer from irrelevant text. Because ThresholdRetriever is a BaseRetriever, it automatically supports invoke, batch and use in chains.

Using real embeddings and MMR

In a real project you replace the fake class with a provider’s embeddings. The key comes from an environment variable, never from your source code. Model names change often, so check your provider’s documentation for the current one.

from langchain_openai import OpenAIEmbeddings
from langchain_core.vectorstores import InMemoryVectorStore

# Reads OPENAI_API_KEY from the environment.
embeddings = OpenAIEmbeddings(model='text-embedding-3-small')
store = InMemoryVectorStore.from_documents(docs, embeddings)

retriever = store.as_retriever(
    search_type='mmr',
    search_kwargs={'k': 4, 'fetch_k': 20, 'lambda_mult': 0.5},
)

MMR is useful when your top results are near copies of each other (for example, the same paragraph repeated in several files). It spends some of your k slots on variety.

Under the hood: step by step

1. Indexing: each chunk is passed to embed_documents. The vectors and chunks are saved in the store.
2. Query embedding: retriever.invoke('question') calls embed_query on the question.
3. Scoring: the store compares the query vector to stored vectors. Exact stores compare with all of them; large databases use an approximate index (such as HNSW) to stay fast, trading a little accuracy for speed.
4. Filtering: metadata filters limit which chunks are candidates.
5. Ranking: the top k chunks are sorted by score and returned as Document objects.
6. Generation: the chain places the text into a prompt and the LLM writes the answer.

Scores are not all the same

Some stores return a similarity (higher is better). Others return a distance (lower is better). Never compare raw numbers across different stores or metrics. Read the store’s documentation and test with a few known questions before choosing a threshold.

Common Mistakes

1. Using different embedding models for indexing and querying

store = InMemoryVectorStore.from_documents(docs, ModelA())
results = store.similarity_search('question')  # store was built with ModelA
# later, a new process creates the store object with ModelB()

Vectors from different models live in different spaces (and may even have different lengths). Comparing them gives meaningless results or errors. Use the same model, and the same version, for documents and queries. If you change the model, re-embed everything.

2. Assuming a higher k is always better

retriever = store.as_retriever(search_kwargs={'k': 50})

Fifty chunks fill the prompt with weak matches, raise cost, and can distract the model. Start with 3 to 6, measure answer quality, and adjust. If you need high recall, retrieve more and then re-rank or filter before sending to the LLM.

3. Using the deprecated method

docs = retriever.get_relevant_documents('question')

This older call is deprecated. Use retriever.invoke('question') so your code works the same way as every other Runnable.

Best Practices

• Always keep useful metadata (source file, page, section, date) on each chunk. It enables filters and citations.
• Tune chunk size and overlap together with k. Retrieval quality depends on chunking as much as on the search itself.
• Use a score threshold or an empty-result check so the app can answer “not found” safely.
• Try MMR when results look repetitive.
• Create a small test set of questions with the chunks you expect, and check retrieval separately from generation.
• Persist a real vector store on disk or in a database; an in-memory store is lost when the program ends.
• Read API keys from environment variables, and use a fake or local embedding class for tests so they run offline.
• Consider hybrid retrieval (keyword plus vector) when users search for exact names, codes or IDs, which pure meaning-based search can miss.

Practice Exercises

1. Add a fifth document about the word “search” and “vector” to the data in Example 1. Run similarity_search_with_score('vector search', k=2) and explain the scores you see.
2. Change ThresholdRetriever so it also accepts an optional topic field and only returns chunks whose metadata topic matches. Hint: use the filter argument of similarity_search_with_score.
3. Build a chain retriever | format_docs and then add a prompt template in front of a chat model so the full RAG flow works. Use a fake or stub model first if you have no API key.

Summary

• Embeddings turn text into vectors; similarity search finds the vectors closest to the query vector.
• A retriever is a Runnable: a string goes in, a list of Document objects comes out.
• store.as_retriever(search_type=..., search_kwargs=...) controls k, filters, MMR and thresholds.
• Use the same embedding model for documents and queries.
• Scores differ between stores; test before you trust a threshold.
• Subclass BaseRetriever to build custom retrieval logic that still works in LCEL chains.