Hybrid Search and Re-ranking

Hybrid search runs two different kinds of search at the same time, keyword search and vector (semantic) search, and merges the results. Re-ranking then takes that merged list and re-scores it with a slower but smarter model so the best chunks come first. Together they are among the most reliable ways to improve a RAG app, because the answer can only be as good as the chunks you retrieve.

Overview: how it works

Vector search turns text into embeddings (lists of numbers) and finds chunks whose meaning is close to the question. It is great for paraphrases: “how do I reset my password” can match “steps to recover account access”. But it is weak on exact tokens such as error codes (ERR_504), product IDs, names, or rare acronyms. An embedding model may not place such tokens precisely in the vector space.

Keyword search is the opposite. The classic algorithm is BM25. It scores a chunk higher when it contains the query words, especially rare words, and it handles document length sensibly. It is excellent for exact matches but knows nothing about synonyms.

Hybrid search uses both retrievers and fuses their ranked lists. The most common fusion method is Reciprocal Rank Fusion (RRF). Each document gets a score of 1 / (k + rank) from every list it appears in, and the scores are added. The constant k (often 60) stops the top few ranks from dominating. RRF uses only ranks, not raw scores, which is useful because BM25 scores and cosine similarities are on completely different scales.

Two stages: retrieve wide, then re-rank narrow

A good production pipeline has two stages:

1. First stage (recall): fast retrievers (BM25 + vectors) fetch maybe 20 to 50 candidates. The goal is to make sure the right chunk is somewhere in the list.
2. Second stage (precision): a re-ranker reads the question and each candidate together and scores them. You keep only the top 3 to 5 for the LLM prompt.

Why not use the re-ranker everywhere? Embedding search is a bi-encoder: the question and each chunk are embedded separately, so chunk vectors can be computed in advance. A cross-encoder feeds the question and chunk into one model together. It is much more accurate, but it must run once per candidate at query time, so it is too slow for the whole corpus. Using it only on the shortlist gives you the best of both.

Syntax / Key concepts

hybrid = EnsembleRetriever(retrievers=[bm25, vector], weights=[0.5, 0.5])
reranker = CrossEncoderReranker(model=cross_encoder, top_n=3)
final = ContextualCompressionRetriever(base_compressor=reranker, base_retriever=hybrid)

Import note: in LangChain 1.x, the older “classic” retrievers such as EnsembleRetriever and ContextualCompressionRetriever live in the langchain-classic package (module langchain_classic). In older versions they are under langchain. The examples below try the new location first and fall back. Check the current docs for your installed version.

Examples

Example 1: Reciprocal Rank Fusion from scratch

Writing RRF yourself is the best way to understand it. This code needs no libraries and no API key.

def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores.items(), key=lambda item: item[1], reverse=True)


bm25_ranking = ['d3', 'd1', 'd7']
vector_ranking = ['d1', 'd4', 'd3']

for doc_id, score in rrf([bm25_ranking, vector_ranking]):
    print(doc_id, f'{score:.4f}')

Output:

d1 0.0325
d3 0.0323
d4 0.0161
d7 0.0159

Documents d1 and d3 appear in both lists, so they receive two contributions and rise to the top. d1 wins because it is rank 2 and rank 1 (1/62 + 1/61), slightly better than d3 at rank 1 and rank 3 (1/61 + 1/63). Documents found by only one retriever fall below. Agreement between retrievers is the signal RRF rewards.

Example 2: Hybrid retrieval with EnsembleRetriever

Now the same idea using LangChain. To run offline we use DeterministicFakeEmbedding, which produces repeatable random vectors. These vectors carry no real meaning, so the vector half is only a stand-in. With a real embedding model (for example from a partner package such as langchain-openai, with the key read from an environment variable) you would get real semantic matches.

from langchain_core.documents import Document
from langchain_core.embeddings import DeterministicFakeEmbedding
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_community.retrievers import BM25Retriever

try:
    from langchain_classic.retrievers import EnsembleRetriever
except ImportError:
    from langchain.retrievers import EnsembleRetriever

docs = [
    Document(page_content='Error ERR_504 means the gateway timed out.', metadata={'id': 'd1'}),
    Document(page_content='To recover account access, use the reset link.', metadata={'id': 'd2'}),
    Document(page_content='Invoices are emailed on the first day of each month.', metadata={'id': 'd3'}),
    Document(page_content='Gateway settings can be changed in the admin panel.', metadata={'id': 'd4'}),
]

bm25 = BM25Retriever.from_documents(docs)
bm25.k = 3

store = InMemoryVectorStore.from_documents(docs, DeterministicFakeEmbedding(size=64))
vector = store.as_retriever(search_kwargs={'k': 3})

hybrid = EnsembleRetriever(retrievers=[bm25, vector], weights=[0.5, 0.5])

for doc in hybrid.invoke('ERR_504'):
    print(doc.metadata['id'], doc.page_content)

Output (illustrative; the order of the lower results depends on the fake vectors):

d1 Error ERR_504 means the gateway timed out.
d3 Invoices are emailed on the first day of each month.
d2 To recover account access, use the reset link.

The exact token ERR_504 appears only in d1, so BM25 ranks it first, and it stays on top after fusion. This is the typical benefit of hybrid search: exact identifiers are not lost. The ensemble calls invoke on each retriever, then applies weighted RRF and removes duplicates by content.

Example 3: Adding a cross-encoder re-ranker

This pipeline retrieves a wide candidate set from the hybrid retriever, then keeps the best three. It downloads a model from Hugging Face on first run, so it needs internet access and the sentence-transformers package. The model name is an example; check the Hugging Face hub for current re-ranker models.

from langchain_community.cross_encoders import HuggingFaceCrossEncoder

try:
    from langchain_classic.retrievers import ContextualCompressionRetriever
    from langchain_classic.retrievers.document_compressors import CrossEncoderReranker
except ImportError:
    from langchain.retrievers import ContextualCompressionRetriever
    from langchain.retrievers.document_compressors import CrossEncoderReranker


def build_final_retriever(hybrid):
    cross_encoder = HuggingFaceCrossEncoder(model_name='BAAI/bge-reranker-base')
    reranker = CrossEncoderReranker(model=cross_encoder, top_n=3)
    return ContextualCompressionRetriever(
        base_compressor=reranker,
        base_retriever=hybrid,
    )


# Usage, with `hybrid` from Example 2 (set bm25.k and the vector k to ~20 first):
# final = build_final_retriever(hybrid)
# docs = final.invoke('Why did my request time out?')

The result of invoke is a list of at most three Document objects, ordered from most to least relevant. You can drop this retriever into an LCEL chain like any other: {'context': final, 'question': RunnablePassthrough()} | prompt | llm | StrOutputParser().

Under the hood: step by step

For the query “Why did my request time out?” the full pipeline does this:

1. BM25 tokenizes the query and scores every chunk by term frequency and rarity. It returns its top 20.
2. The vector store embeds the query, finds the 20 nearest chunk vectors, and returns them.
3. The ensemble applies weighted RRF to both lists and removes duplicates, giving perhaps 30 unique candidates.
4. The cross-encoder scores each (query, chunk) pair, which means 30 model forward passes.
5. The compressor sorts by score and keeps the top 3. Only these go into the LLM prompt.

The cost of step 4 grows with the number of candidates and the chunk length, so choose the candidate count deliberately. A common range is 20 to 50.

Common Mistakes

Mistake 1: re-ranking too few candidates

bm25.k = 3
vector = store.as_retriever(search_kwargs={'k': 3})
reranker = CrossEncoderReranker(model=cross_encoder, top_n=3)

The re-ranker can only reorder what it receives. With 3 candidates in and top_n=3 out, it changes nothing useful, and if the right chunk was rank 8 in a first-stage list it is already lost. Fix: retrieve 20 or more per retriever, then cut to 3 to 5 with top_n.

Mistake 2: comparing raw scores across retrievers

combined = bm25_scores[doc_id] + cosine_scores[doc_id]

BM25 scores are unbounded and depend on the corpus, while cosine similarity is between -1 and 1. Adding them lets BM25 dominate without you noticing. Fix: fuse by rank (RRF), as EnsembleRetriever does, or normalize scores carefully first.

Mistake 3: forgetting that BM25 holds everything in memory

BM25Retriever.from_documents builds an in-memory index each time your program starts, and it does not update when you add documents to your vector store. If the two indexes drift apart, hybrid search silently gets worse. Fix: rebuild both from the same document list, or use a vector database that supports native hybrid search (many do; check your store’s documentation).

Best Practices

• Start with equal weights (0.5 / 0.5), then tune using a small set of real questions with known correct chunks.
• Measure before and after: track hit rate (is the right chunk in the top N?) and answer quality. Do not assume more components always help.
• Retrieve wide (20 to 50), re-rank, and send only 3 to 5 chunks to the LLM. This also reduces token cost and the “lost in the middle” effect.
• Keep chunk IDs in metadata so you can deduplicate and trace which retriever found what.
• Cache re-ranker models in memory; loading one on every request is very slow.
• If latency matters, run on GPU, shorten chunks, or reduce candidates. Hosted re-rank APIs exist too; check each provider’s current models and pricing.
• Use metadata filters (language, tenant, date) in both retrievers so fusion only combines allowed documents.
• For tests and CI, use fake embeddings and small fixed documents so results are repeatable.

Practice Exercises

1. Extend RRF. Change the rrf function so each ranking has its own weight (for example 0.7 for BM25 and 0.3 for vectors). Test it using the lists from Example 1. Hint: multiply each contribution by the weight.
2. Compare retrievers. Using the documents in Example 2, query for “gateway” with BM25 alone, then with the ensemble. Write down which documents appear and why.
3. Measure re-ranking. Build 10 question and expected-chunk pairs from your own notes. Compute the fraction where the expected chunk is in the top 3 before and after adding the cross-encoder.

Summary

• Vector search finds meaning; BM25 finds exact words. Hybrid search combines both.
• Reciprocal Rank Fusion merges ranked lists using 1 / (k + rank) and rewards documents that several retrievers agree on.
• EnsembleRetriever implements weighted fusion; the import path depends on your LangChain version.
• A cross-encoder re-ranker scores (question, chunk) pairs jointly. It is accurate but slow, so use it only on a shortlist.
• Retrieve wide for recall, then re-rank narrow for precision, and always measure the effect on your own data.