Vector Search and Hybrid Retrieval

Vector search helps a generative AI system find passages whose meaning is close to a user question, even when the words are different. Hybrid retrieval adds exact lexical signals and metadata constraints so the system does not depend on semantic similarity alone. The outcome is a retrieval layer that can return grounded evidence for generation, explain why a passage was selected, and expose missing evidence before a language model turns that gap into a confident answer.

In this course, retrieval is the bridge between stored knowledge and generated text. Embeddings make that bridge flexible; hybrid ranking makes it controllable. A good retriever is not just a database query. It is a pipeline that chunks source material, embeds the chunks, stores vectors and metadata, searches by several signals, merges scores, reranks candidates, and measures whether the right evidence was reachable.

Purpose and Outcome

The purpose of vector search is nearest-neighbor lookup in an embedding space. A document chunk and a question are converted into numeric vectors. Vectors that point in similar directions are treated as semantically related. If a user asks, How do I rotate credentials?, vector search can retrieve a passage titled API key rollover because the concepts are close even though the terms differ.

The purpose of hybrid retrieval is to correct the weak spots of pure vector search. Acronyms, product codes, error messages, dates, names, and rare terms often matter exactly. Metadata such as tenant, document type, access level, language, and freshness can be mandatory rather than optional. Hybrid retrieval combines semantic similarity, keyword matching, and filters so the final candidate set is both relevant and eligible.

How It Works Internally

A typical pipeline starts with ingestion. Source documents are normalized, split into chunks, and assigned stable identifiers. Each chunk stores text, source metadata, and an embedding vector. Chunk size is a design choice: small chunks improve pinpoint citation but may omit surrounding context; large chunks preserve context but can dilute the vector and waste prompt budget. Overlap can reduce boundary misses, but too much overlap creates near-duplicates that crowd out diverse evidence.

At query time, the user question is embedded with the same embedding model family used for the corpus. The vector index returns approximate nearest neighbors. Common index structures trade exactness for speed by avoiding a full scan. That means the top result is usually, but not mathematically guaranteed to be, the best possible vector match. Parameters such as candidate count and search depth affect recall, latency, and cost.

Keyword retrieval runs a lexical query over tokenized text. BM25-style ranking rewards exact term overlap while accounting for term rarity and document length. This is valuable for identifiers such as ERR_AUTH_013 or Form 1099-K, where paraphrase is less important than exact match.

Metadata filtering can happen before or after candidate generation. Pre-filtering limits the search space to eligible chunks, which protects access control and improves precision. Post-filtering is sometimes faster to add but can harm recall: if the top vector candidates are filtered out, too few eligible passages may remain. Security-sensitive filters, such as tenant or document permission, should be enforced before retrieval whenever the storage system supports it.

Hybrid ranking then fuses candidates. Two common methods are weighted score normalization and reciprocal rank fusion. Weighted scoring combines normalized vector and keyword scores, for example 0.7 * semantic + 0.3 * lexical. Reciprocal rank fusion ignores raw scores and rewards items that appear near the top of multiple ranked lists. The right choice depends on whether your retrievers produce comparable scores and how much you value exact-term robustness.

API Anatomy

A retrieval request usually contains the query text, embedding model identifier, top-k count, filters, score weights, and optional reranking settings. A retrieval response should include chunk id, source id, text, metadata, component scores, final score, and rank. Keeping component scores visible is important: when an answer is wrong, you need to know whether the vector side, keyword side, filter, reranker, or generator caused the failure.

from math import sqrt

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    left = sqrt(sum(x * x for x in a))
    right = sqrt(sum(y * y for y in b))
    return dot / (left * right)

query = [1.0, 0.0]
chunks = {
    "api-key-rollover": [0.9, 0.1],
    "invoice-export": [0.1, 0.9],
    "credential-rotation": [0.8, 0.2],
}

ranked = sorted(chunks, key=lambda name: cosine(query, chunks[name]), reverse=True)
print(ranked[:2])

This first example uses tiny hand-written vectors so the behavior is visible. The query points mostly along the first dimension, so chunks with high first-coordinate values rank higher. Real embeddings may have hundreds or thousands of dimensions, but the operation is the same: compare direction, then return nearest neighbors.

Progressive Example 1: Semantic Match

The example above prints ['api-key-rollover', 'credential-rotation']. It shows why semantic retrieval can find related language. A chunk named credential-rotation is still near the query even though the query does not contain that exact phrase. The limitation is also visible: if an exact error code mattered, cosine similarity alone would not know that the code is non-negotiable.

Progressive Example 2: Hybrid Fusion

def normalize(scores):
    highest = max(scores.values()) or 1.0
    return {key: value / highest for key, value in scores.items()}

vector_scores = {"A": 0.88, "B": 0.82, "C": 0.30}
keyword_scores = {"A": 0.10, "B": 3.00, "C": 2.50}

v = normalize(vector_scores)
k = normalize(keyword_scores)
combined = {
    doc_id: round(0.65 * v.get(doc_id, 0) + 0.35 * k.get(doc_id, 0), 3)
    for doc_id in set(v) | set(k)
}

print(sorted(combined.items(), key=lambda item: item[1], reverse=True))

This example blends a semantic score with a keyword score. Document A is the strongest semantic match, but document B has much stronger exact-term evidence. The expected output is [('B', 0.956), ('A', 0.662), ('C', 0.484)]. Hybrid scoring moves B to the top because both signals matter. The trade-off is that weights now become a tuning surface. If the keyword weight is too high, spelling overlap can dominate meaning. If it is too low, rare identifiers disappear.

Progressive Example 3: Metadata Filters

chunks = [
    {"id": "public-reset", "tenant": "acme", "text": "Users can reset their password."},
    {"id": "private-sso", "tenant": "acme", "text": "SAML setup requires an admin token."},
    {"id": "other-tenant", "tenant": "globex", "text": "Admin token rotation steps."},
]

query_terms = {"admin", "token"}
eligible = [chunk for chunk in chunks if chunk["tenant"] == "acme"]
ranked = sorted(
    eligible,
    key=lambda chunk: sum(term in chunk["text"].lower() for term in query_terms),
    reverse=True,
)

print([chunk["id"] for chunk in ranked])

The expected output is ['private-sso', 'public-reset']. The Globex chunk contains useful words, but it belongs to another tenant and must not be considered. This is the central security lesson for retrieval-augmented generation: a generator cannot unsee leaked context. Authorization belongs in retrieval, not only in the user interface or final answer template.

Design Choices and Trade-offs

Choose chunking based on answer shape. Troubleshooting articles often work well as short procedure chunks with headings included. Legal or policy documents may need larger chunks so definitions and exceptions stay together. Tables may need special handling because row and column headers carry meaning that plain text extraction can lose.

Choose top_k for recall first, then control prompt size later. If retrieval only asks for three candidates, reranking cannot recover a passage that was never fetched. A common pattern is to retrieve a wider candidate pool, rerank it with a stronger model or cross-encoder, deduplicate by source, then send a smaller evidence set to generation.

Choose hybrid weights empirically. Build a small evaluation set of realistic questions with expected source chunks. Measure recall at k, mean reciprocal rank, citation accuracy, and answer faithfulness. A setting that looks good on average may still fail critical classes such as part numbers or policy exceptions, so segment the evaluation by query type.

Failure Modes and Troubleshooting

Symptom: answers are fluent but cite irrelevant passages. Cause: vector similarity is retrieving broad topical matches instead of the precise evidence. Diagnostic steps: inspect retrieved chunks before generation, compare vector-only and keyword-only ranks, and check whether the gold source appears in the top twenty candidates. Correction: improve chunk boundaries, increase candidate count, add keyword fusion, or rerank with a query-document relevance model.

Symptom: queries containing product codes or error strings miss the correct document. Cause: embeddings blur rare exact tokens, and tokenization may split identifiers strangely. Diagnostic steps: run the query through lexical search, test quoted exact matches, and inspect preprocessing for lowercasing or punctuation removal. Correction: preserve identifiers in chunk text, add keyword retrieval, boost exact matches, and store normalized variants in metadata.

Symptom: a user receives information from another customer or workspace. Cause: tenant filtering was applied after retrieval or only in the application layer. Diagnostic steps: log filter clauses, test with adversarial cross-tenant queries, and verify the vector store enforces filters during search. Correction: make tenant and permission filters mandatory, pre-filter candidates, and fail closed when identity context is missing.

Symptom: retrieval quality drops after a content refresh. Cause: the index contains mixed embedding models, stale chunks, or duplicate old versions. Diagnostic steps: compare chunk version metadata, embedding model ids, source checksums, and deletion logs. Correction: re-embed consistently, use idempotent upserts, delete removed source chunks, and keep a rollback copy of the prior index.

Security, Performance, and Reliability

Security starts with metadata. Every chunk should carry enough information to enforce access: tenant, source, classification, document owner, and effective permissions where practical. Do not rely on generation instructions to hide retrieved secrets. The retriever should return only evidence the caller may see.

Performance depends on candidate volume, index type, filter selectivity, and reranking cost. Approximate vector indexes are fast because they search a navigable subset of vectors, but recall can fall when filters are very selective. Cache embeddings for repeated queries when appropriate, but include the model id and normalization settings in the cache key.

Reliability requires reproducible ingestion. Store source ids, chunk ids, checksums, embedding model ids, timestamps, and parser versions. Without those, you cannot explain why a passage appeared or disappeared. Retrieval evaluation should run before major index changes and after source migrations.

Hands-on Lab

Prerequisites: Python 3, a shell, and no external services. This lab builds an in-memory hybrid retriever so you can inspect every score. Create a temporary file, paste the code, and run it with Python.

from math import sqrt

DOCS = [
    {"id": "rotate-keys", "tenant": "acme", "text": "Rotate API keys by creating a new key, deploying it, then revoking the old key."},
    {"id": "reset-password", "tenant": "acme", "text": "Password reset sends a one-time link to the verified email address."},
    {"id": "token-error", "tenant": "acme", "text": "ERR_AUTH_013 means the admin token expired and must be regenerated."},
    {"id": "foreign-token", "tenant": "globex", "text": "Admin token rotation for a different tenant."},
]

VOCAB = ["rotate", "api", "key", "password", "token", "expired", "admin"]

def embed(text):
    words = text.lower().replace(".", "").replace(",", "").split()
    return [words.count(term) for term in VOCAB]

def cosine(a, b):
    left = sqrt(sum(x * x for x in a))
    right = sqrt(sum(y * y for y in b))
    if left == 0 or right == 0:
        return 0.0
    return sum(x * y for x, y in zip(a, b)) / (left * right)

def lexical(query, text):
    terms = query.lower().split()
    lower = text.lower()
    return sum(term in lower for term in terms) / len(terms)

def search(query, tenant):
    query_vector = embed(query)
    rows = []
    for doc in DOCS:
        if doc["tenant"] != tenant:
            continue
        semantic = cosine(query_vector, embed(doc["text"]))
        keyword = lexical(query, doc["text"])
        score = 0.6 * semantic + 0.4 * keyword
        rows.append((round(score, 3), doc["id"], round(semantic, 3), round(keyword, 3)))
    return sorted(rows, reverse=True)

for row in search("admin token expired", "acme"):
    print(row)

Steps: run the script once, confirm that token-error ranks first, then change the tenant argument from acme to globex. The Acme-only evidence should disappear. Next, change the query to rotate key and observe that rotate-keys rises because both semantic vocabulary and lexical terms match.

Verification: for admin token expired in tenant acme, the first printed tuple should contain token-error. The tuple also exposes final score, document id, semantic score, and keyword score, which is the minimum diagnostic information you need when tuning. Cleanup: delete the temporary file. There is no persistent index or external account to roll back.

Assessment Exercises

• A query for ERR_BILLING_4027 returns a general billing policy instead of the error reference. Which retrieval signal is probably too weak, and how would you prove it?
• You increase top_k from 5 to 40 and answer quality improves, but latency doubles. Design a candidate retrieval and reranking plan that preserves recall while controlling prompt size.
• A policy document is split into chunks that separate exceptions from the rule they modify. What chunking or metadata strategy would reduce misleading retrieval?
• Explain why post-filtering retrieved results by tenant can produce both security and recall problems.
• Given an evaluation set with expected source chunks, which metric would you use to detect whether the right evidence is reachable before generation?

Summary

Vector search retrieves by meaning, keyword search retrieves by exact language, and metadata filters enforce eligibility. Hybrid retrieval combines those mechanisms so generative AI systems can answer from the right evidence instead of the nearest-sounding evidence. The practical skill is not only calling a vector database; it is designing chunking, filters, score fusion, reranking, diagnostics, and evaluation so retrieval failures are visible before they become generated misinformation.