Vector Databases Overview

A vector database stores embeddings (lists of numbers that capture the meaning of text) and finds the ones closest to a query embedding. It is the memory of a RAG (Retrieval-Augmented Generation) system: it lets your app fetch the most relevant chunks of your own data and give them to the LLM. In this lesson you learn how it works, what to look for when choosing one, and how LangChain hides the differences behind one interface.

Overview: how a vector database works

An embedding model turns a piece of text into a fixed-length list of floating-point numbers called a vector. Texts with similar meaning produce vectors that point in similar directions. “How do I reset my password?” and “I forgot my login credentials” share almost no words, yet their vectors are close.

A vector database stores three things for each item:

• The vector itself (for example 1536 numbers, depending on the embedding model).
• The original content (the text chunk), so you can show it to the LLM later.
• Metadata, such as source file, page number, language or user ID, used for filtering.

At query time, the app embeds the user question with the same model, then asks the database for the k nearest neighbours: the k stored vectors most similar to the query vector. The matching chunks are pasted into the prompt, and the LLM answers using them.

Measuring “closeness”

Three distance or similarity measures are common:

Always use the metric your embedding model was designed for. The model’s documentation tells you which one.

Exact vs approximate search

The simplest search compares the query to every stored vector (“brute force” or “flat” search). It is exact, but the cost grows with the number of vectors. For millions of vectors, databases use approximate nearest neighbour (ANN) indexes. They trade a little accuracy (sometimes the true best match is missed) for large speed gains. Common index families are:

• HNSW (Hierarchical Navigable Small World): a layered graph of vectors; search walks the graph towards the query. Fast and popular.
• IVF (inverted file): vectors are grouped into clusters; search looks only in the nearest few clusters.
• Quantisation: vectors are compressed to save memory, at some cost in precision.

Index settings are tuned per database. Check each product’s documentation for the current parameter names and defaults.

Kinds of vector storage

• In-memory (LangChain’s InMemoryVectorStore): no setup, lost when the process ends. Good for learning and tests.
• Local libraries and embedded databases (FAISS, Chroma): run inside your app or on one machine and can save to disk.
• Extensions to existing databases (for example pgvector for PostgreSQL): you keep one database for both normal data and vectors.
• Dedicated services (such as Pinecone, Qdrant, Weaviate, Milvus): built for large scale, with managed hosting options. Features and pricing change often, so check the vendor site.

Key concepts: LangChain’s VectorStore interface

LangChain defines a base class, VectorStore, in langchain_core.vectorstores. Every integration implements the same core methods, so your RAG code does not depend on one vendor.

store = SomeVectorStore(embedding=embeddings)   # backend-specific setup
store.add_documents(docs)                       # embed + save
store.similarity_search(query, k=4)             # returns list[Document]
store.similarity_search_with_score(query, k=4)  # returns (Document, score) pairs
retriever = store.as_retriever(search_kwargs={"k": 4})

• add_documents / add_texts: calls the embedding model for you, then stores vectors, text and metadata. It returns the IDs.
• similarity_search: embeds the query and returns the nearest Document objects.
• similarity_search_with_score: also returns a score. Whether a bigger or smaller score is better depends on the backend (distance vs similarity), so read the backend’s docs.
• as_retriever: wraps the store as a Retriever, a Runnable that plugs into LCEL chains with the | operator. You can also request MMR (maximal marginal relevance) search for more diverse results with search_type="mmr".

The store holds a reference to an embedding object (Embeddings). That object has embed_documents for many texts and embed_query for one query.

Examples

Example 1: a vector search in plain Python

To remove the mystery, here is a tiny vector “database” with 3-number vectors and cosine similarity. Real embeddings are just longer.

import math


def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    na = math.sqrt(sum(x * x for x in a))
    nb = math.sqrt(sum(y * y for y in b))
    return dot / (na * nb)


store = [
    ("doc-a", [1.0, 0.0, 0.0]),
    ("doc-b", [1.0, 1.0, 0.0]),
    ("doc-c", [0.0, 0.0, 1.0]),
]
query = [1.0, 1.0, 0.0]

ranked = sorted(store, key=lambda item: cosine(query, item[1]), reverse=True)
for doc_id, vec in ranked[:2]:
    print(doc_id, round(cosine(query, vec), 4))

Output:

doc-b 1.0
doc-a 0.7071

The query points in exactly the same direction as doc-b, so its cosine is 1.0. doc-a is 45 degrees away (cosine about 0.7071). doc-c is at 90 degrees (cosine 0), so it is cut by [:2]. A real database does the same job, but with indexes so it does not scan everything.

Example 2: LangChain’s in-memory store, offline

This example needs no API key. DeterministicFakeEmbedding makes a repeatable pseudo-random vector from the text. It has no real meaning, so the ranking is not semantically useful, but the API flow is identical to production.

from langchain_core.documents import Document
from langchain_core.embeddings import DeterministicFakeEmbedding
from langchain_core.vectorstores import InMemoryVectorStore

embeddings = DeterministicFakeEmbedding(size=64)
store = InMemoryVectorStore(embedding=embeddings)

docs = [
    Document(page_content="Reset your password from the login page.",
             metadata={"source": "faq", "lang": "en"}),
    Document(page_content="Refunds are processed within 7 days.",
             metadata={"source": "policy", "lang": "en"}),
    Document(page_content="Contact support by email for billing issues.",
             metadata={"source": "faq", "lang": "en"}),
]

ids = store.add_documents(docs)
print(len(ids))

results = store.similarity_search("How do I reset my password?", k=2)
print(len(results))

retriever = store.as_retriever(search_kwargs={"k": 2})
print(len(retriever.invoke("refund policy")))

Output:

3
2
2

Three documents were stored and IDs were generated. A search with k=2 returns two Document objects. The retriever is a Runnable, so .invoke() works and you can place it in a chain: {"context": retriever, "question": RunnablePassthrough()} | prompt | llm. To get meaningful results, replace the fake embeddings with a real model.

Example 3: a persistent Chroma store with real embeddings

For a real project you want vectors saved to disk. Chroma is an easy embedded database. Install the packages, then set the API key as an environment variable (never write keys in code).

pip install langchain-chroma langchain-openai
export OPENAI_API_KEY="your-key-here"

from langchain_chroma import Chroma
from langchain_core.documents import Document
from langchain_openai import OpenAIEmbeddings

# Reads OPENAI_API_KEY from the environment.
# Check the provider docs for the current embedding model names.
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

store = Chroma(
    collection_name="support_docs",
    embedding_function=embeddings,
    persist_directory="./chroma_db",
)

store.add_documents([
    Document(page_content="Refunds are processed within 7 days.",
             metadata={"source": "policy"}),
    Document(page_content="Reset your password from the login page.",
             metadata={"source": "faq"}),
])

hits = store.similarity_search("How long do refunds take?", k=1,
                               filter={"source": "policy"})
print(hits[0].page_content)

Output (illustrative, needs a valid key and network):

Refunds are processed within 7 days.

Data is written to ./chroma_db, so a second run of the program can open the same collection without re-embedding. The filter argument uses metadata to narrow the search before ranking. Filter syntax differs between backends, so check the docs of the one you use.

Under the hood: what happens on add and search

1. Add: the store calls embeddings.embed_documents([...]) with all the texts. The model returns one vector per text.
2. The store writes vector, text, metadata and an ID. The ANN index is updated (or rebuilt later, depending on the backend).
3. Search: the store calls embeddings.embed_query(query) to get the query vector.
4. The index finds the nearest vectors using the chosen metric. Metadata filters are applied before or during the search.
5. The store maps the winning IDs back to text and metadata and returns Document objects.

Because step 3 uses the same embedding model as step 1, vectors live in the same “space” and can be compared.

Choosing a vector database

• Prototype or tests: InMemoryVectorStore, or a local FAISS or Chroma store.
• Already use PostgreSQL: consider pgvector, so vectors sit next to your relational data and you can join and filter with SQL.
• Large scale, many users: a dedicated or managed service gives scaling, replication and monitoring.
• Compare: metadata filtering power, hybrid (keyword + vector) search, update and delete support, hosting cost, and the quality of the LangChain integration.

Common Mistakes

1. Using different embedding models for indexing and querying

# WRONG: index built with one model, queried with another
index_embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
query_embeddings = OpenAIEmbeddings(model="text-embedding-3-large")

Different models produce vectors in different spaces (often with different lengths). Distances become meaningless or the database raises a dimension error. Fix: use one embedding object for both, and re-embed all documents if you change the model.

2. Forgetting persistence

# WRONG for production: data disappears when the program exits
store = InMemoryVectorStore(embedding=embeddings)

Every restart then re-embeds everything, which costs time and money. Fix: use a persistent backend (Chroma with persist_directory, FAISS saved to disk, pgvector, or a hosted service).

3. Trusting scores without checking their meaning

Some backends return a distance (lower is better), others a similarity (higher is better). Sorting or thresholding the wrong way returns the worst matches. Fix: read the backend documentation and test with a query whose answer you know.

Best Practices

• Keep one embedding model per collection and record its name in your config.
• Store useful metadata (source, page, date, tenant ID) from the start; it is hard to add later.
• Use stable document IDs so re-ingesting a file updates records instead of duplicating them.
• Read API keys and connection strings from environment variables.
• Start with a small k (such as 3 to 5) and tune it by looking at real retrieved chunks.
• Write a small evaluation set of questions with known correct source chunks, and re-run it when you change chunking, the model, or the database.
• Hide the store behind your own small function so swapping backends touches one file.

Practice Exercises

1. Extend Example 1 with a fourth document [0.0, 1.0, 0.0] and a query [0.0, 2.0, 0.0]. Predict the ranking and cosine values, then run it. Notice that vector length does not change cosine similarity.
2. Using DeterministicFakeEmbedding and InMemoryVectorStore, store five short documents with a topic metadata field. Print each result’s metadata after a search. Hint: use similarity_search_with_score and print the pair.
3. Write a function build_store(kind) that returns either an in-memory or a Chroma store, so the rest of your code only calls add_documents and as_retriever.

Summary

• A vector database stores embeddings with text and metadata and finds the nearest ones to a query vector.
• Cosine similarity, dot product and Euclidean distance are the main metrics; match the one your embedding model expects.
• Large databases use approximate indexes such as HNSW or IVF to be fast, at a small accuracy cost.
• LangChain’s VectorStore interface (add_documents, similarity_search, as_retriever) works the same across backends.
• Use the same embedding model for indexing and querying, persist your data, and keep metadata.
• Pick the backend by scale, existing infrastructure and filtering needs; check vendor docs for current features and pricing.