Storing and Searching Embeddings

A vector store is a database built to save embeddings (lists of numbers that represent the meaning of text) and to find the ones closest to a query. It is the memory of a RAG (Retrieval-Augmented Generation) app. Without it, you would have to compare your question with every document by hand on each request. In this lesson you will learn how storing and searching works, how LangChain wraps it in one common interface, and how to use it safely.

Overview: how it works

An embedding model turns a piece of text into a vector, for example 1536 floating point numbers. Texts with similar meaning get vectors that point in similar directions. A vector store keeps three things for every record:

• the vector (the embedding),
• the original text (the page content),
• the metadata (a dictionary such as source file, page number or language) and an id.

Storing happens in two steps. First the store calls your embedding model on each text. Then it writes vector, text, metadata and id into its index. Searching repeats the first step for the query: the query text is embedded with the same model, and the store returns the records whose vectors are closest to the query vector.

What does closest mean?

Closeness is measured by a distance or similarity metric. The most common is cosine similarity, which compares the angle between two vectors (1 means same direction, 0 unrelated, -1 opposite). Others are Euclidean (L2) distance and dot product. Different stores use different defaults, so a “score” from one store cannot be compared directly with a score from another. Some stores return a similarity (higher is better), others return a distance (lower is better). Always check the documentation of your store.

Exact and approximate search

The simplest search compares the query with every stored vector. This is exact, but slow for millions of records. Production databases build an approximate nearest neighbour (ANN) index, such as HNSW, that finds very close matches much faster and may occasionally miss a true neighbour. For learning and small datasets, exact search is fine.

One interface for many databases

LangChain defines a common VectorStore base class in langchain_core.vectorstores. Chroma, FAISS, Pinecone, Qdrant, pgvector and others implement it through partner packages. Because the method names are the same, you can start with an in-memory store and later switch to a hosted database by changing only the line that creates the store.

Syntax / Key concepts

store = SomeVectorStore(embedding=embeddings, ...)
ids   = store.add_documents(documents, ids=[...])
hits  = store.similarity_search(query, k=4, filter=...)
pairs = store.similarity_search_with_score(query, k=4)
store.delete(ids=[...])
retriever = store.as_retriever(search_kwargs={'k': 4})

Examples

Example 1: store and search offline

To follow along without an API key we use DeterministicFakeEmbedding, which creates a repeatable pseudo-random vector from the text. It is only for tests: its vectors carry no real meaning, so the ranking of results is not meaningful. The InMemoryVectorStore keeps everything in a Python dictionary.

from langchain_core.documents import Document
from langchain_core.embeddings import DeterministicFakeEmbedding
from langchain_core.vectorstores import InMemoryVectorStore

embeddings = DeterministicFakeEmbedding(size=64)
store = InMemoryVectorStore(embedding=embeddings)

docs = [
    Document(page_content='Reset your password from the login page.',
             metadata={'source': 'faq.md'}),
    Document(page_content='Refunds are processed within 7 days.',
             metadata={'source': 'policy.md'}),
    Document(page_content='Contact support by email or chat.',
             metadata={'source': 'support.md'}),
]

ids = store.add_documents(docs)
print(len(ids))

results = store.similarity_search('How do I reset my password?', k=2)
print(len(results))
print(type(results[0]).__name__)

Output:

3
2
Document

Three documents were embedded and stored, and the store generated three ids for us. The search embedded the question, compared it with the three vectors and returned the top 2 as Document objects. With a real embedding model, the password document would rank first. With the fake model the order is arbitrary, which is why we print only counts and types.

Example 2: scores, metadata filter, retriever and delete

Now we give our own ids, add a lang field to the metadata, and use more features. For InMemoryVectorStore the filter is a Python function that receives a Document and returns True to keep it. Other stores use their own filter syntax (often a dictionary), so read the store’s documentation.

from langchain_core.documents import Document
from langchain_core.embeddings import DeterministicFakeEmbedding
from langchain_core.vectorstores import InMemoryVectorStore

store = InMemoryVectorStore(embedding=DeterministicFakeEmbedding(size=64))

docs = [
    Document(page_content='Our refund policy', metadata={'lang': 'en'}),
    Document(page_content='Shipping takes three days', metadata={'lang': 'en'}),
    Document(page_content='Votre politique de remboursement', metadata={'lang': 'fr'}),
    Document(page_content='Contact our support team', metadata={'lang': 'en'}),
]
store.add_documents(docs, ids=['d1', 'd2', 'd3', 'd4'])

english = store.similarity_search(
    'refund', k=10, filter=lambda doc: doc.metadata['lang'] == 'en'
)
print(len(english))

pairs = store.similarity_search_with_score('refund', k=4)
print(all(-1.0001 <= score <= 1.0001 for _, score in pairs))

retriever = store.as_retriever(search_kwargs={'k': 2})
print(len(retriever.invoke('refund policy')))

store.delete(ids=['d1'])
print(len(store.similarity_search('anything', k=10)))

Output:

3
True
2
3

The filter removed the French document, so only 3 English documents could be returned even though we asked for up to 10. In this store the score is cosine similarity, so it always lies between -1 and 1. The retriever is a Runnable: calling invoke with a string returns a list of documents, and you can put it in a chain with the | operator. After deleting d1, 3 records remain.

Example 3: a persistent store with real embeddings

An in-memory store disappears when the program ends. Chroma can save to a folder on disk. This example needs a package install and an API key, so its output is illustrative.

pip install langchain-chroma langchain-openai
export OPENAI_API_KEY='your-key-here'

from langchain_chroma import Chroma
from langchain_core.documents import Document
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model='text-embedding-3-small')
store = Chroma(
    collection_name='faq',
    embedding_function=embeddings,
    persist_directory='./chroma_db',
)

store.add_documents([
    Document(page_content='Reset your password from the login page.',
             metadata={'source': 'faq.md'}),
    Document(page_content='Refunds are processed within 7 days.',
             metadata={'source': 'policy.md'}),
])

for doc, score in store.similarity_search_with_score('forgot my password', k=1):
    print(doc.page_content, '|', doc.metadata['source'], '|', round(score, 3))

Reset your password from the login page. | faq.md | 0.742

The OpenAI key is read from the environment variable OPENAI_API_KEY, never written in code. The exact score above is only an example. Chroma returns a distance (lower is closer), so do not compare it with the cosine scores from Example 2. Model names change over time, so check your provider’s documentation for current names.

Under the hood: step by step

1. Add: add_documents collects the page_content of every document.
2. Embed: the store calls embeddings.embed_documents(texts) once for the whole batch, which returns one vector per text.
3. Write: vectors, texts, metadata and ids are saved in the index. If an id already exists, most stores overwrite that record.
4. Search: similarity_search calls embeddings.embed_query(query) to get the query vector.
5. Compare: the store scores the query vector against stored vectors (all of them, or ANN candidates), applies the metadata filter and keeps the top k.
6. Return: it rebuilds Document objects from the saved text and metadata.

Common Mistakes

Mistake 1: using different embedding models for storing and searching

store = Chroma(collection_name='faq', embedding_function=OpenAIEmbeddings(model='text-embedding-3-small'), persist_directory='./chroma_db')
store.add_documents(docs)

# later, in another script, a different model is used
store = Chroma(collection_name='faq', embedding_function=OpenAIEmbeddings(model='text-embedding-3-large'), persist_directory='./chroma_db')
store.similarity_search('password')

Vectors from different models live in different spaces, and they often have different lengths. The search will fail with a dimension error, or worse, return meaningless results when the lengths happen to match. Fix: use the same model for the whole collection. If you change the model, re-embed all documents into a new collection.

Mistake 2: adding the same documents on every run

store = Chroma(collection_name='faq', embedding_function=embeddings, persist_directory='./chroma_db')
store.add_documents(docs)   # runs on every start of the app

With a persistent store and no ids, each start adds duplicate records. Search then returns the same text several times and wastes your context window. Fix: give each document a stable id (for example a hash of its source and chunk number) so a repeat run overwrites instead of duplicating, or check whether the collection is already filled.

Mistake 3: treating fake embeddings as real

DeterministicFakeEmbedding is great for tests, but results from it are not based on meaning. Never judge the quality of your retrieval with it.

Best Practices

• Use one embedding model per collection, and record its name next to your data.
• Set your own stable ids so you can update and delete records exactly.
• Store useful metadata (source, page, date, language, access level) at indexing time; you cannot filter on what you did not save.
• Add documents in batches, because embedding calls over a network are the slow and costly part.
• Choose k deliberately. A small value such as 3 to 5 keeps the prompt short; raise it if answers miss facts.
• Use the in-memory store for tests and tutorials, and a persistent or hosted store for production.
• Do not compare scores across different stores or metrics; check whether higher or lower is better.
• Keep API keys in environment variables and out of source control.
• Evaluate retrieval on a few real questions before you connect the LLM.

Practice Exercises

1. Create an InMemoryVectorStore with DeterministicFakeEmbedding(size=32). Add five documents, each with a topic metadata field. Search with a filter that keeps only one topic and print how many documents come back.
2. Add three documents with ids a, b, c. Delete b, then confirm that a search with k=10 returns two documents. Hint: use store.delete(ids=[...]).
3. Turn your store into a retriever with k=1 and call invoke on it. Print the type of the returned value and its length.

Summary

• A vector store saves vectors with text, metadata and ids, and finds the closest vectors to an embedded query.
• Documents and queries must be embedded with the same model.
• LangChain gives all stores one interface: add_documents, similarity_search, similarity_search_with_score, delete and as_retriever.
• Metadata filters narrow results, but their syntax depends on the store.
• Scores differ between stores: some are similarities, some are distances.
• Use stable ids to avoid duplicates, and persistent storage when data must survive restarts.
• Fake embeddings let you test the code offline, but not the quality of retrieval.