Caching, Latency and Cost

A RAG app that works in a demo can still be slow and expensive in production. Every question may trigger an embedding call, a vector search and a large language model (LLM) call, and each step costs time and money. This lesson shows where the time and money go, how to cache repeated work, and how to measure the result so you can improve it with evidence.

Overview: where latency and cost come from

A typical RAG request has these stages: embed the question, search the vector store, build a prompt from the retrieved chunks, call the LLM, and return the answer. Latency (how long the user waits) and cost (what you pay the provider) are both dominated by a few of these stages.

• LLM generation is usually the slowest stage. Time grows with the number of output tokens, because the model writes one token at a time.
• Prompt size drives input cost. Providers bill per token (a token is a small piece of text, roughly three-quarters of an English word). If you retrieve 8 chunks instead of 4, you roughly double the context tokens you pay for on every request.
• Embedding calls happen at ingestion time (every chunk) and at query time (every question). Re-indexing an unchanged document wastes both.
• Vector search is normally fast compared with the LLM, but a slow remote database or a large re-ranking step can matter.

Caching means storing the result of an expensive call so that the same call next time is answered from storage. It helps most when your inputs repeat: the same FAQ question, the same document chunk, the same prompt prefix. It does nothing for inputs that never repeat, so measure before you build.

Kinds of caching you will meet

The first two are built into LangChain and are the focus here. Semantic caches use an embedding comparison, so they can return a wrong answer for a question that only looks similar; use them with care. Provider prompt caching is a feature of some APIs that reduces the price or time of repeated long prefixes. How it is enabled, and its pricing, differ by provider and change over time, so read the current provider documentation.

Syntax and key concepts

from langchain_core.globals import set_llm_cache
from langchain_core.caches import InMemoryCache

set_llm_cache(InMemoryCache())

• set_llm_cache(cache) installs a global cache. Every LLM and chat model call that does not opt out will check it first.
• InMemoryCache keeps entries in process memory. It is fast and needs no setup, but it is lost on restart and not shared between workers.
• Persistent caches exist in langchain_community.cache, for example SQLiteCache. Redis-based caches are available through the community and partner packages.
• CacheBackedEmbeddings.from_bytes_store(underlying, store, namespace=...) wraps an embedding model. It looks each text up in a key-value byte store and only calls the real model for misses.
• The namespace is mixed into the cache key. Set it to the embedding model name so vectors from different models never mix.

Examples

Example 1: an LLM cache you can run offline

We use a fake LLM that returns a fixed list of answers, one per real call. This lets us see exactly when the model is really called, with no API key.

from langchain_core.caches import InMemoryCache
from langchain_core.globals import set_llm_cache
from langchain_core.language_models.fake import FakeListLLM

set_llm_cache(InMemoryCache())
llm = FakeListLLM(responses=['answer one', 'answer two', 'answer three'])

print(llm.invoke('What is RAG?'))
print(llm.invoke('What is RAG?'))
print(llm.invoke('What is a retriever?'))
print(llm.invoke('What is RAG?'))

Output:

answer one
answer one
answer two
answer one

The first call is a miss, so the fake model runs and returns its first response. The second call has the same prompt, so the cache answers and the model is not touched. The third prompt is new, so the model runs again and returns its second response. The fourth call is a hit again. With a real model, each hit saves both the latency and the token cost of that call.

Example 2: caching embeddings

Re-embedding unchanged chunks is pure waste. Here a counting embedding class shows how many texts really reach the underlying model.

from langchain.embeddings import CacheBackedEmbeddings
from langchain_core.embeddings import Embeddings
from langchain_core.stores import InMemoryByteStore


class CountingEmbeddings(Embeddings):
    def __init__(self):
        self.texts_embedded = 0

    def embed_documents(self, texts):
        self.texts_embedded += len(texts)
        return [[float(len(t)), 1.0, 0.0, 0.0] for t in texts]

    def embed_query(self, text):
        return self.embed_documents()[0]


underlying = CountingEmbeddings()
cached = CacheBackedEmbeddings.from_bytes_store(
    underlying, InMemoryByteStore(), namespace='demo-embed-v1'
)

cached.embed_documents(['alpha', 'beta', 'gamma'])
print('after first call:', underlying.texts_embedded)
vectors = cached.embed_documents(['beta', 'gamma', 'delta'])
print('after second call:', underlying.texts_embedded)
print(vectors[0])

Output:

after first call: 3
after second call: 4
[4.0, 1.0, 0.0, 0.0]

The first call embeds all three texts. In the second call, beta and gamma come from the store and only delta reaches the model, so the counter moves from 3 to 4. In production, replace InMemoryByteStore with LocalFileStore (from langchain.storage) or a Redis-backed store so the cache survives restarts. Note that by default only document embeddings are cached; query embeddings are cached only if you pass query_embedding_cache to from_bytes_store.

Example 3: a persistent cache

For a single-server app, a SQLite file is a simple persistent LLM cache. It needs the langchain-community package.

from langchain_community.cache import SQLiteCache
from langchain_core.globals import set_llm_cache

set_llm_cache(SQLiteCache(database_path='.langchain_cache.db'))

After this line, repeated prompts are answered from the file even after you restart the program. Add the file to .gitignore: cached answers may contain user or document text.

Example 4: estimating cost before you ship

Cost is simple arithmetic: tokens multiplied by price. The prices below are placeholders, not real rates. Always read your provider’s current price page.

PRICE_IN = 1.0   # placeholder: currency units per 1 million input tokens
PRICE_OUT = 4.0  # placeholder: currency units per 1 million output tokens


def request_cost(prompt_tokens, completion_tokens, price_in, price_out):
    return (prompt_tokens / 1_000_000 * price_in
            + completion_tokens / 1_000_000 * price_out)


for k in (4, 8):
    prompt_tokens = 200 + k * 250  # question + instructions + k chunks
    cost = request_cost(prompt_tokens, 300, PRICE_IN, PRICE_OUT)
    monthly = cost * 100_000
    print(f'k={k}: prompt={prompt_tokens} tokens, ${cost:.4f} per query, ${monthly:.2f} per 100000 queries')

Output:

k=4: prompt=1200 tokens, $0.0024 per query, $240.00 per 100000 queries
k=8: prompt=2200 tokens, $0.0034 per query, $340.00 per 100000 queries

Doubling the retrieved chunks from 4 to 8 raises the cost by about 40 percent here, because the output tokens stay the same. Cutting k is only safe if answer quality does not drop, so check it with your evaluation set from the earlier lessons.

Under the hood: how the caches decide

The LLM cache key

When you call a model, LangChain builds a key from two parts: the prompt and a string describing the model and its parameters (model name, temperature and similar settings). A lookup is a hit only if both match exactly. That is why changing the temperature or the model name creates a fresh entry, and why one extra space in the prompt is a miss. Cached values are the generations returned by the model, so a hit skips the network call completely.

The embedding cache key

CacheBackedEmbeddings hashes each text and combines the hash with your namespace to form the key. For a batch, it fetches all keys in one call to the store, embeds only the missing texts in one batch call to the model, stores the new vectors, and returns everything in the original order.

Measuring latency stage by stage

You cannot improve what you do not measure. A small wrapper that times each stage of a chain shows where the wait is. The sleeps below stand in for real work.

import time

from langchain_core.runnables import RunnableLambda


def timed(name, fn):
    def wrapper(value):
        start = time.perf_counter()
        result = fn(value)
        print(f'{name}: {(time.perf_counter() - start) * 1000:.0f} ms')
        return result
    return RunnableLambda(wrapper)


def fake_retrieve(question):
    time.sleep(0.05)
    return ['chunk one', 'chunk two']


def fake_generate(docs):
    time.sleep(0.2)
    return 'final answer'


chain = timed('retrieve', fake_retrieve) | timed('generate', fake_generate)
print(chain.invoke('What is caching?'))

Output (illustrative; exact milliseconds vary by machine):

retrieve: 50 ms
generate: 200 ms
final answer

In a real app, swap the fakes for your retriever and model. For full traces across many requests, use a tracing tool such as LangSmith, which records the duration and token counts of every step.

Other ways to cut latency

• Stream the answer. Using chain.stream(...) does not make generation faster, but the user sees the first words sooner, which feels much faster.
• Run independent steps in parallel. RunnableParallel, and batch calls with chain.batch(...), overlap work that does not depend on each other.
• Limit output length. Set a maximum output token setting on the model and ask for concise answers.
• Use a smaller model for simple steps such as query rewriting or classification, and keep the large model for the final answer.
• Trim the context. Fewer, better chunks (through good chunking, a re-ranker or a score threshold) cut both cost and time.

Common Mistakes

Mistake 1: caching answers that depend on the user

set_llm_cache(InMemoryCache())

# The prompt contains private text from this user's documents
chain.invoke({'question': q, 'context': user_private_docs})

The cache is global and keyed only by the prompt. If two users ever produce the same prompt, they share an answer. Worse, private text is now stored in a cache that may be persistent or shared. Fix it by using the LLM cache only for non-sensitive, public-knowledge prompts, by opting out per model where supported (cache=False on the model), or by keeping access control in front of the cache.

Mistake 2: sharing an embedding cache without a namespace

cached = CacheBackedEmbeddings.from_bytes_store(model_a, store)
# later, after switching models, with the same store
cached = CacheBackedEmbeddings.from_bytes_store(model_b, store)

Vectors from different models are not comparable, and they may even have different sizes. Without a namespace you can get stale vectors from the old model and silently broken search. Fix it with namespace=model_name, for example namespace='embed-model-v2', and change it whenever the model changes.

Mistake 3: expecting a cache hit from a changing prompt

If you put a timestamp, a request ID or a random example order into the prompt, every prompt is unique and the hit rate is zero. Keep volatile text out of the prompt, and put stable instructions first so that provider-side prefix caching can also work.

Mistake 4: caching non-deterministic answers and forgetting to expire

A cache returns the first answer forever. If your documents change, old answers go stale. Clear or version the cache when you re-index, and consider a time limit on entries in Redis-style stores.

Best Practices

• Measure first: log latency per stage, token counts per request and the cache hit rate. Optimise the biggest number.
• Always cache embeddings during ingestion, so re-running the pipeline only embeds new or changed chunks.
• Use a persistent, shared cache (Redis or a database) when you run several workers; in-memory caches are per process.
• Put the model name and a version in cache namespaces so upgrades never serve old results.
• Keep prompts stable and put long, unchanging text (instructions, examples) at the start.
• Stream responses to users and set sensible limits on retrieved chunks and output tokens.
• Set budgets and alerts with your provider, and keep price numbers in configuration, not in code, because prices change.
• Re-run your evaluation set after each optimisation to confirm that quality did not fall.
• Do not cache sensitive or personalised outputs unless the cache is protected like your database.

Practice Exercises

1. Hit counter. Using FakeListLLM and InMemoryCache, send ten questions where only four are unique. Count how many real model calls happen. Hint: subclass or wrap the model, or check which response text is returned.
2. Change the namespace. Extend Example 2 so that a second CacheBackedEmbeddings with namespace demo-embed-v2 shares the same byte store. Predict, then verify, how many texts the underlying model embeds when you re-embed alpha, beta and gamma.
3. Budget calculator. Change Example 4 to read prices from environment variables and to print the cost for k from 2 to 10. Which k keeps a query under a cost you choose?

Summary

• RAG latency and cost come mainly from LLM generation, prompt size and repeated embedding work.
• set_llm_cache with InMemoryCache or a persistent cache skips repeated identical LLM calls.
• CacheBackedEmbeddings avoids re-embedding unchanged text; always use a namespace per embedding model.
• Caches only help when inputs repeat, and they can leak or go stale, so control what you store.
• Retrieve fewer, better chunks, limit output length, stream responses and use smaller models for simple steps.
• Estimate cost with token arithmetic, keep prices in config, and check current prices on your provider’s site.
• Measure each stage and re-check quality after every optimisation.