Performance, Caching, and Rate Limits

Published by Programming Line in FastAPI. Last updated .

FastAPI performance work has one practical outcome: keep useful requests fast while preventing one caller or one expensive endpoint from consuming the whole service. In this lesson, performance means measured latency and throughput, caching means deliberately reusing a previous result, and rate limiting means refusing or delaying requests when a caller exceeds a defined budget. These techniques belong together in production FastAPI because ASGI concurrency makes it easy to accept many requests, but it does not make CPU work, database calls, external APIs, or memory free.

You should be able to explain where time is spent inside a request, choose cache keys and expiry rules, send cache-related HTTP headers, and add a fair rate limit around costly paths.

Request Path And Latency Mechanics

A FastAPI app runs on an ASGI server. The server receives a connection, creates an ASGI scope, and calls the application with request and response events. Starlette routes the path and method. FastAPI resolves dependencies, validates path, query, header, cookie, and body values, calls your endpoint, validates or serializes the response, and returns ASGI messages to the server. The slow part is often blocking file or network I/O inside an async def endpoint, repeated database reads, oversized response serialization, unbounded dependency work, or synchronous CPU-heavy code running on the event loop.

The event loop is the shared scheduler for async tasks. When an endpoint awaits an async database driver, HTTP client, or sleep, the loop can run another request. When an endpoint calls blocking code directly, that request occupies the loop until the call returns. For CPU-bound work, move the work to a worker process, a task queue, or a threadpool when appropriate. For I/O-bound work, prefer async libraries end to end and set timeouts. Measure p50, p95, and p99 latency separately because a low average can hide slow tails.

Caching Anatomy

A cache stores a value under a key for reuse. In FastAPI applications you commonly see three layers. A process-local cache is a Python dictionary or LRU cache inside one worker. It is fast but each worker has its own copy and it disappears on restart. A shared application cache such as Redis works across workers and instances, but adds a network hop and operational dependency. HTTP caching uses response headers such as Cache-Control, ETag, and Last-Modified so clients, browsers, reverse proxies, or CDNs can reuse responses without reaching the application every time.

Good cache keys include every input that changes the result: path parameters, normalized query parameters, tenant, locale, authenticated user when the response is private, and sometimes permission version. A TTL, or time to live, bounds staleness by expiring the entry after a duration. Explicit invalidation removes or replaces entries when underlying data changes. A cache stampede occurs when many requests miss at once and all rebuild the same expensive value. Single-flight locking prevents duplicate rebuilds. Negative caching stores selected misses for a short time.

Rate Limit Anatomy

A rate limit answers three questions: who is the caller, what budget applies, and over what time window. The caller key may be an API key, user id, tenant id, IP address, or a combination. The budget might be requests per minute, concurrent requests, tokens per day, bytes uploaded, or expensive operations per billing period. Fixed windows are simple but allow bursts at window boundaries. Sliding windows are smoother and inspect recent timestamps. Token buckets allow controlled bursts while refilling over time. Leaky buckets drain at a steady rate and are useful when the downstream dependency needs a predictable pace.

FastAPI usually enforces limits in dependencies or middleware. A dependency is convenient when only selected routes are costly. Middleware sees every request and is better for whole-service policy. In multi-worker deployments, process memory is not enough for authoritative limits because each worker sees only part of the traffic. Store counters in Redis, a gateway, or another shared system when the limit must be fair across instances.

Example 1: Establish A Baseline

The first example returns one product from an in-memory mapping. This is the baseline shape you time before adding a cache or a limiter. A successful request to GET /products/sku-1 returns {"found": true, "sku": "sku-1", ...}. A request for an unknown SKU returns {"found": false, "sku": "missing"} rather than raising a server error.

from fastapi import FastAPI

app = FastAPI(title="Latency baseline")

PRODUCTS = {
    "sku-1": {"name": "Notebook", "price": 8.50},
    "sku-2": {"name": "Pen", "price": 2.25},
}

@app.get("/products/{sku}")
async def read_product(sku: str):
    product = PRODUCTS.get(sku)
    if product is None:
        return {"found": False, "sku": sku}
    return {"found": True, "sku": sku, "product": product}

If it is already slow, caching will hide the symptom only for cache hits. The cause may be server configuration, response size, dependency setup, or blocking work outside the shown function.

Example 2: TTL Cache With Single Flight

The second example adds a per-SKU TTL cache. The first request for sku-1 waits for fetch_from_store and returns "source": "origin". A second request within thirty seconds returns the same product with "source": "cache". The lock is per key, so two different SKUs can refresh independently, while twenty concurrent requests for one SKU produce one origin fetch.

import asyncio
import time
from typing import Any

from fastapi import FastAPI, HTTPException

app = FastAPI(title="TTL cache with single flight")

CATALOG = {"sku-1": {"name": "Notebook", "price": 8.50}}
_cache: dict[str, tuple[float, dict[str, Any]]] = {}
_locks: dict[str, asyncio.Lock] = {}
TTL_SECONDS = 30.0

async def fetch_from_store(sku: str) -> dict[str, Any]:
    await asyncio.sleep(0.05)
    if sku not in CATALOG:
        raise HTTPException(status_code=404, detail="unknown sku")
    return CATALOG[sku]

@app.get("/cached-products/{sku}")
async def cached_product(sku: str):
    now = time.monotonic()
    cached = _cache.get(sku)
    if cached and cached[0] > now:
        return {"source": "cache", "sku": sku, "product": cached[1]}

    lock = _locks.setdefault(sku, asyncio.Lock())
    async with lock:
        now = time.monotonic()
        cached = _cache.get(sku)
        if cached and cached[0] > now:
            return {"source": "cache", "sku": sku, "product": cached[1]}
        product = await fetch_from_store(sku)
        _cache[sku] = (time.monotonic() + TTL_SECONDS, product)
        return {"source": "origin", "sku": sku, "product": product}

This implementation is useful for teaching internals, not as the final shared cache for a fleet. With multiple Uvicorn workers, each worker has its own _cache. After a deploy, all entries are gone. Under heavy cardinality, unbounded dictionaries can grow until the process runs out of memory. Production versions add maximum size, eviction, hit-rate metrics, refresh timing, and a shared backend when consistency across instances matters.

Example 3: Per-Client Sliding Window Rate Limit

The third example limits each client to three requests per minute on a report route. The caller key prefers x-api-key and falls back to the remote address. The dependency removes old timestamps, checks the remaining budget, and raises 429 when the caller is out of quota. The first three requests return 200 and include remaining-budget headers. The fourth request in the same minute returns {"detail": "rate limit exceeded"} with a Retry-After header.

import time
from collections import defaultdict, deque

from fastapi import Depends, FastAPI, HTTPException, Request, Response

app = FastAPI(title="Per-client sliding window limit")
WINDOW_SECONDS = 60.0
MAX_REQUESTS = 3
hits: dict[str, deque[float]] = defaultdict(deque)

def client_key(request: Request) -> str:
    return request.headers.get("x-api-key") or request.client.host

async def enforce_limit(request: Request, response: Response) -> None:
    key = client_key(request)
    now = time.monotonic()
    bucket = hits[key]
    while bucket and bucket[0] <= now - WINDOW_SECONDS:
        bucket.popleft()
    remaining = MAX_REQUESTS - len(bucket)
    if remaining <= 0:
        retry_after = max(1, int(WINDOW_SECONDS - (now - bucket[0])))
        response.headers["Retry-After"] = str(retry_after)
        raise HTTPException(status_code=429, detail="rate limit exceeded")
    bucket.append(now)
    response.headers["X-RateLimit-Limit"] = str(MAX_REQUESTS)
    response.headers["X-RateLimit-Remaining"] = str(remaining - 1)

@app.get("/reports/{report_id}", dependencies=[Depends(enforce_limit)])
async def report(report_id: str):
    return {"report_id": report_id, "status": "ready"}

The important design choice is that the protected work is never called when the limit fails. Put the dependency before expensive database queries, report rendering, model calls, or third-party API calls. For public endpoints behind proxies, configure trusted proxy handling before relying on IP addresses; otherwise many users may appear to come from one proxy address, or a spoofed header may bypass policy.

Example 4: HTTP Cache Headers

Application caches save server work. HTTP cache headers let clients and intermediaries participate. This example returns an ETag and marks the response private for thirty seconds, which means a browser may reuse it for that user, but a shared proxy should not serve it to other users. The deterministic output printed by the script is private, max-age=30.

from fastapi import FastAPI, Response
from fastapi.testclient import TestClient

app = FastAPI(title="Conditional response")
DOCUMENT = {"version": "v3", "body": "quarterly totals"}

@app.get("/documents/current")
async def current_document(response: Response):
    response.headers["ETag"] = DOCUMENT["version"]
    response.headers["Cache-Control"] = "private, max-age=30"
    return DOCUMENT

client = TestClient(app)
first = client.get("/documents/current")
assert first.status_code == 200
assert first.headers["etag"] == "v3"
assert first.json()["body"] == "quarterly totals"
print(first.headers["cache-control"])

A complete conditional GET also reads If-None-Match and returns 304 Not Modified when the tag still matches. That optimization is valuable for large payloads because the server can avoid resending the body. For personalized responses, use private or no-store rather than public caching. For public catalog pages, a short public, max-age plus an ETag can reduce latency and origin traffic.

Design Choices And Trade-Offs

Start with measurement. Add middleware or ASGI server metrics that record route, status family, and latency buckets without logging secrets. Optimize the highest-cost path that affects users. If the bottleneck is a repeated read, cache it. If the bottleneck is one abusive caller or accidental polling loop, rate limit it. If the bottleneck is blocking code, replace it with async I/O or move it away from the event loop.

Choose TTL length by business tolerance for stale data. Prices, account balances, and authorization decisions need short TTLs or explicit invalidation. Documentation pages, feature flags, and product metadata can tolerate longer TTLs. Choose cache location by consistency and cost. In-memory caches have the lowest latency but weak coordination. Redis adds coordination, atomic counters, and persistence options, but it can become a dependency whose outage affects every request unless you define fallback behavior.

Rate limits are product policy as much as engineering policy. A free API key, a paid tenant, and an internal batch process usually deserve different budgets. Return clear 429 responses so clients can back off. Use Retry-After when the next attempt time is known. Avoid applying the same strict limit to health checks, login attempts, and report generation; they have different security and reliability goals.

Failure Modes And Troubleshooting

Symptom: p95 latency spikes while CPU stays low. Likely cause: blocking network or file I/O inside async endpoints. Diagnose: compare access logs by route, inspect slow traces, and search for synchronous clients called from async def. Correct: use async clients with timeouts, wrap unavoidable blocking calls in a threadpool, or move long jobs to a background worker.

Symptom: users see stale data after an update. Likely cause: the cache key omitted tenant or query parameters, or the update path did not invalidate affected entries. Diagnose: log cache key components, compare origin data to cached data, and reproduce with two tenants or two query variants. Correct: include all result-changing inputs, shorten TTL for risky objects, and invalidate on write.

Symptom: many requests become slow exactly when a popular entry expires. Likely cause: cache stampede. Diagnose: look for simultaneous cache misses and repeated origin calls for the same key. Correct: add per-key single-flight, refresh ahead of expiry, jitter TTLs, or serve stale data briefly while one refresh is running.

Symptom: legitimate users receive 429. Likely cause: limits keyed by proxy IP, too small a window, or process-local counters differing across workers. Diagnose: inspect the caller key, headers, worker count, and remaining-budget headers. Correct: key by authenticated identity where possible, configure trusted proxies, tune budgets from observed traffic, and use shared counters.

Security, Performance, And Reliability Implications

Caches can leak data when private responses use public keys or public headers. Never cache a user-specific response under only the URL unless the URL includes an unguessable per-user component and the cache is private. Rate limits can also become a denial-of-service amplifier if an attacker can force many unique keys and fill memory. Bound key cardinality, expire idle counters, and use shared infrastructure designed for eviction.

Performance improvements should include failure behavior. If Redis is unavailable, decide whether the endpoint should fail closed, bypass the cache, or serve stale data. Authorization and payment limits often fail closed. Read-only catalog caches may fail open and hit the origin. Record cache hit rate, miss latency, limiter allows, limiter rejects, and backend errors so production behavior is explainable.

Hands-On Lab

Prerequisites: Python, FastAPI, an ASGI server such as Uvicorn, and curl or another HTTP client. Create a temporary app file with the TTL cache example. Run it with uvicorn app:app --reload from that directory.

  1. Request /cached-products/sku-1 once and confirm the JSON contains "source": "origin".
  2. Request the same URL again within thirty seconds and confirm the source changes to "cache".
  3. Open several terminals and request the same uncached SKU at the same time after clearing the process or changing the SKU in CATALOG. Confirm only the first rebuild should need origin work for that key.
  4. Add the rate-limit example to the same app or a second temporary app. Call /reports/monthly four times with the same x-api-key.
  5. Verify that the first three calls return 200, the fourth returns 429, and the response includes Retry-After.

Cleanup: stop Uvicorn, delete the temporary file, and clear any local terminal history that contains real API keys. If you used Redis instead of the in-memory dictionaries, remove the test keys or use a dedicated development database and flush only that database.

Assessment Exercises

  • A route is slow because it calls a payment provider on every page load. What would you measure before caching, and which responses should never be cached?
  • Design a cache key for GET /tenants/{tenant_id}/orders?status=open&page=2. Which parts are required to prevent cross-tenant or wrong-page results?
  • Explain why a process-local rate limiter can undercount traffic when the app runs with four workers. What shared primitive would fix it?
  • A cache hit rate is high but users complain about stale prices. Which TTL, invalidation, and header changes would you consider?
  • For an expensive report endpoint, compare a sliding-window request limit with a concurrent-request limit. Which failure does each one prevent?

Summary

FastAPI performance engineering starts by locating the slow operation in the ASGI request path. Caching improves repeated reads when keys, TTLs, invalidation, and privacy rules match the data. Rate limiting protects expensive work by assigning callers a budget before the work starts. The production skill is choosing the smallest mechanism that addresses the measured bottleneck, then verifying normal and failure behavior under realistic traffic.