Rate Limiting Backpressure and Load Shedding

Purpose: keeping useful work alive

Rate limiting, backpressure, and load shedding are three related controls for systems that receive more work than they can safely complete. In a system design interview or production design review, they answer different questions. Rate limiting asks, “Should this caller be allowed to start this request now?” Backpressure asks, “How do we slow senders before queues and dependencies collapse?” Load shedding asks, “Which work should we reject or abandon so the remaining work finishes within its deadline?” The outcome is not merely fewer requests. The outcome is bounded latency, predictable dependency use, and graceful degradation instead of a cascading failure.

These controls sit in the networking and API architecture layer because overload usually arrives through an API boundary before it appears as database saturation, thread starvation, or timeout storms. A resilient design places limits at several points: the edge proxy protects the fleet, application middleware protects expensive handlers, queues protect workers, and clients honor retry signals instead of multiplying traffic. The controls are strongest when they are explicit about the resource being protected: requests per user, writes per tenant, concurrent jobs per shard, outstanding bytes on a stream, or CPU-heavy report generations per minute.

Mechanisms and internals

A rate limiter maintains accounting state keyed by a subject. The subject might be an API key, user id, tenant id, source IP, route, or a composite such as tenant:route. On each request the limiter reads the current state, decides whether the request fits the policy, updates the state atomically, and returns allow or deny. Fixed-window counters are simple: count requests in each time bucket, such as one minute. They are cheap but can permit a burst at the boundary between two windows. Sliding-window counters reduce that edge effect by blending the previous and current bucket. Token buckets and leaky buckets model capacity more naturally: tokens refill at a configured rate, requests spend tokens, and short bursts are allowed up to bucket capacity.

Distributed limiters add a harder internal problem: consistency. If every API server keeps its own in-memory counter, a user can multiply their effective limit by spreading requests across servers. A shared store such as Redis can centralize counters, but it adds network latency and a new dependency in the hot path. Many production designs use local fast limits for broad protection and a shared limiter only for stricter tenant or billing policies. Atomic increments with expirations work for fixed windows. Token buckets typically need an atomic read-modify-write script so concurrent requests do not overspend the same token.

Backpressure is feedback from a saturated receiver to an upstream sender. It can be explicit, such as HTTP 429, HTTP 503 with Retry-After, gRPC resource-exhausted responses, bounded queue rejections, TCP receive-window behavior, or a message broker that stops granting credit. It can also be implicit, such as rising latency that causes well-designed clients to reduce concurrency. Backpressure is different from plain failure because it carries a control signal: slow down, retry later, lower concurrency, or stop sending a particular class of work.

Load shedding is deliberate rejection of work when accepting it would harm higher-value work or violate deadlines. It usually uses queue length, age in queue, estimated cost, priority, dependency health, or current error budget burn. A gateway may shed anonymous search requests before paid checkout requests. A worker may drop jobs whose deadlines have already expired. A service may reject requests once its thread pool queue crosses a threshold because waiting longer would only create timeouts and wasted CPU. Good shedding is early and cheap; bad shedding happens after the service has already done most of the expensive work.

Policy anatomy

A useful policy names five things. First is the subject: who or what is being controlled. Second is the resource: request count, concurrency, bytes, rows scanned, or jobs. Third is the time or occupancy rule: for example, 100 requests per minute, 10 concurrent exports, or queue depth below 1,000. Fourth is the behavior when denied: reject, delay, enqueue, downgrade, or serve cached data. Fifth is the client signal: status code, retry delay, remaining quota headers, or stream credit. Without these details, “add rate limiting” is too vague to implement consistently.

For HTTP APIs, denials commonly use 429 Too Many Requests when the caller exceeded a quota and 503 Service Unavailable when the service is temporarily overloaded regardless of caller identity. Retry-After should be set only when the server has a reasonable retry time. Retry clients should use jitter so thousands of clients do not wake at the same instant. Internal APIs often add deadline propagation: if the caller has 80 ms left and the downstream queue delay is already 200 ms, rejecting immediately is better than accepting doomed work.

Example 1: fixed window at the API edge

The simplest limiter counts requests per subject per minute. This example permits three requests for user-7 in a minute, rejects the fourth, and allows another request in the next minute.

from collections import defaultdict

class FixedWindowLimiter:
    def __init__(self, limit):
        self.limit = limit
        self.counts = defaultdict(int)

    def allow(self, subject, timestamp_seconds):
        window = timestamp_seconds // 60
        key = (subject, window)
        if self.counts[key] >= self.limit:
            return False
        self.counts[key] += 1
        return True

limiter = FixedWindowLimiter(limit=3)
results = [limiter.allow('user-7', t) for t in [1, 2, 3, 4, 61]]
print(results)

The deterministic output is [True, True, True, False, True]. The design is easy to explain and cheap to store because old windows expire naturally. Its trade-off is boundary bursting: three requests at second 59 and three at second 60 are six requests in two seconds while still satisfying the per-minute counter. This may be acceptable for coarse abuse protection but too loose for expensive writes.

Example 2: token bucket for bursts

A token bucket separates average rate from burst size. In the next example, capacity is five tokens and refill is two tokens per second. The first five immediate requests pass, the sixth fails, then later calls pass as tokens refill.

class TokenBucket:
    def __init__(self, capacity, refill_per_second):
        self.capacity = float(capacity)
        self.refill_per_second = float(refill_per_second)
        self.tokens = float(capacity)
        self.last_seen = 0.0

    def allow(self, now):
        elapsed = max(0.0, now - self.last_seen)
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_per_second)
        self.last_seen = now
        if self.tokens >= 1.0:
            self.tokens -= 1.0
            return True, round(self.tokens, 2)
        return False, round(self.tokens, 2)

bucket = TokenBucket(capacity=5, refill_per_second=2)
for at in [0, 0, 0, 0, 0, 0, 0.5, 1.0]:
    print(at, bucket.allow(at))

The output shows the remaining token balance after each decision. The request at time 0 that finds zero tokens is denied. At 0.5, one token has refilled, so the request passes. Token buckets are common for public APIs because they let legitimate clients burst briefly while bounding sustained load. The bucket capacity is the burst allowance; the refill rate is the long-term rate.

Example 3: bounded worker queue and shedding

Backpressure appears inside services as bounded queues. If a queue is unbounded, memory grows while latency becomes meaningless. This worker pool accepts only four queued items and processes two per tick. New work is shed once the queue is full.

from collections import deque

class WorkerPool:
    def __init__(self, queue_limit, service_per_tick):
        self.queue = deque()
        self.queue_limit = queue_limit
        self.service_per_tick = service_per_tick
        self.dropped = 0

    def tick(self, incoming):
        accepted = 0
        for item in incoming:
            if len(self.queue) < self.queue_limit:
                self.queue.append(item)
                accepted += 1
            else:
                self.dropped += 1
        processed = []
        for _ in range(min(self.service_per_tick, len(self.queue))):
            processed.append(self.queue.popleft())
        return {'accepted': accepted, 'processed': processed, 'queued': len(self.queue), 'dropped': self.dropped}

pool = WorkerPool(queue_limit=4, service_per_tick=2)
for tick, incoming in enumerate([['a','b','c'], ['d','e','f','g'], ['h'], []], start=1):
    print(tick, pool.tick(incoming))

At tick one, three items arrive and two are processed, leaving one queued. At tick two, four items arrive, only three fit, and one is dropped. The important behavior is that the queue remains bounded. The caller sees rejection quickly instead of waiting behind work that cannot finish soon. In a real API, this rejection would map to 503 for overload or 429 if the queue is per caller.

Retry signaling example

A denial should be machine-readable. This small response shape demonstrates the pieces a client can use to slow down without guessing.

retry_after = 3
response = {
    'status': 429,
    'headers': {'Retry-After': str(retry_after)},
    'body': {'error': 'rate_limited', 'retry_after_seconds': retry_after}
}
assert response['status'] == 429
assert response['headers']['Retry-After'] == '3'

The status tells the client the request did not start. The header gives a retry hint. The body gives application-level detail for SDKs and logs. For non-idempotent operations, clients should not blindly retry unless the API provides an idempotency key or a clear statement that the operation was not accepted.

Design choices and trade-offs

Edge limiting is cheap and protects the whole platform, but it may lack authenticated tenant context until after request processing begins. Application limiting can use richer identities and route costs, but it consumes more resources before rejecting. Per-user limits promote fairness; per-IP limits catch unauthenticated floods but can punish many users behind the same NAT. Global limits protect shared dependencies, while per-tenant limits prevent one tenant from consuming the entire global allowance.

Rejecting is usually better than delaying when the expected wait exceeds the caller deadline. Delaying can be helpful for smoothing short spikes, especially in batch workers, but it increases tail latency and can hide overload until queues become enormous. Strict distributed counters provide fairness but add latency and dependency risk. Approximate or local counters are faster and more resilient but allow short-term overshoot. Priority shedding improves business outcomes but must be designed carefully so low-priority traffic still receives some capacity when the system is healthy.

Failure modes and troubleshooting

A common symptom is synchronized retry traffic: graphs show periodic spikes after every outage, often at exact one-second or one-minute intervals. The cause is clients retrying at the same fixed delay. Diagnose by checking retry headers, client SDK configuration, and request timestamps. Correct it with exponential backoff, jitter, idempotency keys for safe retries, and server-side Retry-After values that are not unrealistically short.

Another symptom is high latency with low CPU. The service looks idle but requests time out. The cause is often a full downstream connection pool, a blocked dependency, or an unbounded queue accumulating old work. Diagnose by comparing queue depth, queue age, worker utilization, dependency latency, and deadline remaining at handler entry. Correct it by bounding the queue, rejecting when queued work is too old, reducing client concurrency, and setting smaller per-hop timeouts so a failed dependency does not consume all request threads.

A third symptom is unfair throttling: one large tenant or one route consumes the shared budget and smaller tenants see denials. The cause is a limiter keyed too broadly, such as a single global counter or source IP. Diagnose by grouping allowed and rejected requests by tenant, route, method, and cost. Correct it with hierarchical limits: a global cap for safety, per-tenant caps for fairness, and route-specific weights for expensive operations.

A fourth symptom is quota bypass in a horizontally scaled fleet. Each instance says the caller is under limit, but aggregate traffic exceeds the intended cap. The cause is local counters with no coordination. Diagnose by comparing per-instance decisions with aggregate ingress logs. Correct it with a shared atomic counter for strict quotas, consistent hashing of subjects to limiter owners, or conservative local limits when exact fairness is less important than availability.

Reliability and security implications

Rate limiting is also an abuse-control mechanism. It slows credential stuffing, scraping, expensive search amplification, and accidental client loops. It is not a complete security control because attackers can rotate identities, but it buys time and protects expensive paths. Reliability depends on fail behavior. A limiter protecting login attempts might fail closed for unknown users, while a limiter protecting noncritical analytics might fail open if Redis is unavailable. For core APIs, a layered design avoids making the limiter itself a single point of failure.

Performance work should measure limiter overhead on the hot path. A remote limiter call on every request can dominate latency for cheap endpoints. Caching decisions for a few seconds, using local token buckets backed by periodic global reconciliation, or limiting only expensive routes can reduce overhead. Headers that expose remaining quota are useful, but they should not reveal sensitive tenant capacity or internal topology.

Hands-on lab: simulate overload controls

Prerequisites: Python 3 and a terminal. No external packages are required. Step 1: copy the token bucket example into token_bucket_demo.py and run python3 token_bucket_demo.py. Verify that the first five calls at time zero pass, the sixth is denied, and later calls pass after refill. Step 2: change capacity from 5 to 2. Verify that the burst allowance shrinks while the refill behavior remains controlled by refill_per_second. Step 3: copy the worker queue example into queue_shedding_demo.py and run it. Verify that dropped count increases when arrivals exceed queue capacity. Step 4: change service_per_tick from 2 to 1. Verify that queue depth grows faster and shedding begins earlier. Step 5: restore the original values or delete the demo files as cleanup.

To extend the lab, add a priority field to each queued item and shed low-priority items first. Verification should show that high-priority items are processed even during overload. This mirrors real systems where checkout, payment capture, or incident-response traffic may receive capacity before recommendation refreshes or analytics exports.

Assessment exercises

1. A public search endpoint allows 1,000 requests per minute per API key but becomes unstable when a single request scans too many rows. Design a policy that limits both request count and query cost. Explain what response the client receives for each denial type.

2. A service has a mean latency of 20 ms, but p99 jumps to 8 seconds during a dependency outage. What metrics would distinguish unbounded queueing from slow CPU execution, and where would you add load shedding?

3. You run eight API instances with local per-instance limits of 100 requests per minute. A customer is promised 100 requests per minute globally. How can the customer exceed the promise, and what are two ways to correct the design?

4. A mobile client retries every failed write exactly three times after one second. During an outage, traffic triples. Redesign the retry and server signaling behavior so recovery is smoother and duplicate writes are avoided.

Summary

Rate limiting decides whether a caller may start work, backpressure tells senders to slow down, and load shedding rejects work that would damage the system or miss its deadline. The strongest designs combine subject-aware quotas, bounded queues, clear retry signals, deadline-aware rejection, and layered protection around shared dependencies. The trade-off is never simply strictness versus generosity; it is fairness, latency, consistency, operational simplicity, and the business value of the work that remains accepted during overload.