Timeouts Retries Idempotency and Request Deduplication

Published by Programming Line in System Design and Distributed Systems. Last updated .

Purpose and Outcome

Distributed APIs fail in uncomfortable ways: a request may arrive after the caller gives up, a response may be lost after the server commits a write, or a retry may repeat an operation that should happen once. Timeouts, retries, idempotency, and request deduplication are the control system for that uncertainty. They decide how long a caller waits, when it tries again, how a service recognizes a repeated operation, and whether duplicate in-flight work can be collapsed into one result.

By the end of this lesson you should be able to set a timeout budget across a call chain, choose retry conditions that do not amplify an outage, design an idempotency-key protocol for write APIs, and implement deduplication for concurrent identical requests. This belongs in networking and API architecture because the behavior is not local to one function. It is a contract between clients, gateways, services, queues, stores, and operators.

How the Mechanisms Work

A timeout is a deadline after which the caller stops waiting. It does not automatically cancel the server-side work unless cancellation is propagated and honored. Useful systems distinguish connection timeout, TLS handshake timeout, request write timeout, response header timeout, read timeout, and an overall deadline. The overall deadline is usually the most important value because it bounds user-visible latency and resource occupancy.

A retry is a second or later attempt after an ambiguous or known transient failure. Retries help when failures are short-lived, such as packet loss, one overloaded instance, or a leader election. They hurt when the first attempt is still running, the target is already saturated, or the operation is not safe to repeat. A retry policy normally specifies eligible errors, maximum attempts, per-attempt timeout, backoff, jitter, and whether the original deadline is shared across all attempts.

Idempotency means repeated requests with the same intent have the same effect as one request. GET, PUT, and DELETE are commonly designed to be idempotent, while POST often is not unless the API adds an idempotency key. For example, creating a payment with key pay_123 should charge once even if the client sends the same request three times after timeouts. The service stores the key, a fingerprint of the request body, the operation state, and the final response. Later retries with the same key return the stored response instead of executing again.

Request deduplication removes duplicate work. It can happen after completion, using an idempotency-key table, or while work is in progress, using a single-flight map, lock, queue, or cache fill coordinator. Idempotency protects correctness of side effects. Deduplication protects capacity by avoiding repeated expensive reads, computations, or writes.

API Anatomy

A robust client request usually carries an overall deadline and, for unsafe writes, an idempotency key. The server validates that the key has enough entropy, scopes it to the authenticated actor and operation, and stores a request fingerprint. If the key is new, the service reserves it before starting the side effect. If another request with the same key arrives while the first is running, the service can wait, return a conflict such as 409, or return an accepted status such as 202 with a status URL. If the key already completed, the service returns the cached status code and response body.

The storage record commonly has fields like tenant_id, idempotency_key, request_hash, status, response_code, response_body, locked_until, and expires_at. A unique constraint on tenant, endpoint, and key prevents races. A time-to-live keeps the table bounded, but it also defines how long clients may safely retry ambiguous operations.

Example 1: One Timeout Budget Across Attempts

Assume a checkout page has a 900 ms service-level objective for the payment authorization call. A poor design gives each of three attempts a 900 ms timeout, creating a worst case of 2700 ms before the page can respond. A better design uses one deadline. The first attempt might get 500 ms, then after backoff the second gets the remaining time. Once the deadline is exhausted, no further attempt is useful for this user request.

def retry_schedule(total_ms, max_attempts, backoff_ms):
    elapsed = 0
    result = []
    for attempt in range(1, max_attempts + 1):
        remaining = total_ms - elapsed
        if remaining <= 0:
            break
        per_attempt = min(500, remaining)
        result.append((attempt, per_attempt, remaining))
        elapsed += per_attempt
        if attempt != max_attempts:
            elapsed += backoff_ms
    return result

print(retry_schedule(900, 3, 100))

The deterministic output is [(1, 500, 900), (2, 300, 300)]. The third attempt is skipped because the shared deadline has been consumed. In a real client, elapsed time would come from a monotonic clock, and the server would receive the remaining deadline through metadata or headers.

Example 2: Safe Retry Classification

Not every failure deserves a retry. Retrying validation errors wastes capacity. Retrying a connection reset before the server received the request may be safe. Retrying a timeout after a write request is ambiguous: the server may have committed the change and lost the response. That case requires idempotency before automatic retry.

def should_retry(method, status=None, network_error=None, has_key=False):
    if network_error in {"dns_timeout", "connect_timeout", "connection_reset"}:
        return method in {"GET", "PUT", "DELETE"} or has_key
    if status in {429, 500, 502, 503, 504}:
        return method in {"GET", "PUT", "DELETE"} or has_key
    return False

cases = [
    ("GET", 503, None, False),
    ("POST", 504, None, False),
    ("POST", 504, None, True),
    ("POST", 400, None, True),
]
print([should_retry(*case) for case in cases])

The expected output is [True, False, True, False]. The POST with a gateway timeout becomes retryable only when it has an idempotency key. A 400 response is not retried because it indicates the request is invalid, not that the transport or server is temporarily unhealthy.

Example 3: Idempotency Store Behavior

The next example models the core server behavior. A repeated key with the same body returns the first result. The same key with a different body is rejected because the client is trying to reuse an identity for a different operation.

import hashlib

class IdempotencyStore:
    def __init__(self):
        self.records = {}

    def handle(self, tenant, key, body, execute):
        scope = (tenant, key)
        body_hash = hashlib.sha256(body.encode()).hexdigest()
        if scope in self.records:
            saved_hash, saved_response = self.records[scope]
            if saved_hash != body_hash:
                return 422, {"error": "idempotency key reused with different body"}
            return saved_response
        response = execute()
        self.records[scope] = (body_hash, response)
        return response

charges = []
store = IdempotencyStore()

def charge_once():
    charges.append("charged")
    return 201, {"charge_id": "ch_1"}

print(store.handle("acct_7", "key_abc", "amount=50", charge_once))
print(store.handle("acct_7", "key_abc", "amount=50", charge_once))
print(store.handle("acct_7", "key_abc", "amount=60", charge_once))
print(charges)

The first two calls return (201, {'charge_id': 'ch_1'}); the third returns (422, {'error': 'idempotency key reused with different body'}); charges contains one item. Production code would reserve the key transactionally before calling the payment processor, store failures deliberately, and expire old records after a documented retry window.

Design Choices and Trade-offs

Timeouts should be shorter than the caller’s patience and longer than normal tail latency. If they are too short, the system creates false failures and retries healthy work. If they are too long, callers occupy threads, sockets, memory, and connection-pool slots while users wait. Per-hop defaults are dangerous because a chain of services can multiply them. Deadline propagation makes downstream services aware that only 120 ms remain and that starting a 500 ms dependency call is pointless.

Retries need backoff and jitter. Exponential backoff reduces pressure after failure; jitter prevents many clients from retrying at exactly the same time. Retry budgets are also useful: a service might allow retries to add no more than 10 or 20 percent extra traffic. Without a budget, a partial outage can turn into a retry storm where the system spends most of its capacity repeating doomed calls.

Idempotency has storage and semantic costs. Storing only successful responses is simple, but a client that times out after a committed operation may retry while the record is missing if the write and idempotency record are not atomic. Storing both successes and certain failures gives more deterministic client behavior, but teams must decide which failures represent final results. For example, validation errors can be stored; transient 503 responses often should not be stored as the final result for the key.

Deduplication can be local or distributed. Local single-flight deduplication is fast and useful for cache fills inside one process, but it does not collapse work across a fleet. Distributed deduplication through Redis, a database row, or a queue partition works across instances but introduces lock expiry, clock, and ownership concerns. The correction is usually fencing: every lock or lease has a token, and downstream writes accept only the current token.

Failure Modes and Troubleshooting

Symptom: users report duplicate orders after mobile network timeouts. Cause: the client retried POST requests after read timeouts without idempotency keys, and the server treated each request as a new order. Diagnostics: search logs for the same user, cart, and amount within a short window; compare request IDs and idempotency-key headers; inspect payment processor events. Correction: require idempotency keys for order creation, reject missing keys from retrying clients, and store the final order response under a tenant-scoped unique key.

Symptom: latency improves during normal traffic but gets worse during an upstream incident. Cause: every caller performs three immediate retries, tripling load on the failing dependency. Diagnostics: graph retry attempts per original request, upstream saturation, and response codes by attempt number. Correction: add exponential backoff with jitter, cap attempts by deadline, honor Retry-After for 429 and 503 responses, and open a circuit breaker when success probability is low.

Symptom: clients receive permanent conflicts for idempotency keys that never complete. Cause: a worker reserved a key and crashed before updating the record, leaving it stuck in an in-progress state. Diagnostics: query records where status is in progress and locked_until is in the past; correlate with worker crashes. Correction: use leases with expiration, recovery jobs, and clear state transitions from reserved to completed or failed.

Security, Performance, and Reliability Implications

Idempotency keys must not be predictable counters shared across tenants. A guessed key could reveal whether another operation exists or cause a denial of service by occupying the key first. Scope keys to the authenticated principal, avoid logging sensitive request bodies, and compare request fingerprints rather than storing raw payloads when possible. Rate limits should count original attempts and retries so abusive clients cannot hide behind retry behavior.

Performance improves when duplicate expensive reads are collapsed, but deduplication tables and locks are now on the critical path. Keep records small, index by the exact lookup scope, and choose an expiration window that matches client retry behavior. For high-throughput APIs, a write-through idempotency table may need partitioning by tenant or key hash.

Hands-on Lab: Build and Verify an Idempotent Retry Flow

Prerequisites: Python 3, a terminal, and no external services. The lab simulates a client timeout after the server commits a charge.

  1. Create a temporary working directory.
  2. Place the Example 3 code into idempotency_lab.py.
  3. Run python idempotency_lab.py.
  4. Verify that the first and second same-key calls return the same charge id and that the charges list contains only one item.
  5. Change the second body from amount=50 to amount=60 and run again.

Verification: the unchanged run should show one committed charge even though the key is submitted twice. The changed-body run should return a 422 error for key reuse. Cleanup: delete the temporary directory. In a real service, cleanup means expiring records after the documented retry window and monitoring deletion lag.

Assessment Exercises

  1. A service has a 1200 ms user-visible deadline and calls two dependencies in sequence. The first consumes 850 ms. What should the second dependency receive as its maximum deadline, and why is a default 1 second timeout wrong?
  2. Design the idempotency record for a money-transfer API. Which fields need unique constraints, which fields are stored for response replay, and how long should records live?
  3. A team retries POST requests on 502, 503, and 504 but not on client-side read timeouts. Is the policy complete? Explain the ambiguity in each case.
  4. You see many requests waiting on a distributed lock used for deduplication. List two metrics and one code-path inspection that would tell you whether the lock is protecting useful work or becoming the bottleneck.

Summary

Timeouts bound waiting, retries recover from brief uncertainty, idempotency prevents repeated side effects, and deduplication avoids repeated work. The mechanisms only work when designed together: retries need deadlines and backoff, unsafe writes need keys and request fingerprints, and in-progress work needs clear ownership and recovery. In system design interviews and production reviews, the strongest answer is specific about which failures are retried, which operations are safe to repeat, and where the system records the result that makes a duplicate request harmless.