Latency Throughput Concurrency and Resource Limits

Published by Programming Line in System Design and Distributed Systems. Last updated .

Purpose: sizing a service before it melts

Latency, throughput, concurrency, and resource limits are the four quantities that turn a system design from a drawing into an operating envelope. Latency is how long one unit of work takes, usually described with percentiles such as p50, p95, and p99 rather than only an average. Throughput is the rate at which work completes, such as requests per second, jobs per minute, or bytes per second. Concurrency is the number of units of work in progress at the same time. Resource limits are the ceilings imposed by CPU cores, memory, disk I/O, network bandwidth, file descriptors, thread pools, database connections, queues, and external dependencies.

The outcome of this lesson is a practical mental model: for any proposed architecture, you should be able to estimate the steady-state concurrency it needs, identify the resource that caps throughput, predict what happens when arrivals exceed capacity, and choose whether to reject, queue, shed, cache, batch, parallelize, or scale. This is foundational system design because many failures that look like random slowness are simple math: more work arrived than the slowest constrained component could finish.

The internal mechanics

A request spends time in two broad states: waiting and being served. Waiting time is time spent before a worker, connection, CPU slice, lock, partition, or downstream dependency is available. Service time is the time spent actually doing the work after the needed resource is acquired. End-to-end latency is the sum of client network time, load balancer handling, application queueing, application service time, downstream queueing, downstream service time, serialization, and response transfer. When teams say “the database is slow,” they often mean either database service time is high or callers are waiting for a limited database connection pool. The correction differs, so the distinction matters.

The most useful relationship is Little’s Law: average concurrency equals throughput multiplied by average latency. If a service completes 200 requests per second and each request spends 100 milliseconds inside the system, the average in-flight concurrency is about 20 requests. This is not a capacity guarantee; it is a steady-state accounting identity. It helps you check whether pool sizes and queue lengths are plausible. If a gateway must support 2,000 requests per second at 250 milliseconds p95 latency, a maximum of 50 in-flight requests is inconsistent with the target unless most requests bypass that gateway.

Throughput is capped by the bottleneck resource. With one worker and 50 milliseconds of service time, the theoretical maximum is 20 requests per second. Ten identical workers raise the simple upper bound to 200 requests per second, but only if the work does not serialize on a lock, a single database row, a single partition, a saturated CPU, or a downstream service with its own limit. Amdahl’s Law explains why parallelism stops helping when part of the work is serial. If 20 percent of each request must run under a global lock, adding workers eventually causes them to wait behind that lock instead of finishing more work.

Queueing is the mechanism that makes latency curve upward before a system fully fails. At low utilization, a request usually finds an idle worker. As utilization approaches 100 percent, small bursts have nowhere to go, so waiting time grows. This is why a service can look healthy at 50 percent CPU and then become unusable at 85 percent CPU: the missing variable is variability. Bursty arrivals, long-tail service times, garbage collection pauses, disk stalls, and uneven shard distribution all create queues. The p99 request is often waiting behind earlier work.

API and configuration anatomy

System designs express these quantities through concrete controls. A server usually has a listener backlog, accept loop, worker model, request timeout, maximum request body size, and keep-alive policy. A runtime has thread pools, event loops, process counts, memory heaps, garbage collectors, and CPU quotas. A database client has a connection pool, acquisition timeout, query timeout, retry policy, and maximum cursor usage. A queue consumer has batch size, visibility timeout, prefetch count, worker count, and dead-letter behavior.

The names vary by platform, but the anatomy is consistent. A limit bounds how much work can enter a component. A timeout bounds how long work can occupy resources. A pool bounds parallel access to an expensive resource. A queue absorbs bursts by trading higher latency for fewer immediate failures. A rate limiter caps accepted arrival rate. A circuit breaker temporarily fails fast when a dependency is unhealthy. A bulkhead separates capacity so one traffic class cannot consume all workers or connections.

def estimate_concurrency(throughput_per_second, latency_ms):
    return throughput_per_second * (latency_ms / 1000)

scenarios = [
    (200, 100),
    (2000, 250),
    (75, 40),
]

for rps, latency in scenarios:
    print(f"{rps} rps at {latency} ms needs about {estimate_concurrency(rps, latency):.1f} in-flight requests")

This calculator applies Little’s Law. Its output is: 200 rps at 100 ms needs about 20.0 in-flight requests, 2000 rps at 250 ms needs about 500.0 in-flight requests, and 75 rps at 40 ms needs about 3.0 in-flight requests.

Worked example 1: a single web handler

Assume a profile endpoint performs 10 milliseconds of application work and one database query that takes 40 milliseconds. With one worker handling requests sequentially, service time is roughly 50 milliseconds, so the best-case throughput is about 20 requests per second. If requests arrive at 10 per second, utilization is about 50 percent and waiting is usually small. If requests arrive at 25 per second, the worker cannot keep up. A queue forms, latency keeps increasing, and clients eventually time out even though each handler still needs only 50 milliseconds of service time.

The expected behavior is simple: below the service rate, most requests finish quickly; above it, backlog grows without bound unless a queue limit, timeout, or admission control rejects some work. In a design interview or architecture review, this example is enough to explain why “average handler time” is not capacity. Capacity is handler time multiplied by available parallel workers and constrained by any shared downstream resource.

Worked example 2: workers and a database pool

Now give the application 50 request workers but only 10 database connections. The endpoint still does 10 milliseconds of CPU work and 40 milliseconds of database work. At first, increasing workers improves throughput because more requests can overlap while waiting on network and database I/O. But no more than 10 queries can execute concurrently through this pool. If the query takes 40 milliseconds, the pool’s rough upper bound is 250 queries per second. The application may have 50 workers, but the database pool gates the endpoint.

This is a common production symptom: application worker utilization is moderate, but request latency rises and logs show connection acquisition timeouts. The cause is not necessarily a slow query; it can be a pool sized below the desired concurrency, a leak that fails to return connections, or a sudden increase in query duration that lowers pool throughput. The correction might be to optimize the query, add indexes, reduce per-request query count, increase pool size cautiously, or place separate pools around high-priority and low-priority traffic.

Worked example 3: queues, bursts, and backpressure

Consider an image thumbnail pipeline. Users upload images to an API, the API enqueues jobs, and workers generate thumbnails. If uploads average 100 per second and workers process 120 per second, the system is stable on average. During a five-minute campaign, uploads spike to 300 per second. If the queue accepts everything, upload latency may stay low, but job completion latency grows by 180 extra jobs per second during the spike. After five minutes, the backlog is 54,000 jobs. At 120 jobs per second with no new work, draining takes 7.5 minutes; with the normal 100 jobs per second still arriving, only 20 jobs per second are available for drain, so recovery takes 45 minutes.

def backlog_after_spike(arrival_rate, service_rate, spike_seconds):
    return max(0, arrival_rate - service_rate) * spike_seconds

def drain_seconds(backlog, normal_arrival_rate, service_rate):
    spare = service_rate - normal_arrival_rate
    if spare <= 0:
        return None
    return backlog / spare

backlog = backlog_after_spike(300, 120, 5 * 60)
print(f"backlog jobs: {backlog}")
print(f"drain time at normal load: {drain_seconds(backlog, 100, 120) / 60:.1f} minutes")

The deterministic output is backlog jobs: 54000 and drain time at normal load: 45.0 minutes. This shows why asynchronous queues do not remove capacity limits; they move user-visible waiting from the upload path to the completion path. That may be the right trade-off, but it must be explicit. If users expect thumbnails in seconds, the design needs autoscaling, priority queues, lower-cost thumbnail generation, admission control, or a degraded mode such as placeholder images.

Design choices and trade-offs

Increasing concurrency can improve throughput when work is I/O-bound, but it can reduce throughput when it increases context switching, memory pressure, lock contention, cache misses, or downstream saturation. Larger queues absorb bursts, but they also hide overload and increase tail latency. Small queues reveal overload quickly, but they reject more requests during short spikes. More retries can mask transient errors, but retries multiply load during dependency incidents. Short timeouts free resources quickly, but they can fail valid slow operations. Long timeouts reduce false failures, but they hold workers and connections while callers may already have given up.

Scaling horizontally adds more service instances, which helps only if the bottleneck is inside those instances or the load can be partitioned. If every instance calls the same saturated database primary, adding instances may increase pressure and make latency worse. Caching can reduce downstream throughput demand, but it introduces staleness, invalidation complexity, hot-key risk, and memory limits. Batching improves throughput for many storage and network systems by amortizing fixed overhead, but it adds waiting time while a batch fills and can make individual failures harder to isolate.

Failure modes and troubleshooting

Symptom: p99 latency rises while average latency looks acceptable. The likely cause is queueing behind a constrained pool, uneven request cost, lock contention, or a dependency with long-tail latency. Diagnose by separating time spent waiting for local workers, waiting for connection pools, executing downstream calls, and writing responses. Correction usually means reducing utilization on the bottleneck, bounding queue length, isolating slow traffic, or lowering service-time variance.

Symptom: CPU is low but requests time out. The cause is often I/O waiting, database pool exhaustion, DNS delays, thread starvation, or a remote service limit. Check pool acquisition metrics, thread dumps, event loop lag, socket states, and dependency latency. Correction may require increasing a specific pool, fixing connection leaks, adding async I/O, reducing blocking calls on event loops, or failing fast when a dependency is saturated.

Symptom: adding instances does not increase completed requests per second. The bottleneck has moved to a shared resource such as the database primary, message broker partition, object store prefix, global lock, or third-party API quota. Confirm by comparing per-instance throughput, downstream utilization, queue depth, and error codes such as throttling or too-many-connections. Correction involves partitioning, read replicas, caching, sharding, quota negotiation, or reducing work per request.

Symptom: a backlog drains much slower than expected. Normal traffic may consume most worker capacity, jobs may be retried repeatedly, one poison message may block a partition, or worker concurrency may be limited by memory or downstream calls. Inspect enqueue rate, completion rate, retry count, age of oldest message, worker logs, and dead-letter queues. Correction includes pausing low-priority producers, increasing safe worker capacity, fixing poison messages, and making retries exponential with jitter and a cap.

Reliability and performance implications

Resource limits are reliability controls, not just performance knobs. Without limits, overload spreads: a slow database causes application threads to pile up, which causes load balancer queues, which causes clients to retry, which increases database traffic further. Explicit limits create places where the system can say no before all capacity is consumed. The important design question is where refusal should happen. Rejecting at the edge protects the whole system but may deny requests that could have succeeded. Rejecting near the database gives more requests a chance but consumes more upstream work first.

Security also intersects with these concepts. Attackers can exploit expensive endpoints, large bodies, slow uploads, unbounded searches, and high-cardinality fan-out to consume concurrency and memory. Rate limits, request size caps, pagination bounds, query cost limits, and per-tenant quotas prevent one caller from converting shared resources into a denial of service. For multi-tenant systems, fairness usually requires separate budgets for tenants, endpoints, or priorities rather than one global pool.

Hands-on lab: model a service limit

Prerequisites: Python 3 and a shell. No external packages are required.

  1. Create a temporary file named capacity_lab.py.
  2. Paste the program below.
  3. Run python3 capacity_lab.py.
  4. Change arrival_rps, workers, and service_ms to test when the system becomes unstable.
from math import inf

arrival_rps = 180
workers = 8
service_ms = 50
queue_limit = 200

service_rps = workers * (1000 / service_ms)
utilization = arrival_rps / service_rps
stable = arrival_rps <= service_rps
queued_per_second = max(0, arrival_rps - service_rps)
seconds_until_full = inf if queued_per_second == 0 else queue_limit / queued_per_second

print(f"service capacity: {service_rps:.0f} rps")
print(f"utilization: {utilization:.2f}")
print(f"stable: {stable}")
print(f"seconds until queue fills: {seconds_until_full if seconds_until_full != inf else 'never'}")

With the provided values, the output is service capacity: 160 rps, utilization: 1.12, stable: False, and seconds until queue fills: 10.0. To clean up, delete capacity_lab.py. The rollback is returning the constants to the original values. Extend the lab by setting workers to 10; capacity becomes 200 requests per second, utilization becomes 0.90, and the queue no longer grows. Then set service_ms to 80; capacity falls to 125 requests per second, showing how a downstream slowdown can break a stable design without any traffic increase.

Assessment exercises

  1. A service completes 400 requests per second with 150 milliseconds average in-system latency. Estimate average in-flight concurrency. If the worker limit is 40, what inconsistency should you investigate?
  2. An API has 100 workers and a database pool of 20 connections. Each request performs one 25 millisecond query. What is the database-pool throughput ceiling before considering database CPU or locks?
  3. A queue receives 500 jobs per second for two minutes while workers complete 350 jobs per second. Normal arrival rate is 300 jobs per second afterward. How large is the backlog, and how long does it take to drain?
  4. For an endpoint with expensive searches, choose two resource limits and explain what failure they prevent and what legitimate user experience they might harm.
  5. Your p99 latency doubles after adding retries around a dependency. Explain a mechanism by which retries can worsen tail latency and name one mitigation.

Summary

Latency is time per unit of work, throughput is completed work per time, concurrency is work in progress, and resource limits define the capacity envelope. Little’s Law connects the first three, while bottleneck analysis identifies which resource actually sets the throughput ceiling. Queues, pools, timeouts, retries, rate limits, and bulkheads are design tools for choosing how a system behaves near that ceiling. Strong system design does not assume infinite parallelism; it states expected rates, bounds admitted work, protects scarce resources, and makes overload visible before it becomes a full outage.