Capacity Planning Performance Testing and Cost Engineering

Published by Programming Line in System Design and Distributed Systems. Last updated .

Purpose and Outcome

Capacity planning, performance testing, and cost engineering answer one shared system-design question: how much load can this architecture absorb, at what latency, and at what price? A design that is reliable but wildly overprovisioned wastes money. A design that is cheap but saturates during a launch is not production-capable. In this chapter you will learn how to turn product traffic assumptions into resource demand, how to test those assumptions before users arrive, and how to reason about cost as a first-class architectural constraint.

The outcome is a repeatable model. You estimate workload, translate workload into service capacity, validate the estimate with controlled tests, and then choose scaling and purchasing strategies that preserve service-level objectives while avoiding idle spend. This is not only an operations activity. It shapes early choices about cache placement, queueing, database partitioning, data retention, and dependency limits.

How the Mechanism Works

A capacity plan begins with demand. Demand is usually expressed as request rate, active users, messages per second, bytes per second, jobs per hour, or concurrent sessions. These are related but not interchangeable. A chat system may have many connected users but modest message throughput. A reporting system may have low request rate but high CPU and I/O per request. The capacity model must use the unit that actually consumes the bottleneck resource.

The simplest service model uses three terms: arrival rate, service time, and utilization. Arrival rate is how often work reaches the system. Service time is how long one unit of work occupies the constrained resource. Utilization is arrival rate multiplied by service time, divided by available parallel capacity. A single CPU-bound worker that receives 80 requests per second and spends 5 milliseconds of CPU per request has 40 percent CPU demand. The same worker at 180 requests per second needs 90 percent CPU before background work, garbage collection, retries, and variance are considered.

Latency does not rise linearly as utilization approaches saturation. Queueing delay increases sharply because each new request waits behind more in-flight work. This is why planners usually target headroom rather than 100 percent utilization. For interactive services, a common design posture is to keep average bottleneck utilization well below saturation and use p95 or p99 latency as the signal that headroom is disappearing. For batch systems, higher utilization may be acceptable if deadline completion remains inside the business objective.

Performance testing closes the loop. A load test applies expected steady traffic. A stress test pushes past expected traffic to discover the breaking point. A soak test runs for hours or days to expose leaks, compaction issues, queue buildup, and storage side effects. A spike test changes traffic quickly to measure autoscaling lag, connection pool behavior, cache cold-start cost, and retry amplification. Good tests measure throughput, latency percentiles, error rate, saturation metrics, and cost per successful unit of work.

Cost engineering connects usage to money. Compute cost often scales with provisioned instances or consumed CPU. Storage cost scales with retained bytes, replicas, indexes, snapshots, and data transfer. Managed databases add charges for provisioned throughput, I/O, backups, and cross-region replication. The useful metric is not total monthly spend alone; it is unit economics such as cost per thousand requests, cost per tenant, cost per report, or cost per gigabyte processed. Those unit metrics let architects compare design alternatives.

Configuration and API Anatomy

A practical capacity plan has a small set of explicit inputs. Traffic inputs describe baseline rate, peak multiplier, growth horizon, and burst shape. Workload inputs describe CPU milliseconds per request, database reads and writes, payload size, cache hit ratio, and fan-out to dependencies. Resource inputs describe cores per instance, worker concurrency, database IOPS, partition limits, network bandwidth, and quotas. Policy inputs describe target utilization, latency objective, regional redundancy, and required failover headroom.

The following small Python model shows the anatomy. It is intentionally simple: it estimates instances from request rate, measured CPU time per request, cores per instance, and a target utilization. The example is deterministic, so its output can be used as a regression check when the model changes.

from math import ceil


def required_instances(rps, cpu_ms_per_request, cores_per_instance, target_utilization):
    cpu_seconds_per_second = rps * cpu_ms_per_request / 1000
    usable_cpu = cores_per_instance * target_utilization
    return ceil(cpu_seconds_per_second / usable_cpu)


print(required_instances(1200, 8, 4, 0.60))

The output is 4. The service needs 9.6 CPU-seconds per second. Each instance contributes 2.4 usable CPU-seconds per second at 60 percent target utilization, so four instances meet the CPU constraint before redundancy or zonal failure headroom is added.

Worked Example 1: CPU-Bound API

Assume a product detail API expects 1,200 peak requests per second. Profiling shows that each request uses 8 milliseconds of application CPU when the cache is warm. Instances have 4 vCPUs and the team targets 60 percent CPU utilization to protect p95 latency. The model above returns four instances for raw CPU demand. A production design would normally run at least one additional instance per failure domain if losing an instance must not breach the objective.

The expected behavior is straightforward: at or below 1,200 requests per second, p95 latency should stay close to the profiled service time plus network and dependency time. If p95 latency grows while CPU is below target, CPU is not the real bottleneck. The likely bottleneck is a dependency, connection pool, lock, garbage collector, or network limit.

Worked Example 2: Database and Cache Mix

Now the product API reads inventory from a database. Each request checks one product row. At 1,200 requests per second, a 90 percent cache hit ratio means only 120 database reads per second reach the primary datastore. At a 70 percent hit ratio, database reads jump to 360 per second. The application tier may still have enough CPU, but the database plan changes completely.

def database_reads(rps, cache_hit_ratio, reads_per_miss):
    return rps * (1 - cache_hit_ratio) * reads_per_miss


for hit_ratio in (0.90, 0.70):
    print(f"{hit_ratio:.0%} hit ratio: {database_reads(1200, hit_ratio, 1):.0f} reads/s")

The deterministic output is:

90% hit ratio: 120 reads/s
70% hit ratio: 360 reads/s

This example shows why capacity planning must include cache behavior, not only application request rate. A cache outage, low time-to-live, or poor key design can triple database traffic without any change in customer demand. The architecture needs either enough database headroom for expected cache degradation, load shedding for noncritical requests, or a cache design that avoids synchronized expiration.

Worked Example 3: Cost Per Successful Request

Suppose the API runs four instances at 20 cents per instance-hour and handles 1,200 requests per second at peak for two hours each day, then 300 requests per second for the remaining 22 hours. A fixed fleet costs 19.20 dollars per day. If autoscaling runs four instances only during peak and one instance otherwise, compute cost drops to 6.00 dollars per day before minimum redundancy rules. That saving is meaningful only if scale-out is fast enough and if one instance can safely handle the off-peak load.

def daily_compute_cost(instance_price_per_hour, peak_instances, base_instances, peak_hours):
    return instance_price_per_hour * (peak_instances * peak_hours + base_instances * (24 - peak_hours))


fixed = daily_compute_cost(0.20, 4, 4, 2)
autoscaled = daily_compute_cost(0.20, 4, 1, 2)
print(f"fixed=${fixed:.2f}")
print(f"autoscaled=${autoscaled:.2f}")

The output is:

fixed=$19.20
autoscaled=$6.00

The example uses compute only, so it should not be mistaken for total cost. Databases, load balancers, logs, metrics, data transfer, and backups may dominate. Still, this calculation creates a useful unit question: how much does each successful request cost under fixed capacity versus elastic capacity?

Design Choices and Trade-Offs

Vertical scaling buys larger machines. It is operationally simple and often effective for databases, but it creates stepwise cost increases and hard ceilings. Horizontal scaling adds more replicas or workers. It improves redundancy and elasticity, but requires stateless application design, balanced partitioning, idempotent work handling, and careful connection management.

Provisioned capacity gives predictable latency because resources are already warm. It can waste money during quiet periods. Autoscaling reduces idle spend but introduces lag. If traffic spikes faster than instances boot, requests queue or fail. The usual correction is a minimum warm floor, predictive schedules for known peaks, and scaling signals based on saturation rather than only average CPU.

Read replicas, caches, and queues are common capacity tools, but each moves the problem. A read replica increases read capacity but adds replication lag. A cache reduces backend load but can serve stale data and create stampedes. A queue smooths bursts but turns overload into delayed work; if producers keep outpacing consumers, latency becomes backlog age rather than request time.

Failure Modes and Troubleshooting

Symptom: latency rises before CPU is high. The cause is often hidden serialization: a small database connection pool, a hot lock, a single partition, or a downstream service limit. Diagnose by comparing p95 latency with CPU, run queue length, connection pool wait time, dependency latency, and per-partition traffic. Correct by increasing the constrained pool only if the dependency can handle it, removing lock contention, spreading keys, or reducing fan-out.

Symptom: load tests pass, but production fails during launch. The test probably used a smooth traffic curve, warm caches, unrealistic payloads, or no user think-time distribution. Diagnose by comparing production request mix, cache hit ratio, payload sizes, authentication paths, and dependency calls against the test profile. Correct the test so it replays representative mixes and includes cold start and spike phases.

Symptom: cost grows faster than traffic. The cause may be log volume, high-cardinality metrics, inefficient queries, cross-region data transfer, over-retention, or autoscaling based on noisy signals. Diagnose by breaking spend into unit metrics and looking for cost per request, cost per gigabyte, and cost per tenant. Correct by sampling verbose logs, reducing metric cardinality, adding query indexes, changing retention, or placing services closer to the data they consume.

Security, Performance, and Reliability Implications

Capacity is a reliability control because overload causes cascading failures. When a saturated service retries aggressively, it can multiply traffic against an already unhealthy dependency. Capacity plans should therefore include retry budgets, circuit breakers, rate limits, backpressure, and admission control. These are not substitutes for capacity, but they keep overload contained while the system sheds or delays lower-priority work.

Security and cost also intersect. Abuse traffic can consume expensive capacity even when it does not break the application. Authentication endpoints, search APIs, file upload paths, and report generation jobs need quotas and per-principal limits. Performance tests should include authorized and rejected traffic paths because rejected requests still use CPU, network, and logging resources.

Hands-On Lab: Build and Validate a Tiny Capacity Model

Prerequisites: a shell with Python 3 available. No cloud account is required. The lab estimates capacity for a small API and checks how cache hit ratio changes database demand.

  1. Create a scratch file named capacity_lab.py.
  2. Paste the model below.
  3. Run python capacity_lab.py.
  4. Change peak_rps from 1200 to 2000 and run it again.
  5. Change cache_hit_ratio from 0.90 to 0.75 and observe the database read estimate.
from math import ceil

peak_rps = 1200
cpu_ms = 8
cores_per_instance = 4
target_utilization = 0.60
cache_hit_ratio = 0.90

cpu_needed = peak_rps * cpu_ms / 1000
instances = ceil(cpu_needed / (cores_per_instance * target_utilization))
database_reads = peak_rps * (1 - cache_hit_ratio)

print(f"usable instances: {instances}")
print(f"database reads/s: {database_reads:.0f}")

Verification for the initial values is:

usable instances: 4
database reads/s: 120

Cleanup is simply deleting the scratch file. Rollback for a real service would mean restoring the previous autoscaling floor, alert threshold, or purchasing plan after recording the tested assumptions and measured results.

Assessment Exercises

  1. An API handles 800 requests per second and uses 12 milliseconds of CPU per request. Instances have 2 vCPUs and the target utilization is 50 percent. How many instances are required before redundancy?
  2. A load test uses constant request rate and warm caches. Production traffic has sudden bursts and frequent cache misses after deployments. Which two test phases would you add, and what metrics would prove the change is useful?
  3. Your monthly bill doubled while request traffic rose only 20 percent. Name three non-compute cost dimensions you would inspect and the unit metric you would use for each.
  4. A queue-based image processor meets throughput targets but user-visible completion time is increasing. Which capacity metric matters more than request latency, and what design choices could reduce it?

Summary

Capacity planning turns workload assumptions into resource requirements. Performance testing checks whether those requirements survive realistic traffic, variance, dependency behavior, and failure conditions. Cost engineering keeps the design economically honest by measuring spend per useful unit of work. In system design, the strongest answer is not the biggest fleet or the cheapest fleet; it is the architecture whose bottlenecks, headroom, scaling behavior, and unit costs are explicit enough to defend.