Cost, GPU Capacity, and Performance Engineering

Cost, GPU Capacity, and Performance Engineering turns model deployment into a resource planning problem. The outcome is not simply "use a bigger GPU." The outcome is a serving design that can explain how many replicas are needed, why latency changes under load, what each million predictions or tokens costs, and which knob to turn when saturation appears.

In this MLOps architecture section, this topic connects model choice to platform economics. A model artifact that looks acceptable in offline evaluation can still be impossible to run profitably if it needs too much accelerator memory, wastes batches, or scales on a metric that arrives too late. Cost engineering gives release decisions a quantitative foundation: expected demand, model size, concurrency, latency objective, utilization target, and rollback path.

What GPU Capacity Means

A GPU serving system has several capacity ceilings. Memory capacity determines whether model weights, runtime buffers, and request state fit. Compute capacity determines tokens, images, embeddings, or predictions per second. Interconnect and host capacity determine whether the CPU, network, storage, and PCIe path can keep accelerators fed. Scheduler capacity determines whether requests are packed efficiently into batches without violating latency.

For deep learning inference, memory is often the first constraint. Model weights consume parameter count multiplied by bytes per parameter. A 7 billion parameter model in fp16 needs roughly 14 GB just for weights before runtime overhead. Quantization reduces weight memory, but it can affect quality or kernel support. Large language models also use a key-value cache during generation. Longer prompts, larger batches, and longer generated outputs grow that cache, so a server that fits one request can fail when concurrent requests arrive.

Training and fine-tuning have different pressure points. They store activations, gradients, optimizer state, and checkpoints. Inference capacity usually focuses on tail latency and throughput. Training capacity usually focuses on time-to-complete, utilization, spot interruption risk, and checkpoint recovery. Mixing these workloads on the same GPU pool is possible, but only when scheduling, priority, and preemption behavior are explicit.

Mechanism: From Request to Accelerator

A typical GPU inference path starts when the API gateway receives a request and forwards it to a model server. The server tokenizes or preprocesses input on CPU, places work into a queue, batches compatible requests, copies tensors to GPU memory, executes kernels, and postprocesses the result. The apparent latency is the sum of queue wait, preprocessing, host-to-device transfer, GPU execution, device-to-host transfer, and response serialization.

Batching is the central trade-off. Larger batches increase GPU occupancy and improve throughput, but each request may wait longer for the batch to fill. Dynamic batching lets the server hold a request for a short window, such as a few milliseconds, and combine it with nearby requests. Sequence batching for generative models is more complex because requests produce different numbers of output tokens. Continuous batching keeps adding and removing sequences while generation proceeds, improving utilization for chat and completion workloads.

The main operational metrics are gpu_memory_used, gpu_utilization, queue_depth, batch_size, tokens_per_second or predictions_per_second, p50_latency, p95_latency, p99_latency, and cost_per_unit. GPU utilization alone is not enough. A server can show low utilization because traffic is sparse, because requests are blocked on CPU preprocessing, or because kernels are memory-bound rather than compute-bound.

Configuration Anatomy

A practical capacity configuration names the model, instance type, precision, maximum sequence length, maximum batch size, batching delay, replica bounds, scaling metric, and admission limit. Those fields are a contract between the model team and the platform team. If the model team doubles maximum context length without changing capacity assumptions, the key-value cache can consume the memory budget and cause out-of-memory failures.

Setting Engineering effect
precision Controls bytes per parameter and available kernels, such as fp16, bf16, int8, or int4.
max_batch_size Caps throughput packing and worst-case memory growth.
max_queue_delay_ms Trades small wait time for better batching efficiency.
max_concurrency Rejects or queues excess work before memory collapse.
scale_metric Chooses whether replicas react to queue depth, latency, utilization, or request rate.

Example 1: Memory Fit Before Deployment

This first example estimates memory for weights, key-value cache, and runtime overhead. It is intentionally simple, but it captures the first capacity gate: if the estimate exceeds device memory with headroom, the deployment should not proceed.

import math

def gpu_memory_gib(params_billion, bytes_per_param, kv_cache_gib, overhead_gib):
    weights = params_billion * 1_000_000_000 * bytes_per_param / (1024 ** 3)
    return math.ceil((weights + kv_cache_gib + overhead_gib) * 10) / 10

for name, params, bytes_per_param, kv_cache, overhead in [
    ("7B fp16", 7, 2, 6, 3),
    ("7B int8", 7, 1, 6, 3),
    ("13B fp16", 13, 2, 8, 4),
]:
    print(f"{name}: {gpu_memory_gib(params, bytes_per_param, kv_cache, overhead)} GiB")

The deterministic output is 7B fp16: 22.1 GiB, 7B int8: 15.6 GiB, and 13B fp16: 36.3 GiB. The lesson is that quantization may turn a model that needs a larger accelerator into one that fits a smaller class, but key-value cache and overhead still matter. A 16 GB GPU may appear sufficient for int8 weights alone, yet fail once realistic concurrency is included.

Example 2: Cost per Million Tokens

Cost comparisons should normalize by useful work. Hourly GPU price alone is misleading because a more expensive accelerator may complete more work per hour. The next example compares serving choices using tokens per second.

models = {
    "A10-small-batch": {"hourly": 1.20, "tokens_per_s": 550},
    "L4-batched": {"hourly": 0.80, "tokens_per_s": 720},
    "A100-low-latency": {"hourly": 3.40, "tokens_per_s": 2100},
}

for name, metrics in models.items():
    cost = metrics["hourly"] / (metrics["tokens_per_s"] * 3600) * 1_000_000
    print(f"{name}: ${cost:.2f} per 1M tokens")

The output is A10-small-batch: $0.61 per 1M tokens, L4-batched: $0.31 per 1M tokens, and A100-low-latency: $0.45 per 1M tokens. The cheaper hourly instance is not always the cheaper serving path, and the most expensive instance can be cost-effective when latency objectives prevent deep batching.

Example 3: Replica Count from Concurrency

Little’s Law gives a useful first estimate: concurrency is arrival rate multiplied by time in system. If 18 requests per second spend 1.4 seconds in service at p95, the platform should expect roughly 25 concurrent requests before safety margin. If each GPU can handle 12 concurrent sequences, three GPUs are needed.

import math

traffic_rps = 18
p95_seconds = 1.4
safety_factor = 1.25
max_concurrent_per_gpu = 12

required = math.ceil(traffic_rps * p95_seconds * safety_factor / max_concurrent_per_gpu)
print(f"gpus_needed={required}")

The output is gpus_needed=3. This is a starting point, not a final autoscaling policy. If traffic is spiky, cold starts are slow, or model loading takes minutes, the minimum replica count may need to stay above zero even during quiet periods.

Design Choices and Trade-offs

Model size versus serving cost: a larger model may improve quality but reduce throughput, increase memory, and require expensive accelerators. The right decision compares business value per request against cost per request and latency impact.

Quantization versus accuracy: int8 or int4 weights can reduce memory and increase cache residency. The trade-off is possible quality loss, calibration work, or incompatibility with chosen kernels. Evaluate task-specific metrics after quantization rather than assuming quality is preserved.

Batching versus latency: dynamic batching improves throughput when requests arrive close together. Interactive APIs usually need a tight queue delay. Offline embedding jobs can use larger batches because user-facing latency is less important.

Autoscaling metric: request rate is easy to understand, but it ignores input length and output length. GPU utilization can lag or hide queueing. Queue depth and oldest-request age are often better for inference because they reveal demand before latency objectives are missed.

Failure Modes and Troubleshooting

Symptom: pods restart with CUDA out-of-memory errors shortly after a traffic increase. Cause: the memory plan counted weights but not key-value cache at production context length. Diagnostics: compare prompt lengths, generated token counts, batch size, and peak GPU memory at the failure time. Correction: lower maximum context or batch size, enable quantization, choose a larger GPU, or enforce admission control before memory exhaustion.

Symptom: p95 latency doubles while GPU utilization stays near 45 percent. Cause: preprocessing, tokenization, or network transfer is bottlenecking before GPU execution. Diagnostics: break latency into queue, CPU preprocess, GPU execute, and serialization spans. Correction: add CPU workers, cache tokenization where valid, move preprocessing closer to the server, or use a model server with better batching support.

Symptom: monthly cost rises faster than request volume. Cause: minimum replicas, idle GPUs, larger outputs, or failed retries are consuming capacity. Diagnostics: graph cost per successful request, tokens per request, retry rate, and idle accelerator hours. Correction: right-size minimum replicas, cap generated output, tune retry policy, and move batch workloads to cheaper scheduled capacity.

Reliability, Security, and Performance Implications

Reliability starts with admission control. It is better to return a controlled overload response than to accept work that causes process death and drops every in-flight request. Use request size limits, maximum output limits, queue limits, and backpressure. For long-running generation, stream partial output only when clients can handle cancellation and retry semantics.

Security affects cost because unbounded prompts and outputs are resource-amplification vectors. Authentication, per-tenant quotas, and abuse limits prevent one caller from occupying the GPU pool. Logs should capture request shape, tenant, model version, and failure category, but not sensitive prompt contents unless policy explicitly permits it.

Hands-on Lab: Build a Capacity Sheet

Prerequisites: access to recent production or staging request counts, p95 latency target, average and p95 input size, average and p95 output size, model parameter count, intended precision, candidate GPU hourly cost, and a place to run the Python snippets.

  1. Measure baseline demand for one representative hour: requests per second, p95 latency, average output units, and peak output units.
  2. Estimate memory using the first example. Add at least 15 percent headroom for allocator overhead and traffic variation.
  3. Benchmark one candidate instance with realistic inputs. Record successful requests per second, tokens or predictions per second, p95 latency, and peak memory.
  4. Compute normalized cost using the second example. Compare at least two instance choices or precision settings.
  5. Estimate replica count using the third example, then set minimum and maximum replicas from traffic shape and cold-start time.
  6. Define verification: a load test that reaches expected peak, an overload test that returns controlled errors, and a dashboard showing queue depth, latency, memory, throughput, and cost per unit.
  7. Cleanup by deleting temporary benchmark endpoints, releasing reserved accelerators, and removing test quotas or synthetic traffic generators.

Verification passes when the chosen configuration meets the p95 target at expected peak traffic, keeps peak memory below the planned limit, rejects overload without process crashes, and produces a cost per unit that the product owner accepts.

Assessment Exercises

  1. A 13B fp16 model fits in memory during a single-request test but fails during production traffic. Identify three measurements needed before changing instance type.
  2. You can cut cost by 40 percent with int4 quantization, but offline quality drops slightly. Design an evaluation that decides whether the cost reduction is acceptable.
  3. An autoscaler uses GPU utilization and reacts slowly during request spikes. Propose a better scaling signal and explain why it detects pressure earlier.
  4. A batch embedding job and a chat endpoint share one GPU pool. Describe the scheduling rules that prevent the batch job from harming interactive latency.
  5. Given hourly cost and measured throughput for two accelerators, compute cost per million outputs and explain when the more expensive instance might still be preferred.

Summary

GPU cost and capacity engineering is the discipline of connecting model behavior to measurable platform limits. Estimate memory before deployment, benchmark throughput with realistic inputs, normalize cost by useful work, scale on queue pressure and latency, and protect the system with admission control. In MLOps, these numbers belong beside accuracy metrics because they decide whether a model can run reliably, quickly, and economically after release.