SLOs Error Budgets and Service-Level Indicators

Published by Programming Line in System Design and Distributed Systems. Last updated .

Purpose: reliability promises that can be measured

Service-level indicators, objectives, and error budgets turn reliability from a vague desire into a design control. In system design, they answer three questions: what user-visible behavior matters, how good does it need to be, and how much unreliability can the service spend while still meeting the promise? An SLI is the measurement, such as the percentage of checkout requests that complete successfully within 300 ms. An SLO is the target over a time window, such as 99.9% of checkout attempts over 30 days. The error budget is the permitted failure: for a 99.9% request-based SLO, 0.1% of eligible events may be bad before the objective is missed.

The outcome is not only a dashboard number. A good SLO changes architecture decisions. It tells a team whether to add redundancy, reduce dependency fan-out, pause risky launches, investigate tail latency, or accept a known imperfection because users are still inside the promised envelope.

Mechanism: from events to budget burn

Most production SLOs are calculated as a ratio of good events to valid events. The numerator counts events that met the definition of acceptable behavior. The denominator counts events that were eligible to be judged. The details matter. If a client cancels a request after 20 ms, should it count? If a health check fails, is that user-visible? If a background retry succeeds after the first failure, did the user experience a bad event? The SLI definition must settle these questions before the numbers are useful.

There are two common shapes. Request-based SLIs classify each event as good or bad, then aggregate the ratio. Window-based SLIs classify time intervals, such as one-minute windows, as healthy or unhealthy. Request-based SLIs usually match high-volume APIs because they weight actual user traffic. Window-based SLIs are useful when traffic is sparse, when the service is either globally available or not, or when monitoring systems report state rather than individual events.

Error budget math is deliberately simple. If the SLO is 99.95%, the budget is 0.05% bad events during the SLO window. With 20,000,000 eligible requests in 30 days, the budget is 10,000 bad requests. If the service has already produced 7,000 bad requests, 3,000 remain. Burn rate compares the current failure rate with the budgeted failure rate. A 10x burn rate means the service is consuming budget ten times faster than it can sustain for the whole window.

SLI anatomy and configuration choices

An SLI specification has a name, a population, a success predicate, an aggregation window, and exclusions. A checkout availability SLI might define population as POST requests to the checkout endpoint from real users, success as HTTP 2xx or a confirmed idempotent duplicate, and exclusions as synthetic probes, load tests, and client cancellations before the request reached the service. A latency SLI might define success as status 2xx with server duration below 300 ms, which means a slow successful request is still bad for that SLI.

The instrumentation should prefer server-side counters for consistency, but client-side measurements are valuable when network, DNS, or CDN behavior is part of the user promise. Many teams keep both: a server SLI for operational diagnosis and a user-journey SLI for product reality. The SLO target should be chosen from user needs and economic trade-offs, not from the best number the system happened to hit last month.

Example 1: request success for a single API

Suppose a profile API receives 1,000,000 valid requests in a day. A request is good when it returns a non-error response in under 500 ms. The counters show 998,400 good requests. The daily SLI is 99.84%. Against a 99.9% objective, only 1,000 bad events are allowed, but the service produced 1,600. The expected behavior is clear: the service missed the daily objective, used 160% of the daily budget, and needs either immediate mitigation or a decision to slow change until the rolling window recovers.

requests = [
    {"window": "morning", "total": 400_000, "good": 399_600},
    {"window": "afternoon", "total": 350_000, "good": 349_300},
    {"window": "evening", "total": 250_000, "good": 249_500},
]

slo = 0.999
total = sum(row["total"] for row in requests)
good = sum(row["good"] for row in requests)
bad = total - good
allowed_bad = round(total * (1 - slo))
remaining = allowed_bad - bad

print(f"availability SLI: {good / total * 100:.3f}%")
print(f"allowed bad events: {allowed_bad}")
print(f"actual bad events: {bad}")
print(f"budget remaining: {remaining}")
print(f"budget used: {bad / allowed_bad * 100:.1f}%")

The deterministic output is: availability SLI 99.840%, allowed bad events 1000, actual bad events 1600, budget remaining -600, and budget used 160.0%. This small calculation is the same mechanism a dashboard uses, only with larger counters and rolling windows.

Example 2: latency as a separate promise

A search service can be available while still feeling broken. Imagine 2,000,000 successful searches, but 80,000 take longer than one second. An availability SLI might report nearly 100%, while a latency SLI reports 96% if the objective is successful search responses under one second. Keeping availability and latency separate prevents a fast error from looking better than a slow success. The expected operational response is also different: availability failures point toward crashes, overload, bad deploys, or dependency errors; latency failures point toward queueing, hot shards, cache misses, or expensive query plans.

SELECT
  COUNT(*) AS eligible_requests,
  SUM(CASE WHEN status_code BETWEEN 200 AND 399 AND latency_ms <= 1000 THEN 1 ELSE 0 END) AS good_requests,
  ROUND(
    100.0 * SUM(CASE WHEN status_code BETWEEN 200 AND 399 AND latency_ms <= 1000 THEN 1 ELSE 0 END) / COUNT(*),
    3
  ) AS latency_sli_percent
FROM search_request_log
WHERE route = '/search'
  AND synthetic = false
  AND received_at >= CURRENT_DATE - INTERVAL '1 day';

This SQL fragment treats a slow success as bad for the latency SLI, excludes synthetic probes, and calculates the ratio from raw request rows. If the table contains 2,000,000 eligible searches and 1,920,000 meet the status and latency predicate, the expected result is a latency SLI of 96.000%. That misses a 99% latency SLO even though most requests returned successful HTTP status codes.

The design choice is whether to measure a percentile or an event ratio. A p99 target under one second is easy to discuss, but ratio SLIs are easier to combine with error budgets: count each request that meets the threshold as good. Percentiles can hide volume and are hard to aggregate correctly across instances. Ratio SLIs make the objective directly auditable.

Example 3: multi-window burn-rate alerting

A 30-day 99.9% SLO has a 0.1% budget. If the last five minutes show a 2% bad-event rate, the burn rate is 20x because 2% is twenty times the allowed 0.1%. A short-window alert catches sharp incidents quickly, but it may page for tiny blips. A long-window alert, such as one hour or six hours, confirms sustained harm but detects slowly. Many teams pair windows: page when both a short window and a longer window burn too quickly. The expected behavior is fewer pages for harmless spikes and faster pages for incidents that are both severe and sustained.

slo:
  name: checkout-availability
  objective: 0.999
  window_days: 30
  sli:
    eligible_events: http_requests_total{service="checkout"}
    bad_events: http_requests_total{service="checkout",result="bad"}
  page_when:
    - short_window: 5m
      long_window: 1h
      burn_rate_above: 14.4
    - short_window: 30m
      long_window: 6h
      burn_rate_above: 6

This YAML fragment records the pieces an alerting system needs: the objective, the rolling window, the total-event counter, the bad-event counter, and paired burn-rate thresholds. The expected outcome is a page only when both windows in a pair show excessive burn, which filters out isolated counter spikes while still catching severe incidents quickly. Low-traffic services often need longer windows or minimum-event guards because a single bad request can otherwise create a misleading burn rate.

Example 4: declaring an SLO contract for review

Teams often keep SLO definitions in a versioned service catalog so reviewers can see what changed when a new endpoint, dependency, or exclusion appears. The definition below is not a monitoring query by itself; it is a compact review artifact that can feed dashboards, runbooks, and release policy.

{
  "service": "checkout",
  "slo": "payment-confirmation",
  "window": "30d",
  "objective": 0.999,
  "population": "customer POST /payments requests that reach the checkout edge",
  "good_event": "2xx response with confirmed payment or idempotent duplicate within 800 ms",
  "exclusions": ["synthetic probes", "load tests", "client disconnect before request body is received"],
  "owner": "checkout-platform"
}

The expected outcome is a definition that can be reviewed before implementation: the event population is user-visible, the success predicate includes both correctness and latency, and the exclusions are narrow. If a dependency timeout causes customers to wait two seconds before success, this SLO still counts the event as bad because it violates the 800 ms promise.

Design choices and trade-offs

The first trade-off is user journey versus component SLO. A component SLO for a database or queue helps internal owners reason about dependencies, but users care about a completed action. For system design interviews and real architecture reviews, start with the user journey, then derive component targets only where they help allocate reliability across dependencies.

The second trade-off is strictness. A 99.99% SLO sounds impressive, but it leaves only about 4.3 minutes of equivalent downtime in 30 days for an availability-style objective. That target may force expensive multi-region architecture, operational staffing, and conservative release practices. A 99.5% SLO may be appropriate for an internal analytics view that users can retry later. The right target is the cheapest target that preserves user trust and business requirements.

The third trade-off is what to exclude. Excluding client errors, test traffic, and clearly irrelevant probes keeps the SLI honest. Excluding dependency failures or peak traffic because they make the graph ugly destroys the purpose of the SLO. If users experienced harm, the event usually belongs in the denominator and may be bad.

Failure modes and troubleshooting

A common symptom is an SLI that stays perfect during a known incident. The likely cause is measuring only successful code paths, such as incrementing counters after a handler completes. Diagnose by comparing load balancer logs, application counters, and client telemetry for the same interval. Correct it by recording total eligible events at the edge of the request path and classifying failures before early returns, panics, and timeouts disappear from the data.

Another symptom is noisy pages on a quiet service. The cause is often a request-based burn alert without enough traffic. One failed request out of ten can look catastrophic even though the absolute user impact is small. Diagnose by checking denominator counts in the alert window. Correct it with a minimum-event condition, a longer alert window, or a window-based SLI if the service is naturally low volume.

A third symptom is budget confusion after retries are introduced. Server logs show more failed attempts, while users report fewer failed actions. The cause is mixing attempt-level and user-operation-level SLIs. Diagnose by tracing one user action through retry attempts and idempotency keys. Correct it by choosing the unit that matches the promise: attempts for backend capacity and dependency health, user operations for product-visible reliability.

Reliability and performance implications

SLOs shape release governance. When budget remains, teams can ship normal changes and accept managed risk. When the budget is exhausted or burning rapidly, the rational action is to reduce change, roll back risky releases, or invest in reliability work. This is not a moral rule; it is a feedback loop between measured user harm and engineering risk.

SLOs also expose dependency fan-out. If a request needs five downstream services and each is allowed to fail 0.1% of requests independently, the composed journey may fail more often than its product SLO permits. Designers can respond with caching, graceful degradation, hedged requests, queues, bulkheads, or fewer synchronous dependencies. Reliability is therefore part of the architecture, not a monitoring afterthought.

Hands-on lab: define and evaluate an SLO

Prerequisites: a shell with Python available and a small table of request counts. No external service is required. The lab models a checkout availability SLO over one day, but the same calculation applies to rolling 7-day or 30-day windows.

  1. Create a file or interactive Python session using the first example code.
  2. Run it and verify that the computed SLI is 99.840% and the budget is negative.
  3. Change the SLO from 0.999 to 0.995. Re-run the script. The allowed bad events become 5,000, so the same day is within budget.
  4. Change the evening good count from 249,500 to 249,950. Re-run the original 99.9% SLO. The actual bad events drop to 1,150, which is still over budget but much closer.
  5. Add a new row for night traffic and observe how both the numerator and denominator change. Do not average the three or four percentages; recompute from raw counts.

Verification is successful when you can explain why the budget changes with traffic volume and why summing raw good and total events gives a different answer than averaging window percentages. Cleanup is simply closing the interpreter or deleting the temporary file. If you changed a shared dashboard while adapting the lab, roll back experimental labels and alert thresholds after the exercise.

Assessment exercises

  1. A login service has 10,000 valid requests and 35 bad events in one hour. For a 99.9% SLO, calculate the budget, budget remaining, and whether the hour met the objective.
  2. Design an SLI for a video upload workflow where processing may complete minutes after the initial request. State the event population and the success predicate.
  3. Explain why averaging p99 latency from ten servers can produce a misleading service-level result. Propose a better SLI.
  4. A burn-rate alert pages during every nightly batch job, but users do not notice. List two diagnostic checks before changing the SLO target.
  5. Given a product SLO of 99.9% for checkout, name one architectural change that reduces dependency-related budget burn and explain its trade-off.

Summary

SLIs define the measured user-visible behavior, SLOs define the acceptable target over a window, and error budgets quantify how much failure can be tolerated before engineering behavior should change. The useful work is in the exact definitions: event population, success predicate, exclusions, windows, and burn-rate alerts. In reliability-oriented system design, these definitions connect user expectations to concrete choices about dependencies, redundancy, release pace, and troubleshooting.