Incident Response Runbooks and Data Reliability SLOs

Published by Programming Line in Data Engineering in Depth. Last updated .

Purpose: turn data incidents into bounded work

Incident response runbooks and data reliability SLOs give a data engineering team a shared way to decide when a pipeline problem is an incident, how severe it is, who acts first, and when the system is healthy enough to hand back to normal operations. The outcome is not a pretty document. The outcome is reduced ambiguity during a late load, a corrupt partition, a runaway backfill, or a dashboard outage that executives are using right now.

In this course, the topic belongs near the end because it connects modeling, orchestration, storage, quality checks, lineage, and platform operations. A runbook is useful only when it refers to real tables, jobs, partitions, service accounts, queues, and consumers. An SLO is useful only when it measures reliability from the data user’s point of view, such as whether the revenue mart is fresh by 8:00 a.m., whether order rows are complete, or whether a feature table remains within a tolerated null-rate threshold.

How data reliability SLOs work internally

A service level objective is a target over a measurement window. For data systems, the target usually describes a property of a dataset or pipeline output rather than an HTTP endpoint. Common indicators are freshness, completeness, correctness, availability, and uniqueness. Freshness asks how long it has been since the last successful publish. Completeness compares observed records, files, or partitions to an expected baseline. Correctness checks whether values satisfy business rules. Availability asks whether a consumer can read the data. Uniqueness tracks duplicate primary keys or duplicate event identifiers.

The SLO mechanism starts with an SLI, a service level indicator. The SLI is the actual measurement, such as fact_orders freshness_minutes. The SLO is the target, such as 99 percent of business-hour checks must find fact_orders no more than 90 minutes stale. The error budget is the allowed amount of badness in the window. If a monthly freshness SLO allows 1 percent bad checks and the monitor runs every ten minutes, then roughly 43 bad checks are available in a 30-day month. Spending the budget too quickly changes behavior: freeze risky schema changes, pause nonessential backfills, or prioritize remediation over feature delivery.

Good data SLOs measure the publish boundary, not only internal tasks. A scheduler may show all tasks green while the table is stale because a final swap failed, a permissions change blocked readers, or the warehouse cache still points at yesterday’s partition. For that reason, SLO monitors should query the same storage layer and access path used by the consumer whenever practical.

Runbook anatomy

A runbook is an executable decision aid written for a stressed operator. It needs a trigger, severity mapping, ownership, diagnostics, mitigation steps, recovery steps, verification, communication guidance, and rollback. The trigger ties directly to an SLO or alert condition. Severity maps impact: a single internal analyst table is different from a regulatory export or a machine learning feature set used in live decisions. Ownership says who has authority to stop jobs, replay data, quarantine a table, or publish a corrected dataset.

The diagnostic section should move from cheap, high-signal checks to deeper inspection. For a freshness incident, start with the most recent successful publish timestamp, scheduler state, upstream source arrival, and warehouse write errors. For a correctness incident, start with the failed rule, affected partitions, sample bad rows, and the first version where the failure appeared. The recovery section must be concrete: disable the publish step, replay partitions for a bounded interval, rebuild derived marts, compare row counts, and only then mark the incident mitigated.

Syntax and configuration choices

Teams store SLOs in monitoring tools, orchestration metadata, dbt tests, warehouse tables, or configuration files. The shape is usually the same: dataset, indicator, target, window, threshold, owner, severity, and runbook link. The exact storage is less important than keeping it close enough to the pipeline that changes are reviewed with the code. A freshness SLO for a table that no longer exists is worse than no SLO because it trains responders to distrust alerts.

A practical SLO record can be read as: for fact_orders, measure freshness every ten minutes by selecting the maximum published timestamp from the production table; consider the check good when freshness is 90 minutes or less; evaluate over a 30-day rolling window; page the analytics platform on-call engineer when the error budget burn rate exceeds the chosen threshold. That record gives both machines and humans enough information to act.

Example 1: measuring freshness and completeness

The first example calculates two SLIs for one publish. The deterministic output shows freshness is inside the 90-minute target, but completeness misses a stricter 99.9 percent target, so the combined SLO result is false.

from datetime import datetime, timedelta

now = datetime(2026, 9, 7, 12, 0, 0)
last_success = datetime(2026, 9, 7, 10, 42, 0)
records_expected = 1000000
records_loaded = 998700

freshness_minutes = int((now - last_success).total_seconds() // 60)
completeness = records_loaded / records_expected

freshness_ok = freshness_minutes <= 90
completeness_ok = completeness >= 0.999

print(f'freshness_minutes={freshness_minutes}')
print(f'completeness={completeness:.4%}')
print(f'slo_met={freshness_ok and completeness_ok}')

Expected output:

freshness_minutes=78
completeness=99.8700%
slo_met=False

This distinction matters operationally. A dashboard may look current while silently missing 1,300 records. The runbook should avoid telling responders to restart the scheduler when the real symptom points to source extract loss or filtering logic.

Example 2: choosing the right runbook from symptoms

The next example turns measurements into a first response. It intentionally prioritizes severe freshness delay before completeness and duplicate checks because a table that has not published for four hours usually blocks many downstream consumers.

from dataclasses import dataclass

@dataclass(frozen=True)
class Symptom:
    table: str
    freshness_minutes: int
    completeness: float
    duplicate_rate: float


def choose_runbook(symptom: Symptom) -> str:
    if symptom.freshness_minutes >= 180:
        return 'RUNBOOK-01: restore pipeline scheduling and replay missing partitions'
    if symptom.completeness < 0.99:
        return 'RUNBOOK-02: isolate source extract loss before publishing downstream marts'
    if symptom.duplicate_rate > 0.001:
        return 'RUNBOOK-03: stop writer, deduplicate idempotency keys, then resume'
    return 'RUNBOOK-00: monitor only, no incident declared'

case = Symptom('fact_orders', freshness_minutes=241, completeness=0.998, duplicate_rate=0.0)
print(choose_runbook(case))

Expected output:

RUNBOOK-01: restore pipeline scheduling and replay missing partitions

In a real system the rules can live in an alert router or incident management platform, but the logic should remain explainable. Responders must be able to see why one page was sent and another was suppressed.

Example 3: tracking an error budget

Error budgets keep the team from treating every defect as equal. This example records three bad intervals against a monthly budget of 43 bad minutes. After the third interval, the budget is exhausted.

from dataclasses import dataclass

@dataclass
class ErrorBudget:
    allowed_bad_minutes: int
    used_bad_minutes: int = 0

    def record(self, bad_minutes: int) -> None:
        self.used_bad_minutes += bad_minutes

    @property
    def remaining(self) -> int:
        return max(self.allowed_bad_minutes - self.used_bad_minutes, 0)

    @property
    def exhausted(self) -> bool:
        return self.used_bad_minutes >= self.allowed_bad_minutes

budget = ErrorBudget(allowed_bad_minutes=43)
for interval in [8, 12, 27]:
    budget.record(interval)
    print(f'used={budget.used_bad_minutes} remaining={budget.remaining} exhausted={budget.exhausted}')

Expected output:

used=8 remaining=35 exhausted=False
used=20 remaining=23 exhausted=False
used=47 remaining=0 exhausted=True

Once exhausted, the policy might require a release hold for the affected pipeline, a post-incident review, and work on the highest recurring cause before more feature work lands.

Design choices and trade-offs

Freshness targets are easy to explain, but they can hide corrupt data. Correctness targets catch business failures, but they require domain rules and careful exception handling. Completeness targets are powerful when expected volume is predictable; they are noisy for seasonal or event-driven sources unless baselines include calendar effects. Availability targets are useful for governed data products because permissions, catalog state, and warehouse outages can break consumers even when the pipeline ran.

Choose paging alerts only for SLOs tied to real user impact. A failed staging-table test might create a ticket, while a missed executive revenue publish creates a page. This protects attention and makes incident declarations credible. Also decide whether to fail closed or fail open. Failing closed means withholding a publish when checks fail, preserving trust but increasing staleness. Failing open means publishing with warnings, reducing delay but risking bad decisions. Financial, regulatory, and model-serving datasets often fail closed; exploratory analytics often fail open with visible annotations.

Failure modes and troubleshooting

Symptom: the orchestrator is green, but the freshness SLO is failing. Cause: the final table swap, catalog update, or permission grant failed after task success was recorded. Diagnostics: query the production table’s max publish timestamp, inspect the final publish logs, and check whether consumers can read the object with their role. Correction: rerun only the publish step if intermediate data is valid, then verify from the consumer role.

Symptom: completeness drops every Monday. Cause: the baseline ignores weekend batching or a source system’s weekly maintenance. Diagnostics: compare several weeks of partition counts by day, inspect source delivery timestamps, and separate missing data from delayed data. Correction: update the expected-volume model and set a different Monday threshold only if the consumer impact is acceptable.

Symptom: duplicate-rate alerts repeat after each replay. Cause: the ingestion job appends reprocessed files without idempotent merge keys. Diagnostics: group by natural key and file name, identify repeated load identifiers, and inspect whether the target write path uses insert-only semantics. Correction: stop the writer, remove affected partitions or merge by idempotency key, then replay from the last known good watermark.

Reliability and security implications

Runbooks often grant responders the power to pause jobs, replay data, and modify production tables. Those actions need least-privilege roles, audited commands, and explicit rollback. Avoid embedding shared passwords or long-lived tokens in runbooks. Link to secret names and access procedures instead. For performance, SLO queries must be cheap and bounded. A monitor that scans a multi-terabyte fact table every five minutes can become its own incident. Store publish metadata, partition counts, and validation summaries in compact audit tables so reliability checks do not compete with customer workloads.

Hands-on lab: build a mini data incident loop

Prerequisites: Python 3, a terminal, and permission to create a temporary working directory. No external service is required.

  1. Create a temporary directory and save the three Python examples as separate files named sli_check.py, runbook_router.py, and budget.py.
  2. Run python sli_check.py. Confirm that freshness is 78 minutes and the combined SLO is false because completeness is below 99.9 percent.
  3. Run python runbook_router.py. Confirm that the selected response is RUNBOOK-01.
  4. Run python budget.py. Confirm that the final line reports exhausted=True.
  5. Change the completeness threshold in the first script from 0.999 to 0.995 and rerun it. The SLO should now pass, which demonstrates how a target change alters incident behavior without changing the data.

Verification: the lab is complete when all expected outputs match, and you can explain which runbook would be used for a stale table versus a low-completeness table. Cleanup: delete the temporary directory. If you performed the lab in a shared repository, remove the scratch files rather than committing them.

Assessment exercises

  1. A payments mart is fresh but missing 3 percent of rows from one source. Which SLI failed, and why would restarting the scheduler be the wrong first action?
  2. Design an SLO for a daily regulatory export. Specify indicator, target, measurement window, owner, and whether the publish should fail closed or fail open.
  3. An alert fires after a backfill, showing duplicate customer records. List the first three diagnostics you would run before deleting data.
  4. Your team has exhausted the error budget for a feature table twice in one quarter. What engineering work should compete with new feature delivery, and how would you justify it?

Summary

Data incident runbooks and reliability SLOs translate vague pipeline trouble into measured operational decisions. The SLI says what is wrong, the SLO says whether it violates the promise to users, the error budget says how much failure remains tolerable, and the runbook says what to do next. The strongest implementations measure the consumer-facing data product, route alerts by impact, keep remediation steps specific to the pipeline, and verify recovery from the same path consumers use.