Pipeline Dependencies Retries SLAs and Failure Recovery

Purpose and outcome

A data pipeline is rarely one job. It is a graph of extraction, validation, transformation, loading, publication, and notification steps. Pipeline dependencies, retries, SLAs, and failure recovery are the mechanisms that let an orchestrator run that graph in the right order, tolerate temporary problems, and make missed business deadlines visible. In a data engineering workflow, these controls decide whether a dashboard waits for validated facts, whether a transient API timeout is retried automatically, and whether a partially loaded table is repaired without duplicating rows.

By the end of this lesson, you should be able to read a pipeline graph, choose dependency edges deliberately, configure retry behavior with bounded side effects, define practical service level agreements, and design recovery steps for common orchestration failures. The examples use Python to model the mechanics, but the ideas map directly to orchestrators such as Airflow, Dagster, Prefect, Argo Workflows, and cloud-native schedulers.

How dependency graphs work

Most orchestrators represent a pipeline as a directed acyclic graph, often shortened to DAG. A node is a task: extract orders, validate customer records, build a dimension table, refresh a mart, or publish a report. A directed edge means one task must complete before another can start. If task B depends on task A, then B is downstream of A and A is upstream of B. The graph must be acyclic because a cycle creates an impossible schedule: A waits for B while B waits for A.

The scheduler repeatedly looks for runnable tasks. A task is runnable when all required upstream tasks are in a successful terminal state for the same logical run. The scheduler then hands the task to an executor, records state transitions, and reevaluates downstream tasks when the task finishes. Common states include queued, running, success, failed, skipped, retrying, and upstream_failed. These states are not cosmetic. They determine whether downstream work proceeds, waits, or is suppressed.

Dependencies should express data correctness, not merely code order. If a daily sales mart consumes orders and customers, it should depend on both upstream datasets or tasks. If a notification only reports final status, it should depend on the publish step, not every intermediate task. Overly broad dependencies reduce parallelism and make unrelated incidents block the pipeline. Missing dependencies create races where a transformation reads stale or incomplete data.

Retry and SLA anatomy

A retry policy defines what happens after a task fails before the orchestrator declares the task permanently failed. The common parameters are retry count, delay, exponential backoff, maximum delay, retryable error classification, and timeout. A good retry policy is narrow enough to avoid hiding real defects and broad enough to absorb transient failures such as network resets, database connection limits, object store throttling, and temporary API errors.

Retries only work safely when task effects are controlled. An extraction task that writes a new file on every attempt can create duplicate input. A load task that appends rows without a run identifier can double count facts. A safer design stages data under a deterministic partition or run key, validates it, and then performs an idempotent merge or atomic swap. Idempotent means running the same task again produces the same final state.

An SLA is a deadline or freshness expectation, such as daily revenue mart available by 07:00 UTC or clickstream partition no more than 30 minutes late. An SLA miss is different from a task failure. A task can succeed but finish too late for the business process that consumes it. Orchestrators usually record SLA misses separately so operators can alert, investigate queue pressure, upstream lateness, or slow transformations even when the data eventually arrives.

Example 1: resolving task order

This first example builds a small dependency graph and computes a valid run order. The exact order matters less than the rule: no task appears before its upstream requirements. Independent tasks, such as extracting orders and customers, can run in parallel in a real executor even though this simple script prints one linear order.

from collections import defaultdict, deque

tasks = {
    "extract_orders": [],
    "extract_customers": [],
    "validate_orders": ["extract_orders"],
    "build_daily_sales": ["validate_orders", "extract_customers"],
    "publish_dashboard": ["build_daily_sales"],
}

children = defaultdict(list)
remaining_upstreams = {task: len(upstreams) for task, upstreams in tasks.items()}
for task, upstreams in tasks.items():
    for upstream in upstreams:
        children[upstream].append(task)

ready = deque(task for task, count in remaining_upstreams.items() if count == 0)
order = []
while ready:
    task = ready.popleft()
    order.append(task)
    for child in children[task]:
        remaining_upstreams[child] -= 1
        if remaining_upstreams[child] == 0:
            ready.append(child)

print("run order:", " -> ".join(order))

The expected output is run order: extract_orders -> extract_customers -> validate_orders -> build_daily_sales -> publish_dashboard. If an edge from extract_customers to build_daily_sales were omitted, the sales build might start before customer data is refreshed. If an unnecessary edge from extract_orders to extract_customers were added, the pipeline would still be correct but slower because two independent extracts could no longer run together.

Example 2: retry schedule

This example models exponential backoff. The first retry waits five minutes, the second waits ten, and the third waits twenty. Backoff protects dependencies such as APIs and warehouses from being hammered by immediate retry loops. The maximum delay prevents later attempts from drifting too far beyond the operational window.

def retry_times(start_minute, retries, delay_minutes, multiplier=2, max_delay=30):
    times = []
    current = start_minute
    delay = delay_minutes
    for _ in range(retries):
        current += min(delay, max_delay)
        times.append(current)
        delay *= multiplier
    return times

print("retry minutes:", retry_times(0, retries=3, delay_minutes=5))

The expected output is retry minutes: [5, 15, 35]. This policy is useful for a flaky source API. It is a poor fit for deterministic errors such as invalid SQL or a missing column introduced by a schema change; those should fail quickly and page the owner or open an incident. Retrying non-retryable failures wastes compute and delays downstream tasks that could otherwise be marked blocked.

Example 3: recovery after a partial run

Failure recovery depends on where the failure happened. A task may fail before writing data, after staging data, after loading a final table, or after sending an external notification. The following example stages data by batch identifier. If the process crashes after staging and before merging, rerunning the same batch overwrites the staging slot and then merges deterministically.

staging = {}
target = {}

batch = {
    "batch_id": "orders_2026_09_07",
    "rows": {101: 42.50, 102: 19.00},
}

def stage(batch):
    staging[batch["batch_id"]] = dict(batch["rows"])
    return "staged"

def merge(batch_id):
    for order_id, amount in staging[batch_id].items():
        target[order_id] = amount
    return "merged"

print(stage(batch))
print("target after crash:", sorted(target.items()))
print(stage(batch))
print(merge(batch["batch_id"]))
print("target after recovery:", sorted(target.items()))
finish_minute = 104
deadline_minute = 90
print("sla status:", f"missed by {finish_minute - deadline_minute} minutes")

The expected output shows an empty target after the simulated crash, then a correct target after rerun: [(101, 42.5), (102, 19.0)]. The final line reports an SLA miss of fourteen minutes. This demonstrates two separate operating concerns: the data was recovered correctly, but it still arrived late enough to violate the service expectation.

Design choices and trade-offs

Fine-grained tasks improve visibility and recovery. If validation fails, only the validation task needs attention, and upstream extraction output can often be reused. The trade-off is scheduler overhead and a larger graph to understand. Coarse-grained tasks are simpler to read but harder to restart precisely; one failure may require repeating expensive work.

Strict dependencies improve correctness when downstream tasks require complete upstream data. They can also increase latency when a task waits for an upstream dataset that is not actually needed for a particular partition. Partition-aware dependencies reduce this waste: the 2026-09-07 sales partition should wait for the 2026-09-07 orders partition, not for every historical customer backfill.

Retry counts should reflect the failure domain. A warehouse lock might clear within minutes, so a few retries are reasonable. A malformed file will not fix itself, so retries should be disabled or limited. Timeouts protect the schedule from tasks that hang indefinitely, but an aggressive timeout can kill legitimate large partitions. SLAs should be attached to business outcomes, not every internal task. Alerting on every small internal delay creates noise; alerting on missed freshness for a critical mart creates useful urgency.

Failure modes and troubleshooting

A common symptom is a downstream task stuck in queued or none state. The likely cause is an upstream task that did not reach success, a dependency on the wrong logical date, or exhausted executor capacity. Diagnostic steps are to inspect the graph view, compare logical run identifiers, check upstream terminal states, and review worker queue depth. The correction is either to fix the failed upstream task, clear and rerun the correct partition, or adjust the dependency so it points at the dataset actually required.

Another symptom is repeated duplicate rows after retries. The cause is usually a non-idempotent append or an output path that changes on every attempt. Diagnose by comparing row counts by run identifier, looking for duplicate natural keys, and checking task logs for multiple successful writes. Correct it by writing to deterministic staging locations, using merge keys, deleting the target partition before replacement, or making the final publish step atomic.

A third symptom is frequent SLA misses without task failures. Causes include upstream source lateness, queue saturation, inefficient transformations, or retry delays that exceed the freshness window. Diagnose by charting task duration, queue wait, retry count, and upstream arrival time separately. Correct the issue by moving the schedule later, increasing worker capacity, optimizing the slow task, splitting independent work, or renegotiating the SLA if the source cannot meet it.

Reliability and performance implications

Dependency design is a reliability control because it prevents consumers from seeing incomplete data. It is also a performance control because it defines available parallelism. Retry design is a reliability control because it absorbs temporary infrastructure faults, but it can become a performance problem when many failed tasks retry at once. Backoff, jitter, concurrency limits, and pools reduce retry storms. Recovery design protects correctness by making reruns boring: the operator should be able to clear a failed task and rerun it without manually editing tables.

Security also matters when recovery touches external systems. Retry logs should not print secrets from connection strings or API payloads. Recovery scripts should use scoped credentials that can repair the intended dataset but cannot overwrite unrelated warehouse schemas. Manual rerun permissions should be limited because clearing a task in an orchestrator can cause real writes to happen again.

Hands-on lab: build a recovery checklist

Prerequisites: Python 3, a terminal, and any text editor. Create a local file named pipeline_recovery_lab.py and paste the third example into it. Step 1: run the script and confirm that staging occurs before the simulated crash. Step 2: change finish_minute from 104 to 86 and rerun it. Verification: the target rows should remain the same, while the SLA line changes from missed to an on-time condition if you also update the print logic to compare finish and deadline. Step 3: add a third order to the batch and rerun the script. Verification: the final target should contain three sorted order rows and no duplicates. Step 4: simulate a bad recovery design by changing target from a dictionary to a list and appending rows during merge. Verification: rerunning merge twice creates duplicates, proving why keyed merge behavior matters.

Cleanup is simple because the lab only uses in-memory dictionaries. Remove the temporary file when finished. In a real warehouse lab, cleanup would include dropping staging tables, deleting test partitions, and disabling any test schedules or alerts.

Assessment exercises

1. A pipeline has extract_orders, extract_payments, join_orders_payments, and publish_finance_mart. Draw the minimum dependency graph that preserves correctness while allowing maximum parallelism.
2. A task calls a partner API and fails twice with connection timeouts before succeeding. Choose retry count, delay, and timeout settings, and explain how you would prevent retries from creating duplicate files.
3. A dashboard refresh succeeds every day but often completes twenty minutes after the executive meeting starts. Is this a task failure, an SLA miss, or both? What metrics would you inspect first?
4. A backfill for thirty historical partitions is running while the current daily partition waits behind it. Explain two scheduler or dependency changes that could protect the current SLA.
5. A load task failed after inserting half its rows into a final table. Describe a recovery approach that lets an operator rerun the task safely.

Summary

Pipeline dependencies define the order and parallelism of data work. Retries define how the system reacts to temporary task failures. SLAs define whether successful work arrived in time to be useful. Failure recovery defines how operators restore a correct state after something breaks. In orchestration and transformation work, these mechanisms are inseparable: a dependable data pipeline needs accurate edges, bounded retry policies, meaningful freshness targets, and idempotent task design so reruns repair the pipeline instead of making the data worse.