Failure Modes Fault Domains and Dependency Mapping

Purpose and outcome

Reliable system design starts by admitting that every component can fail, and that the important question is not whether a failure happens but how far it travels. Failure modes describe the ways a component can fail. Fault domains describe the boundary within which one fault is likely to affect multiple resources. Dependency mapping shows which services, data stores, queues, networks, credentials, vendors, and human processes are required for a user-visible workflow to succeed.

The outcome of this lesson is a practical method for turning a vague architecture diagram into a failure model. You will learn to name the failure modes, draw the fault domains, mark hard and soft dependencies, reason about blast radius, and choose mitigations that match the actual risk. In the reliability and resilience section of system design, this is the connective tissue between availability targets, redundancy, incident response, and graceful degradation.

How the mechanism works

A dependency map is a directed graph. A node is something the system relies on: a service, database, cache, queue, object store, DNS provider, load balancer, region, identity provider, payment vendor, batch job, or operational team. An edge means one node needs another to complete a behavior. Edges should be labeled because not all dependencies are equal. A synchronous request dependency can block the caller immediately. An asynchronous queue dependency may delay work without failing the current user request. A control-plane dependency may be needed for deploys or failover but not for already-running traffic. A data dependency may be needed only for fresh results, while cached stale data may be acceptable.

Failure modes are attached to nodes and edges. A database can be down, slow, returning stale replicas, accepting writes but losing replication, rejecting connections because a pool is exhausted, or producing corrupt results. A network edge can have packet loss, high latency, partial partition, DNS resolution failure, or TLS certificate failure. A third-party API can return hard errors, throttle, hang until timeout, change schema, or succeed after charging the user twice. The map becomes useful when these modes are specific enough to predict symptoms and correction steps.

Fault domains add physical and administrative boundaries to the graph. A single process is a fault domain for memory leaks and crashes. A host is a domain for disk, kernel, and power faults. A rack or availability zone is a domain for top-of-rack networking and localized facility failures. A region is a domain for regional control planes, networking, and data replication. A cloud account, identity tenant, DNS zone, certificate authority, shared library, deployment pipeline, and on-call team can also be fault domains. Two services in separate zones still share a fault domain if both require the same identity provider during every request.

The internal reasoning loop is: choose a critical workflow, trace every required dependency, classify each edge, identify shared fault domains, inject a concrete failure mode, predict symptoms, then decide whether the current design contains, absorbs, retries, bypasses, or amplifies the failure. This loop is more precise than asking whether the architecture is highly available because it connects mechanisms to user impact.

Map anatomy and notation

A useful dependency map needs a small, consistent vocabulary. Use solid arrows for synchronous dependencies, dashed arrows for asynchronous dependencies, and dotted arrows for operational or control-plane dependencies if you are drawing. In text, record each edge with the caller, callee, dependency type, timeout, retry policy, fallback, data freshness requirement, owner, and criticality. Critical dependencies are required for the workflow to complete. Degraded dependencies allow a lower-quality result. Optional dependencies can be skipped without changing the main outcome.

For each node, record its fault domain labels: process, host, zone, region, account, vendor, tenant, schema, credential, and team where relevant. This is what exposes hidden coupling. A checkout service, payment service, and invoice service may be separate deployments, but if they share one database primary, that database is the real availability boundary for writes.

Example 1: checkout request path

Start with an online checkout workflow. The browser calls the web tier. The web tier calls cart, pricing, inventory, payment, and order services. Payment calls a card processor. Order service writes to the orders database and publishes an event to a queue. The critical synchronous path is web to cart, pricing, inventory reservation, payment authorization, and order write. The queue is important, but if email confirmation is asynchronous, a temporary queue outage may not need to fail checkout.

Expected behavior during a card processor timeout depends on design. If payment authorization is mandatory, the user should receive a clear retryable failure and the order must not be created as paid. If the payment service retries without idempotency keys, the same checkout can create duplicate charges. The dependency map makes the dangerous edge visible: payment service to card processor must have a timeout, bounded retries, idempotency key, and reconciliation path.

Example 2: regional search service

Consider product search served from two regions. Each region has web nodes, search API nodes, a local search index, and a replication stream from the catalog source of truth. If the catalog database in region A fails, region B can still serve searches if the search index is already replicated. However, if both regions query a single global ranking service synchronously, that ranking service becomes a shared fault domain. A ranking outage can break search globally even though the search indexes are regional.

A better design might treat ranking as a degraded dependency. The search API can return lexical or popularity-based results when ranking is unavailable. The expected output is not identical ranking quality, but the workflow still returns product results. That is graceful degradation: the map distinguishes the dependency needed for correctness from the dependency needed for optimal quality.

Example 3: analytics pipeline

An analytics pipeline receives events through an ingestion API, writes them to a durable log, processes them with workers, stores aggregates, and serves dashboards. The user-facing dependency is different from the data-freshness dependency. If workers stop, ingestion can continue and dashboards can show stale data with a freshness warning. If the durable log is unavailable, ingestion must fail or buffer locally with clear limits.

The expected behavior during worker failure is increasing processing lag, not lost events. The symptom should be lag metrics rising while ingestion success remains steady. If dashboard queries fail immediately when workers are down, the dependency map reveals an unnecessary synchronous coupling between serving and processing.

Design choices and trade-offs

The first major choice is whether to remove, soften, or isolate a dependency. Removing a dependency simplifies reliability but can reduce features. Softening a dependency with caches, fallbacks, or asynchronous processing preserves behavior under partial failure but creates staleness and reconciliation work. Isolating a dependency with bulkheads, separate pools, circuit breakers, and quotas reduces blast radius but increases operational complexity.

The second choice is where to place redundancy. Replicating every component across zones or regions is expensive and can introduce consistency problems. A stateful primary database is harder to make multi-region than stateless web servers. Sometimes the right answer is not active-active everywhere but a deliberately smaller failure domain: independent read replicas, per-tenant queues, per-vendor circuit breakers, or regional cells that prevent one tenant or region from consuming all capacity.

The third choice is timeout and retry behavior. Retrying can hide transient failures, but retries also multiply load against a struggling dependency. A dependency map should show retry layers so designers can avoid retry storms. If the browser retries, the gateway retries, the service retries, and the client library retries, one user action can become dozens of backend calls. Use deadlines, jittered backoff, idempotency, and retry budgets.

Failure modes and troubleshooting

Scenario: elevated latency across unrelated endpoints. Symptoms include slow requests in several services, connection pool saturation, and thread pools waiting on one downstream call. Cause may be a shared dependency that is slow rather than down, such as identity token validation or a shared database. Diagnostic steps: compare latency by dependency edge, inspect pool wait time, check whether timeouts exceed caller deadlines, and look for retry amplification. Correction: lower timeouts, add circuit breaking, cache validation results where safe, and split critical traffic into separate pools.

Scenario: one availability zone outage becomes regional outage. Symptoms include rising errors in all zones after one zone starts failing. Cause may be load balancers continuing to send traffic to unhealthy instances, cross-zone calls from healthy zones into the failing zone, or shared queues being consumed by workers in the bad zone. Diagnostic steps: group errors by zone, inspect service discovery records, and trace cross-zone calls. Correction: remove unhealthy zone endpoints, prefer same-zone calls when appropriate, partition queues by zone or cell, and test zonal evacuation.

Scenario: deploy rollback does not restore service. Symptoms include errors continuing after application rollback. Cause may be a schema migration, cache format change, feature flag, secret rotation, or external vendor behavior outside the application binary. Diagnostic steps: check recent control-plane changes, schema versions, config pushes, certificate updates, and dependency health. Correction: design reversible migrations, version cache keys, stage secret rotation, and include non-code dependencies in the incident timeline.

Security, performance, and reliability implications

Dependency maps are reliability tools, but they also expose security and performance issues. A shared credential used by many services is a security blast radius. A single identity tenant required on every request is both a security control point and an availability dependency. A common logging library that performs synchronous network writes can become a hidden performance dependency. A map should therefore include credentials, authorization checks, certificate chains, DNS, and telemetry paths when they affect request completion.

For performance, the map helps locate fan-out. A service that calls twenty downstream services synchronously has a high probability of tail latency because the slowest dependency determines the response. For reliability, the map shows where backpressure should exist. Queues, rate limits, bounded worker pools, and load shedding should protect scarce dependencies from overload.

Hands-on lab: build and query a dependency map

Prerequisites: Python 3, a terminal, and a small service workflow you can model. The following standalone script defines dependencies for checkout, lists impacted services when a node fails, and prints shared fault domains. It is intentionally small so you can adapt the data structure to your own architecture review.

from collections import defaultdict, deque

services = {
    'web': {'domains': {'region-us', 'zone-a'}},
    'checkout': {'domains': {'region-us', 'zone-a'}},
    'cart': {'domains': {'region-us', 'zone-a'}},
    'payment': {'domains': {'region-us', 'zone-b'}},
    'orders-db': {'domains': {'region-us', 'zone-b', 'primary-db'}},
    'card-processor': {'domains': {'vendor-cardco'}},
    'email-queue': {'domains': {'region-us', 'zone-c'}},
}

edges = [
    ('web', 'checkout', 'sync-critical'),
    ('checkout', 'cart', 'sync-critical'),
    ('checkout', 'payment', 'sync-critical'),
    ('checkout', 'orders-db', 'sync-critical'),
    ('payment', 'card-processor', 'sync-critical'),
    ('checkout', 'email-queue', 'async-degraded'),
]

reverse = defaultdict(list)
for caller, callee, kind in edges:
    reverse[callee].append((caller, kind))

def impacted_by(failed_node):
    seen = set()
    queue = deque([(failed_node, 'source')])
    while queue:
        node, via = queue.popleft()
        for caller, kind in reverse[node]:
            if caller not in seen:
                seen.add(caller)
                queue.append((caller, kind))
    return sorted(seen)

print('orders-db failure impacts:', ', '.join(impacted_by('orders-db')))
for name, data in services.items():
    if 'zone-b' in data['domains']:
        print('zone-b member:', name)

Steps: copy the script into a temporary file, run it, then change one dependency from sync-critical to async-degraded and discuss whether that change is technically honest. Verification: the original run should show that an orders database failure impacts checkout and web, and that payment and orders-db share zone-b. Cleanup: delete the temporary file after the exercise. Rollback for a real architecture document is simple: keep the original map version, record each modeling change as a review note, and do not update production runbooks until service owners confirm the edges.

The deterministic output is:

orders-db failure impacts: checkout, web
zone-b member: payment
zone-b member: orders-db

A compact review record can also be stored as YAML so dependency changes are inspectable in code review. This example captures the checkout-to-card-processor edge, the timeout, the retry budget, the required idempotency key, and the vendor fault domain.

workflow: checkout
edges:
  - from: checkout
    to: card-processor
    type: sync-critical
    timeout_ms: 1200
    retry_budget: 1
    idempotency_key: required
    fallback: fail-order-pending-payment
    expected_failure_behavior: user_sees_retryable_error_no_duplicate_charge
fault_domains:
  checkout:
    - region-us
    - zone-a
  card-processor:
    - vendor-cardco

Expected outcome: reviewers can see that the card processor is a synchronous critical dependency outside the company’s fault domains. During a timeout, checkout should fail into a pending-payment path or show a retryable error, and the idempotency requirement prevents duplicate charges if the user retries.

Assessment exercises

1. Pick a login workflow and classify each dependency as critical, degraded, optional, asynchronous, or control-plane. Which dependency is most likely to be forgotten in an incident?
2. A service has three replicas in three zones but all replicas use one regional cache synchronously. What failure modes can still affect every replica, and what design changes would reduce blast radius?
3. Explain why a slow dependency can be more dangerous than a failed dependency. Include timeout, retry, and connection pool behavior in your answer.
4. For a payment workflow, define the difference between retrying safely and retrying dangerously. What data must be included to make retries idempotent?
5. Given a dashboard that can tolerate ten minutes of stale data, redesign its dependency path so worker failure does not become dashboard outage.

Summary

Failure modes, fault domains, and dependency maps turn resilience from a slogan into a concrete design practice. The map shows what each workflow needs, how dependencies fail, which resources share a blast radius, and where graceful degradation or isolation is possible. Strong system designs do not merely add replicas; they understand the boundaries of failure, prevent unnecessary coupling, and make expected behavior under stress explicit before the incident happens.