Circuit Breakers Bulkheads Graceful Degradation and Recovery

Published by Programming Line in System Design and Distributed Systems. Last updated .

Purpose: contain failure before it becomes an outage

Circuit breakers, bulkheads, graceful degradation, and recovery are resilience patterns for keeping a distributed system useful when dependencies misbehave. In a system design interview or production design review, the goal is not to claim that failures disappear. The goal is to decide where failure is detected, how far it is allowed to spread, what users still receive, and how the system returns to normal without a second surge of traffic causing another collapse.

A circuit breaker protects a call path. It watches recent outcomes for a dependency, stops sending requests when the dependency appears unhealthy, and later probes cautiously. A bulkhead protects capacity. It partitions threads, connections, workers, queues, or budgets so one failing feature cannot consume the entire service. Graceful degradation protects user experience. It serves a reduced but honest response, such as cached recommendations, disabled comments, or asynchronous checkout confirmation. Recovery controls the path back to full service through probing, backoff, draining, and gradual re-enablement.

Mechanism: the moving parts

A circuit breaker is usually a small state machine placed around an outbound operation. In the closed state, requests flow normally and the breaker records results in a rolling window. The window might count the last 100 calls, all calls in the last 30 seconds, or calls grouped into time buckets. When the failure rate, timeout rate, or slow-call rate crosses a configured threshold and a minimum sample size has been reached, the breaker opens. In the open state, new calls fail fast without touching the dependency. After a sleep window, the breaker becomes half-open. It allows a limited number of trial calls. If enough trials succeed, the breaker closes. If trials fail or time out, it opens again.

The details matter. Counting every HTTP 404 as a dependency failure would open the breaker because of client mistakes. Ignoring timeouts would let callers pile up while waiting on an already-saturated service. A robust breaker classifies outcomes: success, business rejection, transient dependency failure, timeout, and local cancellation. It also defines the fallback path. A fallback may return cached data, a default value, a queued command, or a typed error that higher layers convert into a degraded page.

A bulkhead is a capacity boundary. A service may use one worker pool for payments, another for search, and a third for analytics. It may set separate connection pools for each downstream, queue limits per tenant, or concurrency semaphores per feature. If search stalls, only the search pool fills; login and checkout still have workers. Bulkheads work best when paired with deadlines. A separate queue with no maximum length still lets memory grow until the process fails.

Graceful degradation decides what the user sees when a protected call is skipped or fails. The degraded response should be explicit inside the system even if the user-facing copy is subtle. For example, a product page can render inventory, price, and add-to-cart from core services while replacing personalized recommendations with popular items from cache. The important design decision is the dependency hierarchy: which data is essential, which data is optional, and which commands must never be faked.

Configuration and API anatomy

Most resilience libraries expose the same concepts even when names differ. The breaker needs a measurement window, a minimum number of calls, thresholds, open-state duration, half-open trial count, timeout, and an exception or status classifier. The bulkhead needs a concurrency limit, queue limit, queue timeout, and rejection behavior. Degradation needs a fallback function and a response contract that can represent freshness, omission, or deferred work.

recommendationsClient:
  timeoutMillis: 250
  circuitBreaker:
    rollingWindowMillis: 30000
    minimumCalls: 40
    failureRatePercent: 50
    slowCallMillis: 200
    openForMillis: 10000
    halfOpenMaxCalls: 5
  bulkhead:
    maxConcurrentCalls: 30
    maxQueuedCalls: 20
    queueWaitMillis: 50
  fallback:
    strategy: cached_popular_items
    maxCacheAgeSeconds: 300

This fragment says recommendations are optional and latency-sensitive. Calls have a short deadline. The breaker will not open on the first few failures because it requires 40 samples, but it will open if half of the recent calls fail or become too slow. The bulkhead permits 30 active calls and 20 short-waiting calls. Once those limits are reached, callers immediately use cached popular items instead of waiting behind a growing queue.

Worked example 1: one dependency behind a breaker

The first example models a profile service calling an avatar service. The avatar is useful but not essential. After three failures in the last four calls, the breaker opens. While open, it returns a placeholder URL without calling the dependency. After a short reset period, it allows one probe.

from collections import deque

class CircuitBreaker:
    def __init__(self, window=4, failures_to_open=3, reset_after_ticks=2):
        self.window = deque(maxlen=window)
        self.failures_to_open = failures_to_open
        self.reset_after_ticks = reset_after_ticks
        self.state = "closed"
        self.opened_at = None

    def call(self, tick, operation, fallback):
        if self.state == "open" and tick - self.opened_at < self.reset_after_ticks:
            return fallback("open")
        if self.state == "open":
            self.state = "half-open"
        try:
            value = operation()
        except TimeoutError:
            self._record(False, tick)
            return fallback("timeout")
        self._record(True, tick)
        return value

    def _record(self, ok, tick):
        if self.state == "half-open":
            self.state = "closed" if ok else "open"
            self.window.clear()
            if not ok:
                self.opened_at = tick
            return
        self.window.append(ok)
        if len(self.window) == self.window.maxlen and self.window.count(False) >= self.failures_to_open:
            self.state = "open"
            self.opened_at = tick

failures = {2, 3, 4, 8}
breaker = CircuitBreaker()

def avatar_call(tick):
    if tick in failures:
        raise TimeoutError("avatar service timed out")
    return f"avatar-{tick}.png"

for tick in range(1, 10):
    result = breaker.call(
        tick,
        lambda tick=tick: avatar_call(tick),
        lambda reason: f"placeholder.png ({reason})",
    )
    print(tick, breaker.state, result)

The deterministic behavior is that ticks 2, 3, and 4 record timeouts, which opens the breaker. Tick 5 is blocked immediately and returns the placeholder with reason open. Tick 6 becomes a half-open probe; because it succeeds, the breaker closes. Tick 8 is a single later timeout and does not open the breaker by itself. The design prevents repeated waits on a dependency that is already showing a concentrated failure pattern.

Worked example 2: bulkheads stop queue starvation

Imagine an API process with 100 worker threads. Without bulkheads, slow report-generation calls can occupy all workers, and a simple password reset waits behind them. With bulkheads, reports might receive 20 workers and account operations 30 workers. When the report pool is full, new report requests are rejected or scheduled for background processing, while account operations continue.

class Bulkhead:
    def __init__(self, limit):
        self.limit = limit
        self.active = 0

    def try_start(self, request_name):
        if self.active == self.limit:
            return f"{request_name}: rejected by report bulkhead"
        self.active += 1
        return f"{request_name}: admitted"

reports = Bulkhead(limit=2)
accounts = Bulkhead(limit=2)

for request in ["report-1", "report-2", "report-3"]:
    print(reports.try_start(request))

print(accounts.try_start("password-reset"))

The expected behavior is isolation. In this small simulation, the third report request is rejected because the report bulkhead is full, but the password-reset request is admitted by the separate account bulkhead. In a real API, that rejection might become HTTP 429, a background job, or a cached report response. If 200 report requests arrive and each waits on a slow warehouse query, at most the configured report capacity executes and a bounded number queue. Password reset latency remains tied to the account pool, not to the report backlog. The trade-off is that report users see more immediate rejections during the incident, but the whole site does not become unavailable.

Worked example 3: graceful degradation on a product page

A product page usually has essential and optional data. Price, availability, and product title are essential because showing stale or invented values can cause incorrect orders. Reviews, recommendations, delivery estimates, and recently-viewed widgets may be optional. A degraded page can render the product with an add-to-cart button, omit reviews, and replace recommendations with a cached best-seller list marked internally as non-personalized.

{
  "product_id": "sku-42",
  "title": "Travel Mug",
  "price_status": "fresh",
  "recommendations": ["bestseller-1", "bestseller-2"],
  "degraded_components": ["recommendations"],
  "can_checkout": true
}

Expected behavior is a partial response rather than a blank page. This response keeps checkout enabled because price is fresh, substitutes best-sellers for personalized recommendations, and records degraded_components: ["recommendations"] for logs and metrics. If price lookup fails, the page should not guess; it should block checkout for that item or show a retryable error because the failed dependency is essential.

Design choices and trade-offs

Thresholds control sensitivity. A breaker that opens after two failures protects aggressively but may trip during harmless network jitter. A breaker that waits for hundreds of samples may react too slowly for a high-latency dependency. Minimum sample size is as important as failure percentage: 100 percent failure on two calls is weak evidence, while 60 percent failure on 100 calls is strong evidence.

Timeouts must be shorter than the caller’s remaining deadline. If the page service has 800 milliseconds to respond, a recommendation call should not wait 800 milliseconds by itself. The breaker should count slow calls because a dependency can be alive but too slow to be useful. Bulkhead limits should be sized from capacity, not hope: if a downstream can handle 50 concurrent requests from this service, setting a local limit of 500 only moves the overload downstream.

Fallbacks have correctness costs. Cached reads are usually safer than fabricated writes. Queuing a command can be valid for email sending or analytics, but dangerous for payment capture unless idempotency keys, user-visible status, and reconciliation are designed. Degradation should be boring and predictable: users should not see contradictory states across page refreshes because each component invented a different fallback.

Failure modes and troubleshooting

Symptom: the breaker opens during a small deploy and many users lose an optional feature. Cause: the minimum-call threshold is too low, or deploy warm-up produced a tiny burst of timeouts. Diagnostics: inspect breaker state transitions, sample counts, and failure classification during the time window. Correction: raise the minimum sample size, ignore local cancellations caused by deploy shutdown, and add jittered warm-up for new instances.

Symptom: latency stays high even though the breaker is open. Cause: callers still wait on a saturated queue or fallback performs another slow network call. Diagnostics: compare time spent before the protected call, inside the breaker, and inside fallback. Check thread pool queue depth and connection acquisition time. Correction: bound the queue, make fallback local or cached, and reject quickly when the bulkhead is full.

Symptom: recovery causes a second outage. Cause: all instances leave open state at the same time and flood the dependency with half-open probes and resumed traffic. Diagnostics: graph half-open transitions by instance and downstream request rate. Correction: add jitter to open duration, limit half-open probes, ramp traffic gradually, and keep downstream rate limits active during recovery.

Reliability and security implications

These patterns improve reliability by turning uncontrolled waiting into explicit decisions. They also reduce blast radius: one tenant, endpoint, or downstream should not consume every worker and connection. There are security implications too. Fallbacks must not bypass authorization checks. A cached response still needs to be scoped to the user or tenant. Error responses should not reveal internal breaker names, hostnames, or dependency topology to clients. Operationally, expose metrics for breaker state, fallback count, bulkhead rejection count, queue depth, timeout count, and recovery probes.

Hands-on lab: simulate failure and recovery

Prerequisites: Python 3 and a terminal. No external packages are required. The lab demonstrates the difference between waiting on a bad dependency and failing fast with degradation.

  1. Create a file named resilience_demo.py with the Python example from this lesson.
  2. Run python3 resilience_demo.py.
  3. Confirm that repeated avatar timeouts open the breaker and produce placeholder responses.
  4. Change failures = {2, 3, 4, 8} to failures = {2, 5, 8} and rerun it.
  5. Observe that isolated failures no longer open the breaker because the rolling window does not reach the failure threshold.

Verification is the sequence of printed states. In the first run, you should see an open state after the concentrated failures and a later return to closed after a successful probe. In the second run, the breaker should mostly remain closed because the dependency is flaky but not continuously failing. Cleanup is simply deleting resilience_demo.py. In a real service, rollback means disabling the new breaker configuration or restoring the previous thresholds, not removing timeouts entirely.

Assessment exercises

  1. A checkout page depends on tax calculation, recommendations, fraud scoring, and email receipt delivery. Classify each dependency as essential, deferrable, or optional, and justify the fallback behavior.
  2. A breaker opens whenever 5 of the last 10 calls fail. During low traffic, one user triggers all 10 calls. Explain the risk and propose a safer threshold policy.
  3. Your service has separate bulkheads, but all of them share one database connection pool. Describe how that weakens isolation and how you would redesign the pool limits.
  4. A half-open breaker allows 100 trial calls per instance across 80 instances. Estimate the possible probe surge and suggest a safer recovery design.
  5. Design the metrics and alerts that distinguish a healthy degraded mode from a growing outage.

Summary

Circuit breakers decide when to stop calling an unhealthy dependency, bulkheads decide how much capacity each dependency or feature can consume, graceful degradation decides what useful response remains, and recovery decides how normal traffic resumes. The patterns work together: a breaker without a fallback creates fast errors, a fallback without a breaker still waits too long, and a bulkhead without recovery planning can protect the caller while repeatedly overwhelming the dependency. Good system design makes these choices explicit for each call path.