Backup Restore Disaster Recovery and Multi-Region Design

Published by Programming Line in System Design and Distributed Systems. Last updated .

Purpose and Recovery Outcome

Backup, restore, disaster recovery, and multi-region design are the parts of system design that decide what happens after data is deleted, a database is corrupted, a region is unreachable, or an operator ships a destructive change. The outcome is not simply having copies of data. The outcome is a practiced path from failure to a known good service state, with an acceptable amount of data loss and downtime.

Two terms control the design. Recovery point objective, or RPO, is the maximum age of data the business can tolerate losing. If orders must lose at most five minutes, a nightly backup is irrelevant for that workload. Recovery time objective, or RTO, is the maximum time the service can be unavailable or degraded. A system with a one-hour RPO and a ten-minute RTO needs very different machinery from one with a ten-minute RPO and a one-day RTO.

This lesson connects directly to reliability and resilience because backup and multi-region architecture are reliability controls with measurable behavior. They turn broad availability goals into concrete choices: snapshot frequency, log retention, replication topology, failover authority, restore automation, DNS behavior, and post-restore validation.

How Recovery Works Internally

A recoverable system usually has three layers of protection. The first layer is a backup: an independent copy of state stored outside the failure domain of the primary system. The second layer is a restore mechanism: tooling that can materialize that copy into a running database, object store, search index, or filesystem. The third layer is a disaster recovery plan: the orchestration that decides which components are restored, in what order, who approves promotion, and how clients are routed.

Database backups often combine full snapshots, incremental backups, and transaction logs. A full snapshot gives a complete base image. Incremental backups store changed blocks or changed records after a known base. Transaction logs record committed changes in order, allowing point-in-time recovery. To restore to 12:25, the system loads the latest valid snapshot before 12:25, replays committed log records up to 12:25, and refuses later records. That is why log retention must cover the desired restore window.

Object storage and file systems add different concerns. Versioning can protect against accidental overwrite, but it is not a full backup if the same compromised identity can delete every version. Cross-account or immutable storage reduces that risk. Search indexes and caches are usually reconstructed from a source of truth, so their recovery plan should emphasize rebuild throughput and queue replay rather than expensive snapshotting of derived state.

Multi-region design extends recovery from data copies to service placement. In active-passive design, one region serves traffic while another region receives backups or replication and waits for promotion. In warm standby, the passive region has running capacity but may be smaller than production. In active-active, multiple regions serve live traffic and must resolve writes, route users, and handle partial partitions. Active-active can reduce failover time, but it raises consistency and conflict-resolution complexity.

Configuration Anatomy

A useful recovery specification names the workload, the authoritative data store, the RPO and RTO, backup frequency, retention window, encryption boundary, restore target, validation checks, and failover decision owner. For multi-region systems it also names replication direction, maximum tolerated replica lag, health-check source, traffic-routing method, and split-brain prevention rule.

Backups should be classified by failure domain. A local snapshot helps with accidental table deletion but not regional storage failure. A cross-zone backup helps with host or zone loss but not credential compromise. A cross-account, cross-region, immutable backup helps with a larger set of disasters, at the cost of restore latency, storage charges, and more complicated access control.

Example 1: Choosing a Recovery Pattern

The first design step is mapping each workload to recovery objectives. A checkout database, an analytics warehouse, and an archive do not deserve the same recovery budget. This example turns RPO and RTO into a first-pass architecture choice.

from dataclasses import dataclass

@dataclass(frozen=True)
class Workload:
    name: str
    rpo_minutes: int
    rto_minutes: int

def choose_pattern(workload: Workload) -> str:
    if workload.rpo_minutes <= 15 and workload.rto_minutes <= 30:
        return "continuous replication plus warm standby"
    if workload.rpo_minutes <= 240 and workload.rto_minutes <= 480:
        return "hourly incrementals plus tested restore runbook"
    return "daily snapshots plus cold recovery"

for workload in [
    Workload("checkout", 5, 20),
    Workload("analytics", 180, 360),
    Workload("archive", 1440, 2880),
]:
    print(f"{workload.name}: {choose_pattern(workload)}")

The deterministic output is checkout using continuous replication and warm standby, analytics using hourly incrementals with a tested runbook, and archive using daily snapshots with cold recovery. The important lesson is that the recovery target comes before the tool choice. Buying synchronous multi-region replication for archival records wastes money, while relying on nightly snapshots for checkout violates the business requirement.

Example 2: Point-in-Time Restore

A point-in-time restore chooses a base snapshot and replays only the log entries committed before the requested time. This protects against both infrastructure failure and application bugs. If a bad migration ran at 12:26, restoring to 12:25 can recover the database just before the damage.

from datetime import datetime

snapshots = [
    ("snapshot-00", datetime(2026, 9, 6, 0, 0), 1000),
    ("snapshot-06", datetime(2026, 9, 6, 6, 0), 1180),
    ("snapshot-12", datetime(2026, 9, 6, 12, 0), 1400),
]
log_entries = [
    (datetime(2026, 9, 6, 12, 10), 42),
    (datetime(2026, 9, 6, 12, 20), -7),
    (datetime(2026, 9, 6, 12, 30), 15),
]
restore_to = datetime(2026, 9, 6, 12, 25)
base_name, base_time, value = max(s for s in snapshots if s[1] <= restore_to)
for entry_time, delta in log_entries:
    if base_time < entry_time <= restore_to:
        value += delta
print(base_name)
print(value)

The output is snapshot-12 and 1435. The restore begins at value 1400 from the noon snapshot, applies +42 at 12:10 and -7 at 12:20, and stops before the +15 entry at 12:30. In a real database, the same idea applies to write-ahead logs, binlogs, archived redo logs, or change streams. The operational risk is that missing, corrupt, or expired logs silently reduce the restore point choices.

Example 3: Regional Failover Selection

Failover should not simply pick any healthy region. A replica can be healthy but too far behind. Promoting it may bring the service back while losing more data than the RPO allows. This example filters candidates by both health and replication lag.

regions = {
    "us-east": {"healthy": False, "lag_seconds": 0, "priority": 10},
    "us-west": {"healthy": True, "lag_seconds": 18, "priority": 20},
    "eu-central": {"healthy": True, "lag_seconds": 95, "priority": 30},
}
maximum_safe_lag = 60
candidates = [
    (data["priority"], name)
    for name, data in regions.items()
    if data["healthy"] and data["lag_seconds"] <= maximum_safe_lag
]
selected = min(candidates)[1]
print(f"promote {selected}")
print("dns ttl seconds: 30")

The output is promote us-west and a DNS TTL of 30 seconds. The failed primary is excluded because it is unhealthy. The European region is excluded because its 95 seconds of lag exceeds the 60 second safety limit. The lower priority healthy region with acceptable lag is promoted. In production, this decision also needs fencing: the old primary must be prevented from accepting writes before the new primary becomes authoritative.

Design Choices and Trade-Offs

Synchronous replication can produce a very low RPO because a write is not acknowledged until another region or zone has it. The trade-off is write latency and reduced availability during network partitions. Asynchronous replication is faster and more tolerant of distance, but it creates lag. Lag becomes potential data loss during failover.

Cold recovery is cheap because capacity is created only after the disaster. It suits low-urgency workloads. Warm standby costs more because databases, queues, and application nodes already exist, but recovery is faster. Hot active-active service is the most demanding option: it needs global routing, duplicate-write handling, idempotent APIs, conflict resolution, and careful data locality rules.

Retention length is another trade-off. Long retention helps with slow detection, such as discovering a corrupted billing table days later. It increases storage cost and expands the set of sensitive historical data that must be encrypted, access controlled, and deleted according to policy. Immutable retention protects against ransomware and operator mistakes, but it also means accidental backup of forbidden data may be harder to purge.

Failure Modes and Troubleshooting

Symptom: a restore finishes but application checks fail because recent rows are missing. Cause: the latest transaction logs were not archived before the primary failed, so the real RPO was worse than the stated RPO. Diagnose: compare the requested restore timestamp, latest available log timestamp, and replica lag metrics at the time of failure. Correct: shorten log shipping intervals, alert on archive lag, and test restores using the same log source used in emergencies.

Symptom: failover succeeds, then data diverges after the old region comes back. Cause: both regions accepted writes, creating split brain. Diagnose: inspect DNS changes, load balancer target history, database promotion logs, and write timestamps from both regions. Correct: use fencing tokens, remove write credentials from the demoted primary, and require a single promotion controller.

Symptom: backup jobs are green, but restore takes many hours longer than the RTO. Cause: backups were measured by creation success, not restore throughput. Diagnose: run a timed restore drill into an isolated environment and measure snapshot hydration, log replay, index rebuild, and application smoke tests separately. Correct: pre-provision restore capacity, keep warm replicas for critical stores, or reduce backup fragmentation.

Security, Performance, and Reliability Implications

Backups contain production data and often bypass normal application authorization. They need encryption, separate access roles, audit logs, and tested key recovery. A perfectly preserved encrypted backup is useless if the key management system is only available in the failed region or if no emergency principal can decrypt it.

Backup jobs consume I/O, CPU, network bandwidth, and storage API quota. Poorly scheduled full backups can slow the primary database at peak hours. Incremental and log-based designs usually reduce this pressure, but they make restore ordering more important. Reliability improves when backup success, restore success, replication lag, immutable-copy status, and recovery drill duration are all monitored as first-class signals.

Hands-On Lab: Design and Verify a Recovery Plan

Prerequisites: a local shell with Python available, a text editor, and a disposable directory. No cloud account is required because the lab models the decisions rather than creating real infrastructure.

  1. Create a small inventory with three workloads: orders, reports, and media metadata. Assign each one an RPO, RTO, data size, and whether writes are user-facing.
  2. Run the first example and adjust thresholds until each workload maps to a pattern you can justify.
  3. Run the point-in-time restore example and change restore_to to 12:15 and 12:35. Verify that only log entries at or before the target time are included.
  4. Run the failover example and change us-west lag to 75 seconds. Verify that the selected region changes or that no candidate exists if every healthy region violates the lag limit.
  5. Write a five-step runbook: freeze writes, select restore point or replica, restore or promote, validate data and dependencies, route traffic.

Verification: your runbook should state the expected RPO and RTO for each workload, the exact signal that permits promotion, and the validation query or smoke test that proves the service is usable. Cleanup: delete the disposable directory and any temporary files. In a real environment, also remove test restore instances, revoke temporary access, and confirm that drill data is not connected to production clients.

Assessment Exercises

  1. An orders database has a five-minute RPO and a thirty-minute RTO. Which is more dangerous: daily snapshots with fast restore, or continuous replication with an untested promotion process? Explain using both RPO and RTO.
  2. A product catalog is served from two active regions. Users can edit the same product in either region. Name two conflict-resolution strategies and the user-visible problem each one can create.
  3. Your backup dashboard shows 100 percent job success for three months, but the first restore drill misses the RTO by six hours. What metrics should have existed before the drill?
  4. A replica is healthy but 20 minutes behind during a primary-region outage. Under what business conditions is promotion acceptable, and what must the incident commander communicate?
  5. Design a validation check for restoring a payment ledger. Why is a simple row count insufficient?

Summary

Backup and disaster recovery design starts with RPO and RTO, then selects the cheapest mechanism that meets them under realistic failure conditions. Snapshots, incrementals, transaction logs, replicas, standby regions, and active-active routing each solve a different part of the recovery problem. The design is only credible when restores are practiced, failover prevents split brain, security controls protect backup data and keys, and validation proves that recovered services are correct enough to accept traffic.