Disaster Recovery, Rollback, and Break-Glass Operations

Terraform disaster recovery is the ability to rebuild control of infrastructure after the normal delivery path, state backend, cloud account, or a managed resource has failed. Rollback is the narrower act of returning infrastructure to a previous known-good shape. Break-glass operations are emergency changes made outside the usual approval path when waiting would create more damage than acting.

The outcome for this lesson is practical: you should be able to decide whether to restore state, re-apply known configuration, import an object, revert a module change, or perform a controlled manual change and reconcile it afterward. In Terraform, recovery work is rarely a single undo button. It is a sequence of state custody, graph evaluation, provider API calls, and verification.

How Terraform Recovery Actually Works

Terraform does not store a complete backup of every remote system. State records Terraform resource addresses, provider-specific object identifiers, dependency metadata, selected attributes, and outputs. During planning, Terraform refreshes known objects, compares refreshed state with configuration, and builds an action graph. Recovery therefore depends on three separate things: available configuration, usable provider credentials, and trustworthy state history.

If configuration is lost but state remains, Terraform knows what exists but not what should exist next. If state is lost but configuration remains, Terraform knows desired declarations but not which remote objects they are bound to. If credentials are lost, Terraform can parse configuration but cannot refresh, import, plan accurately, or apply. A good disaster recovery design protects all three.

Rollback is also graph-based. Reverting a Git commit and running terraform apply asks Terraform to converge to older configuration; it does not reverse the previous provider calls transactionally. Some cloud changes are reversible, some create replacement objects, and some cannot recover lost data. For example, reducing an autoscaling maximum is reversible, replacing a database can be destructive, and deleting an unversioned bucket object is outside Terraform’s normal rollback ability.

Configuration Anatomy

Recovery-oriented Terraform configuration normally includes a remote backend, state locking, versioned source control, lifecycle rules, explicit imports or moved blocks during refactors, and outputs that help operators verify the restored system. The backend protects the state file, locking prevents competing writes, lifecycle settings prevent accidental destruction, and import or moved declarations preserve the relationship between resource addresses and real infrastructure.

terraform {
  backend "s3" {
    bucket         = "acme-terraform-state-prod"
    key            = "platform/network/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "acme-terraform-locks"
    encrypt        = true
  }
}

This backend fragment stores state remotely and uses a lock table so two operators do not write the same state at once. In a real recovery run, the expected behavior is that one active plan or apply owns the lock. A second apply should wait or fail with a lock message rather than silently racing.

Example 1: Restoring Control After State Loss

Suppose the state object for a small DNS workspace was deleted, but the hosted zone and records still exist. Re-running apply without state may make Terraform propose new records, duplicate names, or fail when the provider rejects conflicts. The safer path is to restore a backend version if available. If no state backup exists, create a new empty workspace, run terraform import for each surviving object, then plan until Terraform reports no unintended changes.

import {
  to = aws_route53_zone.primary
  id = "Z1234567890EXAMPLE"
}

resource "aws_route53_zone" "primary" {
  name = "example.com"
}

This example uses an import block to bind an existing hosted zone to the address aws_route53_zone.primary. The deterministic expected result after a successful import and matching configuration is a plan with no create action for the zone. If Terraform still proposes replacement, the cause is usually a mismatched argument, a provider-default difference, or importing the wrong remote identifier.

Example 2: Rolling Back a Risky Resource Change

A common rollback is reverting a module release that changed a replacement-sensitive argument. For databases, queues, and buckets, the rollback design must prevent accidental deletion before the incident occurs. Terraform’s prevent_destroy lifecycle setting is not a backup, but it is a useful guardrail because it forces an operator to make destruction explicit.

variable "restore_snapshot_id" {
  type        = string
  description = "Snapshot identifier to use only during an approved database restore."
  default     = null
}

resource "aws_db_instance" "orders" {
  identifier          = "orders-prod"
  engine              = "postgres"
  instance_class      = "db.m6g.large"
  allocated_storage   = 100
  snapshot_identifier = var.restore_snapshot_id

  lifecycle {
    prevent_destroy = true
  }
}

If a later change would destroy aws_db_instance.orders, the plan fails instead of applying. During a restore, setting restore_snapshot_id must be reviewed carefully because some database arguments force replacement. Expected behavior is not automatic in-place rollback; Terraform will describe whether the provider can update the object or must create a replacement.

Example 3: Break-Glass With Reconciliation

Break-glass work should be temporary, attributable, and reconciled back into Terraform. Imagine production traffic is down because a security group rule is too strict. An incident commander approves a short-lived manual rule in the cloud console. After service is restored, Terraform configuration must be updated or the next apply will remove the manual rule as drift.

variable "break_glass_cidr" {
  type        = string
  description = "Temporary approved CIDR for incident access; empty means closed."
  default     = ""
}

resource "aws_security_group_rule" "temporary_incident_access" {
  count             = var.break_glass_cidr == "" ? 0 : 1
  type              = "ingress"
  security_group_id = aws_security_group.app.id
  protocol          = "tcp"
  from_port         = 443
  to_port           = 443
  cidr_blocks       = [var.break_glass_cidr]
  description       = "Temporary incident access tracked in Terraform"
}

With the default value, Terraform plans zero temporary rules. With an approved CIDR, it plans one ingress rule. Cleanup is deterministic: set the variable back to an empty string and apply, and Terraform removes the tracked temporary rule. The trade-off is speed versus exposure; this pattern is safer than an untracked console change but still requires a time-bound approval and monitoring.

Design Choices and Trade-Offs

Restoring a backend state version is fastest when the state is known good, but it can discard legitimate changes made after the backup. Importing surviving resources is slower but useful when state history is unavailable. Reverting configuration is clean when the prior version still matches reality, but it is unsafe if the failed apply partially completed and changed only part of the graph.

Manual break-glass changes can restore service quickly, but they create drift. Terraform-managed emergency switches are more auditable, but they require prior design. For high-risk systems, prefer rehearsed recovery modules, protected state storage, separate read-only diagnostic credentials, and narrowly scoped emergency roles that expire automatically.

Failure Modes and Troubleshooting

Symptom: Terraform wants to recreate many existing resources after a backend incident. Cause: the workspace is using empty or wrong state. Diagnose: check backend key, workspace name, state object version, and resource addresses in terraform state list. Correct: restore the correct state version or import objects before applying.

Symptom: an urgent apply fails with a lock error. Cause: another run owns the remote state lock or a previous run exited without releasing it. Diagnose: identify the lock holder, pipeline run, and timestamp. Correct: let the active run finish, or use force unlock only after proving no apply is still running.

Symptom: reverting a commit still plans replacements. Cause: remote objects changed during the failed rollout, or provider defaults differ from the older configuration. Diagnose: inspect the saved failed plan, current plan, and provider replacement markers. Correct: adjust configuration to match the intended surviving objects, import or move addresses when needed, then apply a reviewed plan.

Reliability and Security Implications

State is sensitive because it may contain identifiers, generated passwords, endpoint names, and outputs. Encrypt it, restrict access, enable versioning, and audit reads as well as writes. Recovery credentials should be powerful enough to restore service but not broad enough to bypass every control. Break-glass roles should require human approval, produce durable audit records, and expire quickly.

Performance matters during recovery because refresh and planning can be slow in large workspaces. Smaller workspaces reduce blast radius and speed diagnosis, but too many workspaces make dependency recovery harder. Choose boundaries around ownership and failure domains, not merely around repository folders.

Hands-On Lab

Prerequisites: Terraform installed locally, a disposable cloud account or local provider sandbox, a remote backend with versioning enabled, and a Git repository containing a small workspace.

  1. Create a baseline resource and apply it through the normal pipeline. Save the plan artifact and record the state backend key.
  2. Enable state versioning or confirm it is already enabled. Run terraform state list and record the resource addresses.
  3. Make a harmless change, such as adding a tag, and apply it. Verify the provider shows the changed tag.
  4. Simulate rollback by reverting the configuration change in Git. Run a saved plan and confirm Terraform proposes only the tag reversal.
  5. Simulate drift by changing the same tag manually in the provider console. Run terraform plan and confirm Terraform detects the difference.
  6. Practice reconciliation by either accepting the manual value in configuration or applying Terraform to restore the declared value.
  7. Practice cleanup by removing any temporary break-glass variables, applying the closed configuration, and confirming no emergency rule remains.

Verification: the final plan should report no unexpected creates, replacements, or destroys. The backend should retain earlier state versions, and audit logs should show who performed each recovery action.

Cleanup: destroy only disposable lab resources after removing prevent_destroy guards intentionally. Do not delete backend history until the lab review is complete.

Assessment Exercises

  1. A state file is restored from yesterday, but two legitimate resources were added today. What plan symptoms do you expect, and how would you avoid deleting the new resources?
  2. Why is reverting a Terraform module version not equivalent to database point-in-time recovery?
  3. Design a break-glass variable for temporary administrator access. What value closes access, and what evidence proves it was removed?
  4. A plan shows replacement after an import. List three likely causes and the order in which you would investigate them.
  5. Where would you draw workspace boundaries for an application with shared networking, databases, and stateless services, and how does that affect recovery time?

Summary

Terraform recovery depends on configuration, credentials, and state agreeing about real infrastructure. Protect remote state, rehearse imports and restores, treat rollback as a new reviewed convergence plan, and make break-glass access temporary and reconciled. The strongest recovery process is the one practiced before the outage, with clear verification and a cleanup path.