Incident Response, Runbooks, Resilience Testing, and DR

AWS incident response is the practiced ability to detect a workload failure, classify impact, run the right recovery procedure, and preserve evidence while customers are still depending on the system. Runbooks make that response repeatable. Resilience testing proves the runbooks and architecture before a real outage does. Disaster recovery, or DR, covers the larger case: restoring service when an Availability Zone, Region, account, critical data store, or deployment pipeline is unavailable.

The outcome for this lesson is concrete: you should be able to design an AWS incident path from signal to action, write a runbook that an operator can execute under pressure, run a controlled resilience experiment, and define recovery evidence for RTO and RPO. In the AWS Cloud Engineering course this sits after deployment and operations topics because response quality depends on prior choices: tagging, IAM, alarms, logging, backup policy, infrastructure as code, and network isolation all decide how recoverable the workload really is.

How AWS Incident Response Works

AWS incident response usually starts with telemetry. CloudWatch metrics, logs, synthetic canaries, load balancer target health, Route 53 health checks, GuardDuty findings, Security Hub findings, CloudTrail events, and application traces become signals. A useful signal has an owner, severity, customer impact statement, and response path. An alarm without those fields may be noisy information; it is not yet an incident mechanism.

The internal flow is event driven. A CloudWatch alarm changes state when datapoints breach its evaluation rule. That state change can notify Amazon SNS, create an EventBridge event, trigger Systems Manager Automation, open an incident in AWS Systems Manager Incident Manager, or start a Lambda function. CloudTrail records control-plane API calls that are often needed during diagnosis, such as who changed a security group, deployed a new Lambda alias, disabled a KMS key, or modified an Auto Scaling policy. Logs and metrics provide workload behavior; CloudTrail provides AWS API history.

A runbook is the executable operating procedure for a known condition. A strong runbook has entry criteria, permissions, inputs, decision points, commands, expected output, escalation triggers, rollback steps, and evidence to attach to the incident record. AWS Systems Manager Automation can turn parts of a runbook into controlled steps with typed parameters, IAM service roles, approval gates, and bounded execution. Manual runbooks are still useful, but they should avoid hidden knowledge such as “check the usual dashboard” or “restart the service” without naming the service and verification command.

Resilience testing deliberately injects or simulates failure to validate design assumptions. AWS Fault Injection Service can stop instances, disrupt network connectivity for selected resources, pause I/O on supported storage targets, or stress CPU and memory through SSM documents. Experiments should be scoped by resource tags, protected by stop conditions, and scheduled when responders are available. The goal is not drama; the goal is evidence that the system degrades as expected and that the runbook restores the stated objective.

DR extends incident response across a larger blast radius. The two key measures are RTO, the maximum tolerable time to restore service, and RPO, the maximum tolerable amount of data loss measured as time. A nightly backup might support an RPO near one day, not five minutes. Cross-Region replication may reduce RPO, but it adds cost, failover complexity, and the risk of replicating bad data. DR designs commonly fall into backup and restore, pilot light, warm standby, and multi-site active-active patterns. Each pattern trades cost and operational complexity against recovery speed.

Configuration Anatomy

The main building blocks have specific responsibilities. CloudWatch alarms define metric, threshold, evaluation period, datapoints to alarm, missing data behavior, and action targets. EventBridge rules match event source and detail fields, then route to targets. SNS topics fan out notifications. Incident Manager response plans define contacts, chat channels, engagements, deduplication, and runbook associations. SSM Automation documents define parameters and steps such as aws:executeAwsApi, aws:runCommand, aws:branch, and aws:approve. AWS Backup plans define resource selection, lifecycle, vault, copy rules, and retention. Route 53 failover records use health checks to select primary or secondary endpoints.

These parts must share naming and tagging conventions. A response plan cannot select the right automation if alarms use inconsistent service names. A resilience experiment cannot be safely targeted if production and test resources share vague tags. A DR drill cannot prove recovery if backup vaults, KMS keys, and IAM roles are not available in the recovery Region.

Example 1: Classifying an Alarm Event

The first example treats a CloudWatch alarm state change as the incident entry point. The event tells the responder which alarm changed, the old and new states, the Region, account, and reason. A runbook should map this alarm to a severity and first diagnostic action.

{
  "source": "aws.cloudwatch",
  "detail-type": "CloudWatch Alarm State Change",
  "region": "us-east-1",
  "detail": {
    "alarmName": "payments-api-5xx-rate-high",
    "state": {
      "value": "ALARM",
      "reason": "Threshold Crossed: 3 datapoints were greater than the threshold"
    },
    "previousState": {
      "value": "OK"
    }
  }
}

Expected behavior is deterministic at the routing layer: an EventBridge pattern matching source, detail-type, and detail.alarmName routes this event to the chosen target. The operational interpretation is that the payment API is returning too many server errors. The runbook should start with customer impact, recent deployments, target group health, and application error logs rather than generic infrastructure checks.

Example 2: A Read-Only Triage Command Set

The next example is a safe first page for an operator. It avoids mutation and gathers identity, alarm state, recent deployments from CloudTrail, and unhealthy load balancer targets. Replace names with your workload values.

set -euo pipefail

aws sts get-caller-identity --output json

aws cloudwatch describe-alarms \
  --alarm-names payments-api-5xx-rate-high \
  --output table

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=UpdateFunctionCode \
  --max-results 10 \
  --output table

aws elbv2 describe-target-health \
  --target-group-arn arn:aws:elasticloadbalancing:us-east-1:111122223333:targetgroup/payments-api/abc123 \
  --output table

The expected output is not a fixed string because it depends on the account. The expected behavior is that every command is read-only. If the caller lacks permission, the failure should be AccessDenied, which is useful evidence that the incident role is incomplete. If target health shows many unhealthy targets immediately after a deployment event, the runbook can branch to rollback or traffic shifting.

Example 3: Resilience Experiment With Stop Conditions

A resilience test should constrain both target selection and abort criteria. This example shows the shape of an AWS Fault Injection Service experiment template that stops selected instances by tag and aborts if the service alarm enters ALARM. The stop condition is the safety control: it connects the experiment to the same customer-facing signal used in operations.

{
  "description": "Stop one tagged payments instance and verify Auto Scaling replacement",
  "roleArn": "arn:aws:iam::111122223333:role/fis-payments-experiment-role",
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:us-east-1:111122223333:alarm:payments-api-5xx-rate-high"
    }
  ],
  "targets": {
    "paymentsInstances": {
      "resourceType": "aws:ec2:instance",
      "selectionMode": "COUNT(1)",
      "resourceTags": {
        "Service": "payments-api",
        "Environment": "staging"
      }
    }
  },
  "actions": {
    "stopInstance": {
      "actionId": "aws:ec2:stop-instances",
      "parameters": {
        "startInstancesAfterDuration": "PT10M"
      },
      "targets": {
        "Instances": "paymentsInstances"
      }
    }
  }
}

The expected behavior is that exactly one tagged staging instance is stopped, then restarted after the duration unless the alarm stop condition aborts the experiment. A successful test does not mean “nothing happened”; it means Auto Scaling, load balancing, and application retries kept the user-visible alarm within bounds while replacement occurred.

Example 4: DR Evidence Script

The final example measures recovery evidence. It records the time, checks whether a recovery endpoint answers, and asks AWS Backup for recent recovery points for a tagged resource. It belongs in a DR drill after failover, not only during a real disaster.

set -euo pipefail

RECOVERY_URL="https://recovery.example.com/health"
BACKUP_VAULT="payments-prod-vault"

printf 'dr_check_started=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
curl --fail --silent --show-error "$RECOVERY_URL"
printf '\n'

aws backup list-recovery-points-by-backup-vault \
  --backup-vault-name "$BACKUP_VAULT" \
  --max-results 5 \
  --output table

The deterministic expectation is that the script exits nonzero if the health endpoint fails or the AWS CLI command fails. In a drill, attach start time, health response, selected recovery point creation time, DNS change time, and customer validation to the incident record. That evidence is what proves or disproves the promised RTO and RPO.

Design Choices and Trade-Offs

Manual runbooks are flexible and cheap to start, but they vary by operator skill and are hard to audit. SSM Automation improves repeatability and evidence, but it requires careful IAM roles, parameter validation, and testing of failure branches. Fully automatic remediation can reduce recovery time for known faults, such as replacing unhealthy instances, but it can amplify damage if the diagnosis is wrong. Use automatic remediation for narrow, reversible actions with strong stop conditions.

For resilience testing, staging experiments are safer but may not reflect production scale, traffic, quotas, or data shape. Production experiments produce better evidence, but they require blast-radius limits, communication, and fast rollback. A good compromise is progressive scope: one task, one instance, one AZ dependency, then a larger service-level experiment only after smaller tests pass.

For DR, backup and restore is usually lowest cost and slowest to recover. Pilot light keeps minimal core infrastructure ready in another Region. Warm standby runs a reduced copy and scales during recovery. Active-active can offer the shortest recovery but requires global routing, data conflict handling, deployment discipline, and higher steady-state cost. Pick the pattern from business impact and data requirements, not from architectural preference.

Failure Modes and Troubleshooting

Symptom: the alarm fires, but no responder is paged. Cause: the alarm action points to the wrong SNS topic, EventBridge rule pattern does not match the state-change event, or Incident Manager contacts are outside the engagement schedule. Diagnose: inspect alarm actions, EventBridge matched event metrics, SNS delivery status, and Incident Manager timeline. Correct: repair the target ARN or event pattern, send a test event, and add a scheduled notification test.

Symptom: the runbook starts but fails halfway with AccessDenied. Cause: the incident role can read alarms but cannot describe target health, start automation, decrypt logs, or assume the remediation role. Diagnose: capture the denied API action and resource from CloudTrail or the CLI error. Correct: add the narrow missing permission and rerun the runbook in a nonproduction account before relying on it.

Symptom: a resilience experiment affects the wrong resources. Cause: tag selectors are too broad or production and staging share tags. Diagnose: preview selected resources where the service supports it and compare tags with the inventory. Correct: require environment, service, owner, and experiment-allowed tags, then deny experiments without those tags through IAM conditions.

Symptom: DR failover succeeds technically but users still cannot complete transactions. Cause: DNS, secrets, KMS keys, database replicas, third-party allowlists, or asynchronous queues were not included in the recovery design. Diagnose: follow a real user transaction through the recovery Region and check each dependency. Correct: add the missing dependency to infrastructure as code and the DR drill checklist.

Security, Reliability, and Performance Implications

Incident response roles are powerful because they operate during stress. Use separate break-glass and automation roles, require MFA or approval for destructive actions, log every role assumption, and keep session duration short. Do not paste secrets into incident notes or chat. Prefer links to encrypted logs and bounded identifiers such as request IDs, alarm ARNs, and deployment IDs.

Reliability improves when alarms track user outcomes, not just resource symptoms. CPU can be high during healthy batch work; checkout failure rate is closer to customer impact. Performance matters during incidents because diagnostic commands can worsen overload. Avoid broad log scans and fleet-wide commands during peak failure. Query narrow time windows, filter by service and request ID, and use sampling where appropriate.

Hands-On Lab: Build a Small Response Path

Prerequisites: an AWS sandbox account, AWS CLI configured for a least-privilege lab role, one nonproduction workload or test EC2 Auto Scaling group, CloudWatch access, and permission to create an SNS topic, alarm, and optional FIS experiment in the sandbox.

  1. Create or identify a test metric that can safely alarm, such as a custom metric named LabErrorRate in a sandbox namespace.
  2. Create a CloudWatch alarm for that metric with a short evaluation window and an SNS topic action. Subscribe an email address or test webhook you control.
  3. Write a runbook page with entry criteria, read-only diagnostic commands, decision branches, rollback action, escalation contact, and evidence fields.
  4. Publish datapoints that move the alarm to ALARM. Verify that notification arrives and that the alarm state-change event is visible in EventBridge or CloudWatch alarm history.
  5. Execute the read-only triage commands from the runbook. Record command time, caller identity, alarm state, and one workload-specific finding.
  6. If your sandbox supports it, create a tightly scoped FIS experiment against one tagged test instance with a CloudWatch alarm stop condition. Run it only after confirming the selected target.
  7. Verify recovery by checking alarm return to OK, instance or service health, and application health endpoint behavior.
  8. Cleanup by deleting the lab alarm, SNS subscription, SNS topic, custom metric producer, and FIS template if created. Roll back any temporary IAM permissions.

Lab verification is complete when you can show the alarm transition, notification, runbook evidence, recovery signal, and cleanup record. If any step required console guessing, revise the runbook until the next operator can repeat it from written instructions.

Assessment Exercises

  1. A payment API has a four-hour database snapshot schedule and no replication. The business asks for fifteen-minute RPO. Explain why the current design cannot meet that target and name two AWS mechanisms that could reduce data loss.
  2. Your runbook says “restart the service if latency is high.” Rewrite that instruction so it includes entry criteria, exact AWS target, safety check, command, and verification.
  3. An FIS experiment in staging caused no customer impact. Give three reasons that result might still fail to predict production behavior.
  4. A failover drill restored the application in 35 minutes, but DNS caching kept some clients on the failed endpoint for two hours. Which recovery measure was missed, and how would you change the next drill?
  5. Design an IAM boundary for an incident automation role that can diagnose a load-balanced EC2 service but cannot delete data or alter unrelated services.

Summary

Incident response on AWS is a chain: precise alarms, routed events, prepared responders, executable runbooks, controlled automation, resilience experiments, and DR evidence. The technical details matter because each AWS service contributes a different part of the chain. CloudWatch detects, EventBridge routes, SNS and Incident Manager engage, SSM Automation executes, CloudTrail explains control-plane change, FIS tests assumptions, and backup or replication mechanisms determine recovery limits. Treat every incident procedure as production code: scope permissions, test failure branches, measure RTO and RPO with evidence, and clean up temporary access after the work is done.