Resilient Multi-AZ Web Architecture

A resilient Multi-AZ web architecture keeps a web application reachable when an instance, subnet, or Availability Zone stops serving traffic. On AWS, the common pattern is an internet-facing Application Load Balancer in public subnets, an Auto Scaling group in private subnets across at least two Availability Zones, and shared state stored outside the web instances. The outcome is practical: clients connect to one DNS name, the load balancer sends requests only to healthy targets, and Auto Scaling replaces failed capacity.

This lesson belongs in the compute and scaling section because resilience is not a property of EC2 alone. It comes from how EC2 instances, target groups, load balancer nodes, VPC routing, launch templates, health checks, and scaling policy timing work together.

Purpose and Outcome

The design protects against two common failures. First, an individual instance can fail because the process crashes, a deployment breaks startup, a disk fills, or the host becomes unhealthy. Second, one Availability Zone can become unusable for your workload because of network, power, dependency, or control-plane impairment. A Multi-AZ web tier spreads interchangeable capacity across independent facilities so one failure domain is not the entire service.

The target behavior is reduced capacity rather than outage. When one target fails, the Application Load Balancer stops sending it new requests after the unhealthy threshold is met. When Auto Scaling sees unhealthy capacity, it launches replacement instances from the launch template. When one zone is impaired, healthy targets in the remaining zone or zones continue serving traffic, assuming they were sized to absorb the shifted load.

How the Mechanism Works

An Application Load Balancer is regional, but it creates load balancer nodes in each enabled Availability Zone. Each enabled zone needs a subnet. For an internet-facing ALB, those subnets are public and route to an internet gateway. The ALB terminates or passes client connections at a listener, evaluates listener rules, then forwards matching HTTP or HTTPS requests to a target group.

The target group is the routing and health boundary. It stores registered targets, the backend protocol and port, deregistration delay, load balancing attributes, and health check settings. The health check is separate from normal user paths. A site can serve /checkout while the target group probes /health. A target becomes healthy only after enough successful checks and unhealthy only after enough failed checks, so one slow response does not immediately change routing.

Deregistration delay is part of resilience, not just deployment polish. When an instance is removed, the ALB stops assigning new requests but allows existing requests to finish until the delay expires. Short delays speed replacement but can interrupt long responses. Long delays protect in-flight work but keep draining targets around longer, which can matter during urgent rollback or scale-in.

The Auto Scaling group owns the EC2 fleet. Its launch template defines the AMI, instance type, security groups, IAM instance profile, user data, block devices, and instance metadata settings. Its subnet list decides where instances may launch. Attaching the group to the target group registers new instances automatically. Setting the group health check type to ELB lets load balancer health influence replacement, which is important when EC2 is running but the application is not serving.

The application must be stateless at the web tier. Session state belongs in a shared store such as ElastiCache or DynamoDB, uploaded files belong in S3, and relational state should use a database pattern such as Amazon RDS Multi-AZ. Sticky sessions can reduce visible disruption, but they do not fix state stored on one instance.

Configuration Anatomy

A typical VPC has one public and one private subnet in each selected Availability Zone. Public subnets host the ALB and route 0.0.0.0/0 to an internet gateway. Private subnets host web instances. If instances need package repositories or external APIs, private subnets need outbound access through NAT gateways or VPC endpoints. For stronger AZ independence, route each private subnet to a NAT gateway in the same zone.

Security groups should mirror the request path. The ALB security group allows inbound HTTPS from clients and outbound traffic to the application port. The instance security group allows inbound application traffic only from the ALB security group. The instance role grants narrow permissions for logs, parameters, secrets, or application APIs. Production listeners normally use HTTPS with an ACM certificate, while the target group can use HTTP on a private port when traffic remains inside the VPC and the risk model allows it.

Example 1: Target Group and Auto Scaling Wiring

This fragment shows the key relationship. The target group defines the backend port and readiness check. The Auto Scaling group spans two private subnets, keeps two instances running, uses ELB health, and attaches instances to the target group. With desired capacity two, normal placement should put one target in each zone when capacity is available.

Resources:
  WebTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Protocol: HTTP
      Port: 8080
      VpcId: !Ref VpcId
      HealthCheckPath: /health
      HealthCheckIntervalSeconds: 15
      HealthCheckTimeoutSeconds: 5
      HealthyThresholdCount: 2
      UnhealthyThresholdCount: 2
      Matcher:
        HttpCode: "200"
  WebAutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      MinSize: "2"
      MaxSize: "6"
      DesiredCapacity: "2"
      VPCZoneIdentifier:
        - !Ref PrivateSubnetAzA
        - !Ref PrivateSubnetAzB
      HealthCheckType: ELB
      HealthCheckGracePeriod: 120
      TargetGroupARNs:
        - !Ref WebTargetGroup
      LaunchTemplate:
        LaunchTemplateId: !Ref WebLaunchTemplate
        Version: !GetAtt WebLaunchTemplate.LatestVersionNumber

Expected behavior is deterministic at the control layer: instances launched by the group are registered in the target group, but they receive traffic only after the target group reports them healthy. If the application takes 90 seconds to boot, the grace period prevents immediate replacement while startup completes.

Example 2: A Readiness Endpoint for New Instances

The load balancer should check readiness, not just whether Linux booted. This simple user data installs a web process that returns 200 on /health and version text on /. In a real service, the health route should confirm required local configuration is loaded and the app can accept requests, while deeper dependency checks are usually reported to telemetry rather than used to remove every target during a shared database incident.

#!/usr/bin/env bash
set -euo pipefail

yum install -y python3
cat >/opt/web.py <<'APP'
from http.server import BaseHTTPRequestHandler, HTTPServer

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == '/health':
            self.send_response(200)
            self.end_headers()
            self.wfile.write(b'OK')
        else:
            self.send_response(200)
            self.end_headers()
            self.wfile.write(b'web-version=blue')

HTTPServer(('0.0.0.0', 8080), Handler).serve_forever()
APP
nohup python3 /opt/web.py >/var/log/web.log 2>&1 &

Expected output from the ALB root path is a successful response containing web-version=blue. If /health returns anything outside the matcher, the target remains out of service even if the instance is running.

Example 3: Verifying a One-Target Failure

During failure testing, inspect both target health and Auto Scaling state. The commands below show whether the ALB has healthy targets and whether the group is replacing capacity. They are deliberately read-only so they can be used during an incident without changing the fleet.

set -euo pipefail

REGION="us-east-1"
TARGET_GROUP_ARN="arn:aws:elasticloadbalancing:REGION:ACCOUNT:targetgroup/example/1234567890abcdef"
AUTO_SCALING_GROUP_NAME="example-web-asg"

aws elbv2 describe-target-health \
  --region "$REGION" \
  --target-group-arn "$TARGET_GROUP_ARN" \
  --query 'TargetHealthDescriptions[*].{Target:Target.Id,Port:Target.Port,State:TargetHealth.State,Reason:TargetHealth.Reason}' \
  --output table

aws autoscaling describe-auto-scaling-groups \
  --region "$REGION" \
  --auto-scaling-group-names "$AUTO_SCALING_GROUP_NAME" \
  --query 'AutoScalingGroups[0].{Desired:DesiredCapacity,Min:MinSize,Max:MaxSize,Instances:Instances[*].LifecycleState}' \
  --output json

Before failure injection, expect at least two healthy targets across different zones. After stopping the web process on one instance, expect that target to move to unhealthy after two failed checks, while the other target stays healthy. If ELB health replacement is enabled, Auto Scaling should eventually terminate or replace the unhealthy instance according to its health evaluation.

Design Choices and Trade-offs

Two Availability Zones are the minimum. Three zones improve maintenance flexibility and make one-zone loss easier to absorb, but they can increase baseline instance count, NAT gateway cost, database replica planning, and test paths. For small internal systems, two zones with tested recovery may be enough. For customer-facing workloads with stricter availability goals, three zones often gives a better capacity margin.

Health check depth is a trade-off. A shallow health check avoids removing all targets during a temporary database problem, but it may send users to an instance that cannot serve real requests. A deep health check catches more broken instances, but it can turn a downstream outage into an empty target group. A common compromise is a readiness endpoint for load balancer admission, separate metrics for dependencies, and alarms on elevated errors or latency.

Capacity policy must account for failover, not only normal traffic. If four instances are required at peak and you run two total, losing one zone leaves one or two overloaded targets. Request count per target, latency, queue depth, or custom application metrics may describe web pressure better than average CPU. Warm-up and cooldown values should match boot time, configuration fetch time, and health threshold timing.

Failure Modes and Troubleshooting

Symptom: the ALB returns 503 Service Unavailable. Cause: the target group has no healthy targets. Diagnostic steps: check target health reason codes, the application port, instance security group ingress from the ALB security group, the health path, and application startup logs. Correction: fix the path or port, restore allowed ALB-to-instance traffic, repair startup, then wait for the healthy threshold.

Symptom: users lose login state after replacement. Cause: session data is stored in instance memory or local disk. Diagnostic steps: compare requests before and after target changes, inspect session cookies, and check whether stickiness is hiding a stateful web tier. Correction: move sessions to a shared store, use signed client-side tokens where appropriate, and make every instance able to handle the next request.

Symptom: failover works but latency spikes during one-zone loss. Cause: remaining instances, NAT, cache, or database capacity cannot absorb the shifted load. Diagnostic steps: compare request count per target, target response time, CPU, memory, connection counts, database connections, and downstream throttling. Correction: raise baseline capacity, use three zones, tune scaling policies, or increase downstream capacity.

Symptom: new private-subnet instances never become healthy. Cause: bootstrapping cannot fetch packages, secrets, images, or configuration. Diagnostic steps: inspect user data logs, route tables, NAT gateway placement, VPC endpoints, IAM permissions, DNS, and egress rules. Correction: add required endpoints or NAT paths, narrow but complete IAM permissions, and fail clearly when configuration is missing.

Security, Reliability, and Performance Implications

Instances should not be directly reachable from the internet. The ALB is the controlled entry point, with TLS termination, listener rules, WAF integration when needed, and logging. Treat forwarded headers such as X-Forwarded-For as information added at the load balancer boundary, not proof of identity from the client.

Reliability depends on independent failure domains. Spreading instances across subnets is insufficient if every instance depends on one NAT gateway, one self-managed database, one file server, or one deployment job that can remove all capacity. Performance depends on warm capacity because replacement instances take time to launch, configure, register, pass health checks, and receive traffic.

Hands-on Lab

Prerequisites: an AWS account with permission to inspect and manage EC2, Elastic Load Balancing, Auto Scaling, VPC, IAM, and CloudWatch; AWS CLI configured; a sandbox VPC or permission to create temporary resources; and a non-production workload.

  1. Select one Region and two Availability Zones. Create or identify one public and one private subnet in each zone.
  2. Create an internet-facing Application Load Balancer in the public subnets and a target group with HTTP health checks on /health.
  3. Create a launch template for a simple web server like the example above, using an instance security group that accepts the application port only from the ALB security group.
  4. Create an Auto Scaling group across the private subnets with desired capacity two, attach the target group, and enable ELB health checks.
  5. Verify that both targets become healthy and repeated requests to the ALB DNS name return successful responses.
  6. Stop the web process on one instance or temporarily block the application port from the ALB security group.
  7. Watch the target become unhealthy, confirm the ALB still serves through the remaining target, and confirm replacement behavior if enabled.
  8. Restore the process or security rule, return desired capacity to the original value, and verify all targets are healthy.

Verification: before failure injection, target health should show healthy targets in different Availability Zones. During one instance failure, at least one target should remain healthy and the ALB should return application responses. After recovery, desired capacity should match the Auto Scaling group setting and new instances should pass health checks.

Cleanup: set desired, minimum, and maximum capacity to zero or delete the Auto Scaling group first. Then delete the launch template, load balancer, target group, NAT gateways, temporary subnets, route tables, security groups, and VPC resources created only for the lab. Keep logs long enough to review the timeline, then apply your normal retention policy.

Assessment Exercises

  1. Your application runs two instances across two zones, but one-zone loss makes the site too slow. Which metrics identify the bottleneck, and what capacity change would you test first?
  2. A target passes /health, but checkout requests fail with database connection errors. How would you change readiness checks and telemetry without removing every target during a brief database issue?
  3. An operator enables sticky sessions to avoid login loss. What design problem does that hide, and what is the more resilient fix?
  4. A deployment launches instances that never become healthy in private subnets. List the network, IAM, and application startup checks you would perform in order.
  5. For a low-traffic internal tool, compare two-zone and three-zone designs. Which reliability benefit must justify the extra baseline cost?

Summary

A resilient Multi-AZ web architecture uses AWS failure domains deliberately. The ALB accepts traffic across enabled zones, target group health checks decide which instances receive requests, and the Auto Scaling group replaces capacity from a repeatable launch template. The essential application property is interchangeability: any healthy instance can serve the next request. Strong designs size for zone loss, keep instances private, externalize state, test health behavior, and rehearse recovery before a real impairment.