Well-Architected Review and Production Readiness

An AWS Well-Architected review is a structured way to decide whether a workload is ready to run in production. The outcome is not a badge or a meeting note. The useful outcome is a prioritized list of risks, the evidence behind each risk, and a launch decision that says what must be fixed now, what can be accepted temporarily, and who owns the follow-up.

In this capstone lesson, the review connects every earlier AWS Cloud topic: accounts, IAM, VPC design, compute, data stores, observability, automation, resilience, and cost control. Production readiness means the workload has been examined against real operating conditions, not just deployed successfully once.

How the Review Works

The AWS Well-Architected Framework organizes review questions into six pillars: operational excellence, security, reliability, performance efficiency, cost optimization, and sustainability. A workload is the system being reviewed, such as a customer API, analytics pipeline, internal admin application, or event-driven order processor. A lens is an additional question set for a domain or technology area. The Well-Architected Tool records workloads, answers, notes, risks, milestones, and improvement plans.

Internally, the review is a comparison between intended workload behavior and AWS best practices. Each question represents a design pressure. For example, reliability questions ask whether failures are isolated, whether recovery objectives are defined, and whether changes can be rolled back. Security questions ask how identities are granted access, how data is protected, and whether incidents can be investigated. The tool classifies gaps as high risk, medium risk, or no improvement required. A high risk issue is not automatically a launch blocker, but it must be explicitly accepted or remediated because it can materially affect security, availability, recovery, or cost.

A milestone is a saved point in time. Teams use milestones before launch, after major architecture changes, after incident-driven fixes, and during periodic reviews. This matters because readiness is not permanent. A system that was acceptable before traffic growth, a new region, or a database migration may no longer satisfy its original objectives.

Configuration Anatomy

A review needs precise inputs. The workload name identifies the system. The environment tells whether the review is for production or pre-production. Regions identify where the workload runs. Account IDs identify ownership and blast radius. Pillars define the scope. Review notes should cite evidence, such as CloudWatch alarms, backup policies, IAM access paths, deployment runbooks, recovery test results, and cost reports.

A useful readiness record separates objective facts from judgments. Facts include RTO, RPO, traffic estimates, dependency lists, alert names, backup retention, and rollback commands. Judgments include accepted risks and target remediation dates. Keeping those separate prevents the review from becoming vague consensus.

{
  "workload": "checkout-api",
  "environment": "production",
  "regions": ["us-east-1"],
  "objectives": {
    "rto_minutes": 30,
    "rpo_minutes": 5,
    "monthly_error_budget_percent": 0.1
  },
  "evidence": {
    "deployment": "blue-green with automatic rollback",
    "database_backup": "automated snapshots plus tested point-in-time restore",
    "alarm_prefix": "checkout-prod-"
  }
}

This JSON is not an AWS API request. It is a compact readiness artifact. The expected behavior is that a reviewer can trace every later answer back to these facts. If the team cannot point to where the RTO is tested, the reliability answer should not be marked complete.

Example 1: Static Site

A marketing site on Amazon S3 behind CloudFront has a simple architecture, but it still needs review. Security focuses on blocking public S3 bucket access and serving users through CloudFront. Reliability focuses on object versioning, deployment rollback, and DNS ownership. Cost focuses on cache behavior, transfer volume, and logging retention.

The review might find no high risk issues but one medium risk issue: access logs are retained indefinitely without lifecycle rules. The expected output is a cost remediation item, not a redesign. A good improvement item says: add an S3 lifecycle policy that transitions or expires logs after the required audit period, verify with the bucket lifecycle configuration, and record the policy name in the review notes.

Example 2: API with RDS

A production API using Elastic Load Balancing, Auto Scaling, and Amazon RDS has more failure paths. The review should ask whether application instances are spread across Availability Zones, whether health checks reflect real application health, whether database backups meet RPO, and whether schema migrations can be rolled back or safely rolled forward.

Suppose the API runs in two Availability Zones but the RDS instance is single-AZ. Symptoms during an AZ impairment would include healthy application instances that cannot reach the database, elevated 5xx responses, and failed connection attempts. The cause is a database availability design that does not match the application tier. Diagnostics include checking the RDS Multi-AZ setting, reviewing recent failover events, and testing restore time from a snapshot. The correction could be enabling Multi-AZ deployment, documenting failover behavior, and running a controlled failover test before launch. The expected review classification is high risk until the recovery objective is proven.

Example 3: Event-Driven Worker

An order workflow using Amazon SQS, Lambda, DynamoDB, and EventBridge needs a different review. Reliability depends on idempotent processing, dead-letter queues, retry settings, and poison-message handling. Performance depends on reserved concurrency, batch size, and downstream capacity. Cost depends on event volume, retry storms, and hot partitions.

A realistic finding is that the worker retries failed payments without an idempotency key. Symptoms include duplicate payment attempts, repeated messages in the queue, and customer support reports of multiple charges. The cause is treating at-least-once delivery as exactly-once delivery. Diagnostics include inspecting SQS receive counts, Lambda logs for repeated order IDs, and DynamoDB conditional write failures. The correction is to store an idempotency record keyed by order ID and payment attempt, use conditional writes, and route exhausted messages to a dead-letter queue. The expected behavior after remediation is that duplicate deliveries return the stored result instead of charging again.

A small local model makes the idempotency requirement concrete. The first delivery for an order creates the charge record. A duplicate delivery with the same order ID and attempt number is ignored, while a different order still proceeds.

processed = {}
messages = [
    {"order_id": "A100", "attempt": 1, "amount": 42},
    {"order_id": "A100", "attempt": 1, "amount": 42},
    {"order_id": "B200", "attempt": 1, "amount": 19}
]

for message in messages:
    key = f"{message['order_id']}:{message['attempt']}"
    if key in processed:
        print(f"duplicate ignored for {key}: {processed[key]}")
        continue
    processed[key] = "charged"
    print(f"charge submitted for {key}")

print(f"unique_charges={len(processed)}")

The deterministic output is charge submitted for A100:1, duplicate ignored for A100:1: charged, charge submitted for B200:1, and unique_charges=2. In AWS, the in-memory dictionary would be replaced by a DynamoDB item written with a condition such as attribute-not-exists on the idempotency key.

Scoring Readiness Locally

The following small program shows the mechanics behind a launch gate. It does not replace the AWS review, but it models the decision: unresolved high risks block launch, accepted high risks require an owner and expiration date, and medium risks become tracked remediation work.

import json

review = {
    "workload": "checkout-api",
    "risks": [
        {"id": "REL-001", "level": "HIGH", "status": "resolved"},
        {"id": "SEC-004", "level": "HIGH", "status": "accepted", "owner": "security", "expires": "2026-10-01"},
        {"id": "COST-002", "level": "MEDIUM", "status": "open"}
    ]
}

blocking = [r for r in review["risks"] if r["level"] == "HIGH" and r["status"] == "open"]
accepted = [r for r in review["risks"] if r["level"] == "HIGH" and r["status"] == "accepted"]

if blocking:
    decision = "NO-GO"
elif accepted:
    decision = "GO WITH EXPLICIT RISK ACCEPTANCE"
else:
    decision = "GO"

print(f"workload={review['workload']}")
print(f"decision={decision}")
print(f"medium_risks={sum(1 for r in review['risks'] if r['level'] == 'MEDIUM')}")

The deterministic output is workload=checkout-api, decision=GO WITH EXPLICIT RISK ACCEPTANCE, and medium_risks=1. The important lesson is that risk status changes the launch decision, while the finding remains visible for later audit.

Design Choices and Trade-Offs

The first trade-off is scope. Reviewing every workload in exhaustive detail slows delivery, but reviewing only the visible application tier misses shared dependencies such as DNS, identity, data replication, and third-party integrations. A practical review scope includes the user-facing path, control-plane operations, data stores, deployment process, and recovery process.

The second trade-off is remediation timing. Some high risks should block launch, such as no restore test for a critical database, public write access to storage, or no way to revoke compromised credentials. Other risks may be accepted briefly when compensating controls exist. Acceptance should name an owner, deadline, and monitoring signal.

The third trade-off is automation. Infrastructure as code, policy checks, and deployment pipelines make evidence repeatable, but they do not remove judgment. A template can prove that encryption is enabled; it cannot prove that the recovery objective is realistic unless a restore test has been timed.

Failure Modes and Troubleshooting

  • Symptom: the review shows few risks, but incidents keep recurring. Cause: answers were based on intent instead of evidence. Diagnose: sample answers and ask for the exact alarm, test result, policy, or runbook. Correct: reopen unsupported answers and attach evidence.
  • Symptom: production launch is delayed by late security findings. Cause: IAM paths, encryption, logging, and incident response were reviewed after build completion. Diagnose: compare review date with architecture decision dates. Correct: run a lightweight early review at design time and a full review before launch.
  • Symptom: recovery objectives are documented but not met during a test. Cause: backups exist, but restore steps, permissions, or data dependencies were incomplete. Diagnose: time each restore step and identify manual approvals. Correct: automate restore, pre-create required roles, and lower the documented objective if the business accepts it.
  • Symptom: costs spike after launch despite a completed review. Cause: test traffic did not represent retry behavior, log volume, or data transfer. Diagnose: inspect CloudWatch metrics, Cost Explorer dimensions, and retry counts. Correct: set budgets, tune retention, and load test realistic failure cases.

Security, Reliability, and Cost Implications

A Well-Architected review improves security when it forces least privilege, centralized logging, encryption, secret rotation, and incident investigation into the launch criteria. It improves reliability when the team proves health checks, backup restoration, failover, throttling behavior, and rollback. It improves cost control when resources are tied to demand, retention is deliberate, and expensive failure loops are visible.

The review can also create false confidence. A stale review, a review without evidence, or a review that excludes shared services can be worse than no review because stakeholders believe production risk has been handled. Treat the review record as living architecture documentation.

Hands-On Lab

Prerequisites: an AWS account with permission to use the AWS Well-Architected Tool, one non-production workload to review, access to CloudWatch alarms or equivalent operational evidence, and a written RTO and RPO for the workload.

  1. Create or select a workload in the Well-Architected Tool. Set the environment, Regions, industry if relevant, and the pillars you will review.
  2. Record workload facts: entry points, data stores, dependencies, deployment method, rollback method, backup policy, and alert names.
  3. Answer the operational excellence, security, reliability, performance, and cost questions using evidence. Do not mark an answer complete unless you can point to the configuration or test result.
  4. Create a milestone named pre-launch-review.
  5. Export or copy the improvement plan into your team tracker. Label each item as launch blocker, accepted high risk, or post-launch improvement.
  6. Run one verification activity: restore a backup, simulate an unhealthy target, trigger a canary alarm, or perform a rollback in a non-production environment.
  7. Update the review notes with the verification result and final launch decision.

Verification is complete when every high risk is resolved or explicitly accepted, every accepted risk has an owner and date, and at least one recovery or rollback path has been exercised. Cleanup is administrative: delete test-only workloads from the Well-Architected Tool, remove temporary alarms or dashboards, and close any test incidents opened during verification.

Assessment Exercises

  1. A workload has automated backups but no restore test. Explain how you would classify the risk and what evidence would change your answer.
  2. An SQS consumer sometimes processes the same order twice. Identify the Well-Architected pillars involved and propose a production-ready correction.
  3. Choose one accepted high risk for a launch. Write the owner, expiration date, monitoring signal, and rollback condition that would make acceptance defensible.
  4. For a two-AZ web application with a single-AZ database, describe the likely symptoms during an AZ failure and the minimum test needed before launch.
  5. Review a cost finding caused by indefinite log retention. Decide whether it blocks launch and justify the trade-off.

Summary

A Well-Architected review turns architecture opinions into evidence-based risk decisions. For production readiness, focus on workload facts, pillar questions, risk classification, milestones, and improvement plans. The best review is specific: it names the failure mode, shows the evidence, identifies the owner, and proves that launch can be operated, recovered, secured, and paid for under realistic conditions.