Capstone: Deliver a Production AWS Platform

This capstone pulls the Terraform course together by delivering a small but production-shaped AWS platform: a network foundation, protected shared storage, an application runtime, and an operational workflow that can plan, apply, verify, and recover changes. The outcome is a platform design where Terraform owns durable AWS objects, engineers review the exact proposed change before it happens, and failures have a known diagnostic path.

Production here means the platform can survive normal operator mistakes and common AWS failure conditions: concurrent applies, accidental public access, partial deployment, missing permissions, bad input values, and drift introduced outside Terraform. You will connect earlier mechanics: providers translate configuration into AWS API calls, state binds resource addresses to remote object identifiers, modules package repeatable infrastructure, and plans describe the action graph Terraform intends to execute.

Platform Shape and Outcome

The reference platform has four layers. The state layer stores Terraform state in an S3 backend and uses a lock table so only one apply mutates a workspace at a time. The network layer creates a VPC spanning two Availability Zones, with public subnets for load balancers and private subnets for workloads. The shared-services layer contains an encrypted, versioned artifact bucket with public access blocked. The runtime layer represents an ECS service behind an application load balancer. ECS is a useful capstone target because it forces decisions about subnets, security groups, health checks, IAM roles, logs, deployment capacity, and DNS.

The important boundary is ownership. Terraform should own long-lived infrastructure shape: VPCs, subnets, route tables, security groups, load balancers, buckets, clusters, IAM roles, and service declarations. It should not own every short-lived runtime fact. An ECS task ID appears and disappears as the scheduler works; Terraform should manage the service definition and desired count, while AWS manages individual task placement.

How Terraform Delivers the Platform

Terraform loads configuration from the root module and every child module. It reads provider requirements, downloads provider plugins during initialization, and asks the AWS provider for schemas that describe supported resource types and attributes. During planning, Terraform refreshes state by reading the real AWS objects named in state, evaluates expressions and dependencies, and constructs a graph of create, read, update, replace, and delete actions. Edges in that graph come from explicit references, such as a service using subnet IDs from the network module, and from provider-declared relationships.

State is the critical internal mechanism. A resource address such as module.network.aws_subnet.private[0] is bound to a real AWS subnet ID. If configuration changes a subnet CIDR, the provider schema tells Terraform whether AWS can update that field in place or whether the subnet must be replaced. If a resource is removed from configuration, Terraform interprets that as intent to destroy unless the address has been moved, imported elsewhere, or protected by lifecycle controls.

The AWS provider implements AWS-specific CRUD operations. Terraform Core decides graph order and state transitions, but the provider calls S3, ECS, EC2, IAM, and Elastic Load Balancing APIs and records returned identifiers and observed attributes. Timeouts, eventual consistency, throttling, and IAM denial all surface through provider operations, so production design must account for AWS API behavior, not just HCL syntax.

Configuration Anatomy

A production root module usually contains backend configuration, provider configuration, shared locals, module calls, environment-specific inputs, and outputs that other systems need. Child modules should expose a narrow interface: variables for values that callers may decide, resources for the implementation, and outputs for stable facts that downstream layers can consume. Avoid leaking every internal ID from a module. Expose subnet IDs, a load balancer DNS name, or a cluster ARN only when another layer genuinely needs them.

Use naming and tags as part of the API. A stable name_prefix, an Environment tag, and a ManagedBy tag make cost allocation, search, incident response, and import safer. Use variable validation where bad input would create invalid topology, such as CIDR ranges, environment names, or minimum replica counts. Use lifecycle settings selectively. prevent_destroy is appropriate for state buckets and critical data stores, but overusing it can block legitimate replacement work.

Example 1: Remote State and Network Foundation

The first example establishes the state boundary and calls a network module. The S3 backend gives Terraform one shared state location for the production network workspace. The lock table prevents two applies from writing conflicting state snapshots. The module call keeps subnet, route table, internet gateway, and NAT gateway details behind a clean interface.

terraform {
  backend "s3" {
    bucket         = "acme-tfstate-prod"
    key            = "platform/network.tfstate"
    region         = "us-east-1"
    dynamodb_table = "acme-tfstate-locks"
    encrypt        = true
  }
}

locals {
  name_prefix = "acme-prod"
  azs         = ["us-east-1a", "us-east-1b"]
}

module "network" {
  source = "./modules/network"

  name_prefix          = local.name_prefix
  vpc_cidr             = "10.40.0.0/16"
  availability_zones   = local.azs
  public_subnet_cidrs  = ["10.40.0.0/24", "10.40.1.0/24"]
  private_subnet_cidrs = ["10.40.10.0/24", "10.40.11.0/24"]
}

Expected behavior: after terraform init, Terraform configures the S3 backend and will acquire a lock before state-writing operations. A plan against an empty workspace should propose creating the network resources implemented by ./modules/network. Network state changes less often than service state, so it deserves its own backend key and a smaller approver group.

Example 2: Guarded Shared Artifact Storage

The next layer creates an artifact bucket. The bucket is versioned so accidental overwrites can be recovered, tagged so ownership is visible, and protected by a public access block so a later bucket policy mistake cannot silently expose artifacts.

resource "aws_s3_bucket" "artifacts" {
  bucket = "acme-prod-platform-artifacts"

  tags = {
    Application = "platform"
    Environment = "prod"
    ManagedBy   = "Terraform"
  }
}

resource "aws_s3_bucket_versioning" "artifacts" {
  bucket = aws_s3_bucket.artifacts.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_public_access_block" "artifacts" {
  bucket                  = aws_s3_bucket.artifacts.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

Expected behavior: a plan should show one bucket, one versioning configuration, and one public access block. The public access block is a separate resource because the AWS provider models it as separate S3 configuration. If someone later adds a policy that would make the bucket public, restrict_public_buckets causes AWS to reject public access rather than relying only on review.

Example 3: Runtime Service Wiring

The third example wires a service module to the platform. It consumes the cluster ARN from compute, private subnet IDs from network, and a target group ARN from the edge layer. Terraform infers graph order from those references: the service cannot be planned with concrete IDs until upstream module outputs are known.

module "service" {
  source = "./modules/ecs-service"

  name             = "orders-api"
  cluster_arn      = module.compute.cluster_arn
  private_subnets  = module.network.private_subnet_ids
  target_group_arn = module.edge.orders_target_group_arn
  desired_count    = 3

  environment = {
    APP_ENV = "prod"
  }
}

output "orders_url" {
  value = "https://${module.edge.dns_name}/orders"
}

Expected behavior: a plan should show the service using private subnets and registering with the existing target group. After apply, the output should print a URL shaped like https://example-alb-name/orders, depending on the real DNS name from the edge module. The service module should configure health checks and deployment settings so new tasks become healthy before old capacity is removed.

Example 4: Tests as Platform Contracts

Terraform tests encode platform rules before human plan review. They do not replace integration tests, but they catch configuration regressions close to the source. This example checks that the artifact bucket remains private and that a production service does not run as a single task.

run "artifact_bucket_is_private" {
  command = plan

  assert {
    condition     = aws_s3_bucket_public_access_block.artifacts.restrict_public_buckets == true
    error_message = "artifact bucket must reject public bucket policies"
  }
}

run "service_has_multiple_tasks" {
  command = plan

  variables {
    desired_count = 3
  }

  assert {
    condition     = var.desired_count >= 2
    error_message = "production service must run at least two tasks"
  }
}

Expected behavior: terraform test should pass when the public access block is enabled and the service replica count is at least two. If a later change sets restrict_public_buckets to false or drops the desired count below two, the test fails before apply. The failure is tied to a platform rule, not to a vague readiness slogan.

Design Choices and Trade-Offs

Split state by blast radius. A network workspace, a shared-services workspace, and one workspace per application environment are often easier to operate than a single giant state file. The trade-off is dependency management: downstream workspaces need stable outputs from upstream layers, usually read through remote state data sources, a parameter store, or CI-provided variables. Too many tiny workspaces create orchestration overhead; one huge workspace makes plans slow and risky.

Choose modules around ownership boundaries. A VPC module is reasonable because network topology changes as a unit. A module that wraps a single tag on a single resource usually hides more than it helps. Keep module variables typed. Prefer objects when a concept travels together, such as listener rules, container port mappings, or scaling thresholds. Avoid passing raw provider resources through module boundaries; pass stable strings, numbers, maps, and lists.

Use automation for repeatability, but keep human review for production applies. A strong pipeline formats configuration, validates it, runs tests and policy checks, creates a saved plan, and applies exactly that saved plan after approval. Do not run a fresh unreviewed plan at apply time for production, because the infrastructure may have changed between review and execution.

Failure Modes and Troubleshooting

Symptom: a pipeline hangs or fails with a state lock error. Cause: another apply is running, or a previous interrupted run left a lock record. Diagnostics: inspect pipeline history, confirm no active apply is still running, and read lock metadata for operation, user, and timestamp. Correction: wait for the active run, or use Terraform’s force-unlock procedure only after proving the writer is gone.

Symptom: the plan proposes replacing subnets, route tables, or the load balancer after a module refactor. Cause: resource addresses changed, list indexes shifted, or an immutable argument such as a CIDR block changed. Diagnostics: compare old and new addresses in the plan, inspect state with terraform state list, and look for moved blocks that should preserve identity. Correction: add moved blocks for pure renames, use stable for_each keys instead of positional lists, and schedule intentional replacements with a migration plan.

Symptom: an ECS service apply times out waiting for stability. Cause: tasks cannot start or pass health checks because of a bad image, missing IAM permission, blocked egress, wrong security group, or mismatched target group port. Diagnostics: inspect ECS service events, task stopped reasons, target group health, and CloudWatch logs. Correction: fix the failing runtime dependency, rerun the plan, and verify that the service reaches steady state before routing user traffic.

Symptom: Terraform wants to undo a console change every run. Cause: drift exists outside Terraform’s desired configuration. Diagnostics: identify the changed attribute in the plan and decide whether it was an emergency fix, an unmanaged system action, or a manual mistake. Correction: encode the desired change in Terraform, import unmanaged infrastructure when appropriate, or revert the console change. Avoid broad ignore_changes unless the external owner is explicit.

Security, Reliability, and Performance Implications

Use short-lived CI credentials for plans and applies. Permissions should match the platform layer, so a service pipeline cannot rewrite the state bucket or replace the production VPC. Protect state because it can contain identifiers, generated secrets, and sensitive outputs. Enable encryption on the backend bucket and restrict access to the automation role and a small break-glass group.

Reliability comes from reducing unknowns at apply time. Pin provider sources and acceptable versions, keep module inputs deterministic, and avoid data sources that select resources by loose names when an exact ID is available. Performance matters in plan time and AWS API throttling. Large monolithic states refresh slowly, and broad data sources can make every plan expensive. Split state and tighten selectors when plans become noisy or slow.

Hands-On Lab: Build and Prove a Platform Slice

Prerequisites: an AWS sandbox account, Terraform installed locally or in CI, an IAM role with permission to create a VPC and S3 bucket, and a pre-created backend bucket and lock table. Use a unique prefix so names do not collide.

  1. Create a new root module with backend configuration like Example 1, changing bucket, key, region, and lock table to your sandbox values.
  2. Add a small network module call with two Availability Zones, two public subnet CIDRs, and two private subnet CIDRs.
  3. Add the artifact bucket from Example 2 with a globally unique bucket name, versioning enabled, and public access blocked.
  4. Add the Terraform tests from Example 4 and adjust variable names to match your module.
  5. Run terraform fmt, terraform init, terraform validate, and terraform test.
  6. Run terraform plan -out=tfplan and review every create, replace, and destroy action. Confirm there are no destroy actions in a new sandbox deployment.
  7. Apply the saved plan with terraform apply tfplan.
  8. Verify in AWS that the bucket has versioning enabled and all four public access block settings set to true. Verify that subnets are distributed across the intended Availability Zones.
  9. Change the bucket public access block to an unsafe value and rerun terraform test. Confirm the test fails, then restore the safe value.
  10. Cleanup by running terraform destroy in the sandbox. Keep the shared backend if other labs use it.

Assessment Exercises

  1. A refactor changes aws_subnet.private[0] to aws_subnet.private["az-a"], and the plan shows replacement. What Terraform feature can preserve the binding, and what evidence would you review before apply?
  2. Your production apply fails because the ECS service never stabilizes. List the AWS places you would inspect first and explain how each one narrows the cause.
  3. Design a state split for a platform with network, database, shared CI artifacts, and three services. Which dependencies cross workspace boundaries, and how would you expose them?
  4. Write a platform rule that belongs in terraform test and a different rule that belongs in an integration or smoke test. Explain why they belong at different layers.
  5. A teammate proposes adding ignore_changes = all to reduce noisy plans. What risks does that introduce, and what narrower alternatives would you consider?

Summary

The capstone is where Terraform becomes an operating model for AWS. A production platform needs protected state, deliberate module boundaries, deterministic plans, reviewed applies, verification after deployment, and practiced recovery for lock, drift, replacement, and service-health failures. The strongest implementation is the one where every durable AWS object has an owner, every cross-layer dependency is explicit, and every plan can be explained before it changes the account.