CI-CD Plans, Approvals, Applies, and Drift Detection

Continuous delivery for Terraform has one job: turn a proposed infrastructure change into a reviewed, reproducible apply, while also detecting when real infrastructure stops matching code. In this lesson, the outcome is a pipeline that plans on every change, requires approval for protected environments, applies only the reviewed plan artifact, and runs drift detection without silently changing infrastructure.

This chapter sits in the testing and delivery part of the Terraform course. Earlier lessons explain state, providers, modules, and tests; here those pieces become a release process. The important point is that Terraform automation is not just a shell script around terraform apply. It is a control loop over configuration, state, provider APIs, credentials, locks, plan files, policies, and human approval.

Pipeline Mechanism

A Terraform CI-CD pipeline normally has separate plan and apply phases. The plan job checks out configuration, installs the pinned Terraform version, initializes the backend, downloads provider plugins from the dependency lock file, refreshes remote objects, compares refreshed state with configuration, and writes a binary saved plan with terraform plan -out=tfplan. That saved plan is not plain text. It contains the exact operation graph Terraform intends to execute, including creates, updates, deletes, and replacements calculated from provider schemas and current state.

The apply job should consume that saved plan with terraform apply tfplan. This matters because applying without a saved plan asks Terraform to calculate a new plan at apply time. A new plan may include different remote data, changed variables, or an unreviewed dependency update. The stronger release pattern is: plan, render a readable summary, attach it to the pipeline, run policy checks, receive approval, then apply the same artifact.

State locking is the concurrency guard. With a remote backend that supports locking, Terraform acquires a lock before planning or applying so two writers do not race over the same state snapshot. The state file maps Terraform resource addresses, such as aws_s3_bucket.logs, to provider object identifiers. A plan is therefore meaningful only for the workspace, backend, variables, provider versions, and state revision that produced it.

Configuration Anatomy

The useful boundary is usually one root module per deployable stack, one backend per state file, and one pipeline environment per Terraform workspace or directory. The following fragment shows the pieces a CI job expects: a remote backend, pinned providers, validated variables, and a managed resource with tags that make later drift checks easier.

terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }

  backend "s3" {
    bucket         = "example-tf-state"
    key            = "network/dev.tfstate"
    region         = "us-east-1"
    dynamodb_table = "example-tf-locks"
    encrypt        = true
  }
}

variable "environment" {
  type = string
  validation {
    condition     = contains(["dev", "stage", "prod"], var.environment)
    error_message = "environment must be dev, stage, or prod."
  }
}

resource "aws_s3_bucket" "artifacts" {
  bucket = "example-${var.environment}-artifacts"
  tags = {
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

The backend block is intentionally environment-specific: changing key changes which state file Terraform reads and writes. The provider constraint prevents surprise major upgrades, while the lock file records the resolved provider version and checksum. The variable validation fails before Terraform asks the provider to make remote changes. The tags provide a simple operational signal: resources created by this stack can be found and compared with state during audits.

Example 1: Plan on Pull Request

The first progressive step is a pull request job that validates configuration and produces a readable plan. It does not have permission to modify infrastructure. The job fails if formatting, initialization, validation, or planning fails.

name: terraform-plan
on:
  pull_request:
    paths:
      - "infra/**"

jobs:
  plan:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - name: Plan
        working-directory: infra
        run: |
          terraform fmt -check
          terraform init -input=false
          terraform validate
          terraform plan -input=false -out=tfplan
          terraform show -no-color tfplan > plan.txt

Expected behavior is deterministic at the command level: a formatting problem stops at terraform fmt -check; invalid syntax stops at terraform validate; a valid change produces tfplan and plan.txt. The text summary should be reviewed for action counts, replacements, and deletes. For example, a rename without a moved block often appears as one destroy and one create, which is not the same as an in-place rename.

Example 2: Approval and Saved Apply

The next step separates planning from applying. The plan job uploads the binary plan as an artifact. The apply job runs only after branch protection and environment approval, downloads that artifact, and applies it. The exact syntax differs by CI system, but the workflow shape is stable.

apply:
  needs: plan
  if: github.ref == 'refs/heads/main'
  runs-on: ubuntu-latest
  environment: production
  permissions:
    contents: read
    id-token: write
  steps:
    - uses: actions/checkout@v4
    - uses: hashicorp/setup-terraform@v3
    - uses: actions/download-artifact@v4
      with:
        name: terraform-plan
        path: infra
    - name: Apply reviewed plan
      working-directory: infra
      run: |
        terraform init -input=false
        terraform apply -input=false tfplan

The expected output includes Terraform acquiring the backend lock and then applying the actions already encoded in tfplan. If a reviewer approved a plan with one bucket tag update, the apply should not calculate a new virtual private cloud replacement. If the state changed after planning, Terraform should reject the stale plan or fail during provider operations, depending on the backend and change. The correction is to create and review a fresh plan.

Example 3: Drift Detection Without Apply

Drift detection asks a different question: if no configuration changed, do remote objects still match state and configuration? A scheduled job should plan with a detailed exit code and should not run apply. Terraform returns 0 when no changes are needed, 2 when changes are present, and 1 for an error.

#!/usr/bin/env bash
set -euo pipefail

terraform init -input=false
set +e
terraform plan -detailed-exitcode -input=false -out=drift.tfplan
status=$?
set -e

case "$status" in
  0) echo "No drift detected" ;;
  2) terraform show -no-color drift.tfplan > drift.txt
     echo "Drift detected; review drift.txt" ;;
  1) echo "Terraform plan failed" >&2
     exit 1 ;;
  *) echo "Unexpected Terraform exit code: $status" >&2
     exit 1 ;;
esac

If an operator changed a bucket tag in the cloud console, this job exits through the 2 branch and writes a plan showing Terraform will restore the configured tag. That is not automatically good or bad. The team must decide whether to import the intentional manual change into code, revert it with an apply, or remove the resource from Terraform ownership. The drift job should open an issue or alert with the plan summary, stack name, state key, and run link.

Design Choices

Choose directory-based environments when each environment has different backends, credentials, or release cadence. Choose workspaces when the configuration is truly identical and only variable values differ. Directory separation is more verbose but makes blast radius obvious in code review. Workspaces reduce duplication but can hide which state file is active unless the pipeline prints it clearly.

Choose speculative plans for pull requests and saved plans for protected applies. A speculative plan is useful feedback, but it should not be blindly applied days later. A saved plan is stronger when its lifetime is short, artifacts are access controlled, and the apply job uses the same commit. For high-risk stacks, require policy checks that reject deletes, public exposure, missing tags, unencrypted storage, or replacement of critical resources unless an explicit exception is attached.

Decide whether drift detection runs against every stack on a schedule or only critical stacks. Broad coverage finds more issues but can consume provider API quotas and create alert fatigue. A practical compromise is daily drift checks for production, less frequent checks for development, and immediate drift checks after incidents or emergency console changes.

Failure Modes and Troubleshooting

Symptom: the apply job says the saved plan is stale. Cause: state changed after the plan was created, often because another pipeline applied first or someone changed the same stack locally. Diagnosis: compare commit SHA, backend key, workspace, and state serial in the plan log; check recent applies. Correction: discard the artifact, rerun plan from the current state, and review the new diff.

Symptom: Terraform waits and then fails with a lock timeout. Cause: another run holds the state lock or a previous run exited after acquiring it. Diagnosis: inspect the backend lock record and the CI run history before forcing unlock. Correction: let the active run finish, cancel duplicate pipelines, or use terraform force-unlock only after proving no writer is active.

Symptom: pull request plans work, but production apply fails with access denied. Cause: plan credentials are broader than apply credentials, or the provider needs read permissions during apply that were not granted. Diagnosis: read the denied API action and resource ARN from provider logs or cloud audit logs. Correction: grant the narrow missing action to the apply role, keep destructive privileges behind approval, and rerun the saved plan only if it is still current.

Symptom: drift detection reports changes every day for the same field. Cause: the provider reads a computed value differently than the configuration sets it, or an external controller continuously mutates the object. Diagnosis: inspect the exact attribute path in terraform show, provider documentation, and cloud audit history. Correction: model the external controller in code, change ownership, or use ignore_changes narrowly for fields Terraform should not manage.

Security, Reliability, and Performance

CI credentials should be short lived and tied to the pipeline identity, not stored as long-lived cloud keys. The plan role needs read access and sometimes limited write-like permissions for provider validation; the apply role needs the actual mutation permissions and should be restricted to protected branches or environments. Treat state and plan files as sensitive because they may include resource identifiers, generated secrets, or values copied from provider responses.

Reliability comes from serialization and evidence. Use backend locking, CI concurrency groups per state key, immutable plan artifacts, and logs that print the Terraform version, workspace, backend key, provider lock file hash, commit SHA, and plan action counts. Performance issues usually come from very large state files, slow provider reads, or too many stacks planned at once. Split stacks along ownership and dependency boundaries, cache provider downloads when supported, and avoid scheduled drift jobs that all start at the same minute.

Hands-On Lab

Prerequisites: Terraform CLI, a Git repository, and either a real remote backend in a sandbox cloud account or a local backend for command practice. Do not use production credentials for the lab.

Create an infra directory with the HCL example above, adjusting the backend to a sandbox or replacing it with local state for practice.
Run terraform fmt -check, terraform init -input=false, and terraform validate. Verification: each command exits with status 0.
Run terraform plan -input=false -out=tfplan and then terraform show -no-color tfplan > plan.txt. Verification: plan.txt lists the expected creates, updates, or no-op result for your sandbox.
Review plan.txt as if it were a pull request comment. Mark any delete, replacement, public exposure, or missing tag as requiring explicit approval.
Apply the reviewed artifact with terraform apply -input=false tfplan. Verification: Terraform reports completion and a second terraform plan -detailed-exitcode returns 0.
Introduce drift by changing one managed tag outside Terraform, then run the drift script. Verification: it exits through the drift branch and writes drift.txt.
Cleanup: either restore the tag in code and apply, or run terraform destroy in the sandbox after removing any lifecycle rule that intentionally blocks destruction. Confirm the final plan is clean or the sandbox resources are gone.

Assessment

A teammate proposes running terraform apply -auto-approve on every merge without saving a plan. What specific risks does that introduce, and which pipeline step removes each risk?
A drift job reports that Terraform wants to undo a console change made during an incident. How would you decide whether to apply, import into code, or stop managing that field?
Your organization has ten environments with mostly identical configuration but different compliance controls. Would you choose workspaces or directories, and what evidence would change your answer?
An apply fails halfway through after creating one resource and failing on the next. What commands and logs would you inspect before rerunning the pipeline?
Design a policy rule for plans that blocks a dangerous change but still allows reviewed exceptions. What data must the plan expose to make the rule reliable?

Summary

Terraform CI-CD is a release system around state reconciliation. A dependable design plans with pinned dependencies, stores a short-lived reviewed plan, applies only that artifact after approval, and runs drift detection as a reporting workflow rather than an automatic repair job. The trade-offs are about blast radius, artifact trust, credential scope, lock behavior, API cost, and how quickly the team can explain and correct unexpected differences between code, state, and real infrastructure.