Troubleshoot Plans, Partial Applies, and State Failures

Terraform troubleshooting is the practice of proving what Terraform thinks exists, what actually exists, and what action graph Terraform is about to execute. In this operations section, the outcome is practical: when a plan looks wrong, an apply stops halfway, or state cannot be read or written, you should know how to slow the change down, inspect the right evidence, and recover without making the drift worse.

The most important habit is to treat Terraform as a stateful reconciliation engine, not as a script runner. A plan is not merely a preview. It is Terraform’s calculated transition from the current state snapshot and refreshed provider observations to the desired configuration. A partial apply is not a mysterious half-run. It is a graph walk that completed some operations, failed on another, and then wrote whatever successful changes it could safely record. A state failure is serious because state is the binding between Terraform resource addresses and remote object identities.

How Terraform Builds a Plan

Terraform starts with configuration files, input variables, provider schemas, dependency edges, and the latest available state. During planning, it refreshes managed objects unless refresh is disabled, asks providers to compare prior state with configuration, and builds a graph of actions such as create, update in place, replace, read, or delete. The plan is therefore a combination of Terraform Core logic and provider-specific behavior. If a provider marks an argument as requiring replacement, Terraform shows a destroy-and-create even when the visible HCL change looks small.

Resource addresses are central. The address aws_instance.web[0], for example, is the configuration identity. State then records the provider type, object id, attributes, dependencies, and metadata associated with that address. If the address changes because a resource was renamed, moved into a module, or converted from count to for_each, Terraform may think the old object should be destroyed and a new object should be created unless you declare the move or adjust state intentionally.

Provider refresh is another common source of plan surprises. Terraform may show a change you did not edit because the remote object drifted, a default value changed after provider normalization, a computed value became known, or a sensitive value cannot be compared exactly. The first question for any surprising plan is not “why is Terraform wrong?” It is “which input changed: configuration, state, provider schema, variables, workspace, credentials, or the remote object?”

Plan Anatomy

A readable plan has several signals. The leading symbol tells you the action: + create, ~ update in place, - destroy, and -/+ or +/- replace. Attribute markers matter too. (known after apply) means Terraform cannot know the value until the provider creates or reads the object. (sensitive value) means the value is intentionally hidden. Text such as forces replacement points to a provider schema decision, not merely Terraform preference.

Use saved plans when applying through automation. A saved plan file ties review and execution to the same calculated graph, reducing the risk that a variable, provider, or remote object changes between review and apply. The trade-off is that a saved plan can become stale; if the infrastructure or state changes after the plan is created, rerun planning instead of treating the stale file as authority.

terraform init
terraform plan -out=tfplan
terraform show tfplan
terraform apply tfplan

The expected behavior is that terraform apply tfplan executes the reviewed plan rather than recalculating a new one. If a teammate changed the same state after the plan was saved, Terraform should refuse or fail instead of silently merging two separate intentions.

Example 1: A Simple Update Plan

The first example changes metadata on a local file. It is intentionally small so the plan shape is easy to read. The resource address remains the same, and the file path remains the same, so Terraform can update the object in place.

resource "local_file" "runbook" {
  filename = "runbook.txt"
  content  = "owner=platform\nseverity=low\n"
}

If the previous state recorded the same filename with severity=medium, the plan should show an in-place update for local_file.runbook. The deterministic outcome after apply is that runbook.txt contains owner=platform and severity=low. This example teaches the normal case: same address, same remote identity, changed mutable attribute.

Example 2: Replacement From Identity Change

The next example changes the object identity. For many providers, names, regions, immutable sizes, or parent relationships can force replacement. The local provider models that idea through the filename: changing it creates a new file and removes the old one.

resource "local_file" "runbook" {
  filename = "incident-runbook.txt"
  content  = "owner=platform\nseverity=low\n"
}

If state currently binds local_file.runbook to runbook.txt, the plan should show replacement because the filename is the object’s identity for this provider. The expected behavior after apply is that incident-runbook.txt exists with the configured content and the old managed file is removed. In cloud providers, the same pattern appears when changing an immutable database name, subnet placement, or launch template property.

Example 3: Address Moves Without Destruction

Refactors are dangerous when Terraform cannot connect the old address to the new address. A moved block tells Terraform that the object identity is continuing under a new configuration address. This is safer than letting Terraform plan one destroy and one create for the same underlying object.

moved {
  from = local_file.runbook
  to   = module.docs.local_file.runbook
}

The expected plan should say that the object has moved rather than showing a destroy-create pair. This does not change the remote file by itself; it changes Terraform’s address mapping. Use this pattern when reorganizing modules, renaming resources, or replacing count addresses with stable for_each keys. The trade-off is that moved blocks must be reviewed and kept long enough for every workspace to pass through the migration.

Partial Applies

Terraform applies a dependency graph. If resource A succeeds, resource B succeeds, and resource C fails, Terraform does not automatically undo A and B. It records completed operations in state when possible, reports the failure, and exits nonzero. That behavior is deliberate: providers cannot reliably roll back arbitrary infrastructure. A created DNS record, database, IAM role, or network gateway may have side effects outside Terraform’s control.

When an apply fails, do not immediately run broad cleanup commands. First capture the error, preserve the state file or backend version, and inspect which resources changed. Then run a new plan. The next plan is the best reconciliation document because it starts from the updated state and refreshed real objects. If the failed operation created a remote object but Terraform did not record it, the next plan may propose creating a duplicate. In that case, import the object, remove the duplicate configuration, or manually delete the orphan before applying again.

Failure Scenario: Plan Wants to Destroy Too Much

Symptoms: a plan shows many destroys after a small edit, especially after renaming resources, switching workspaces, changing variable files, or moving code into modules.

Likely cause: Terraform addresses changed, the wrong workspace or backend is selected, or required variables now resolve to different names. Terraform is not seeing the existing objects at the addresses it expects.

Diagnostics: run terraform workspace show, inspect backend configuration, list state addresses with terraform state list, and compare the plan against the exact code diff. Search for resource renames and changed for_each keys.

Correction: add moved blocks for legitimate refactors, restore the correct workspace or variable file, or use terraform state mv only when you have verified the old and new addresses represent the same remote object.

Failure Scenario: Apply Failed After Creating Some Objects

Symptoms: apply exits with an API error, but some infrastructure is visible in the provider console. A later plan proposes either finishing remaining objects or creating something that appears to already exist.

Likely cause: the graph partially completed. Terraform recorded successful operations it knew about, but the failed provider call may have left an object outside state.

Diagnostics: read the apply log, run terraform state list, query the provider for the named object, and run a fresh terraform plan. Identify whether the object is managed, missing, or orphaned.

Correction: fix the original cause, such as quota, permissions, invalid arguments, or dependency readiness. Import any orphan that should be managed, or delete it manually if it was an unwanted failed attempt. Then rerun plan before apply.

Failure Scenario: State Lock or Backend Error

Symptoms: Terraform reports that it cannot acquire a lock, cannot read state, cannot write state, or found a checksum or version conflict.

Likely cause: another run is active, a previous run died while holding a lock, credentials cannot access the backend, or the backend storage has a consistency or permission problem.

Diagnostics: confirm no automation job is still running, inspect backend access permissions, check lock metadata, and review backend object version history where available. Never assume a lock is stale just because it is inconvenient.

Correction: let active runs finish. If a lock is proven abandoned, use the backend’s normal unlock procedure or terraform force-unlock with the exact lock id. Restore a previous state version only after comparing it with real infrastructure, because old state can be as harmful as corrupt state.

Design Choices and Trade-offs

Remote state with locking is usually worth the operational complexity for team environments because it serializes writes and provides a central recovery point. Local state is simpler for a lab but fragile in collaboration. Saved plans improve review discipline, but they require automation that prevents stale execution. Targeted applies can unblock recovery, but routine use of -target trains Terraform on an incomplete graph and can hide dependency problems. Manual state commands are powerful recovery tools, not normal deployment tools.

Plan noise is also a design issue. Highly dynamic names, timestamps, unstable ordering, and provider defaults that are not pinned in configuration can make every plan look busy. Busy plans reduce review quality. Prefer stable keys, explicit arguments for important defaults, and lifecycle rules only when they express a real invariant. ignore_changes can be correct for externally managed tags, but it can also mask drift that Terraform should repair.

Reliability and Security Implications

State often contains sensitive values, provider ids, private endpoints, and dependency details. Store it in a protected backend, restrict read access, and avoid attaching state files to tickets. Reliability depends on state versioning, locking, provider pinning, and small changes that can be reviewed. Security depends on the same mechanics because a mistaken plan can remove protections, expose data, or recreate resources with weaker defaults.

For production, keep run logs, saved plan artifacts, and approval records long enough to reconstruct what happened. A useful incident review can answer which configuration commit ran, which state version it read, which plan was approved, which credentials applied it, and what state version resulted.

Hands-On Lab: Recover a Partial Local Apply

Prerequisites: Terraform installed locally, a clean temporary directory, and permission to create files in that directory. This lab uses the local provider so it does not require cloud credentials.

  1. Create a new directory and add the configuration below.
  2. Run terraform init and terraform apply. Approve the apply.
  3. Change the second file path to a directory that does not exist, such as missing/path/b.txt, and run terraform apply again.
  4. Observe that one resource may already be updated while the other fails. Run terraform state list and inspect the files on disk.
  5. Fix the bad path or create the missing directory, then run terraform plan. Apply only after the plan explains the remaining work.
  6. Verify by reading both files. Cleanup with terraform destroy and remove the temporary directory.
resource "local_file" "a" {
  filename = "a.txt"
  content  = "alpha\n"
}

resource "local_file" "b" {
  filename = "b.txt"
  content  = "bravo\n"
}

The verification is deterministic after a successful final apply: a.txt contains alpha and b.txt contains bravo. If the failed apply left a changed a.txt but no valid b.txt, that is the partial apply behavior you are learning to diagnose.

Assessment Exercises

  1. A plan shows one destroy and one create after a resource rename, but the cloud object should remain the same. What evidence would prove a moved block is the right fix?
  2. An apply failed with a timeout, and the provider console shows the object exists. How would you decide between import, manual deletion, and retry?
  3. Why can terraform apply without a saved plan be riskier in automation than applying a reviewed plan file?
  4. A teammate suggests using -target for every urgent fix. What graph and state risks would you raise?
  5. State restoration is available from yesterday’s backend version. What must you compare before restoring it?

Summary

Troubleshooting Terraform means following the relationship among configuration addresses, provider refresh, planned graph actions, remote objects, and state snapshots. Surprising plans usually come from changed inputs or changed identity. Partial applies are graph failures that require reconciliation, not panic. State failures require disciplined locking, version inspection, and minimal repair. The operator’s job is to preserve the mapping between Terraform’s model and real infrastructure while making the next plan boring again.