Manage Kubernetes and Helm Resources

Terraform can manage more than cloud infrastructure. In an AWS and Kubernetes platform, it can also create Kubernetes API objects and install Helm charts after the cluster exists. The practical outcome is a single dependency graph that can say: create the namespace, write the configuration object, then install the chart that consumes it. That is useful for platform-owned resources such as namespaces, ingress controllers, certificate managers, storage drivers, metrics agents, and baseline application scaffolding.

The important boundary is ownership. Terraform is strongest when it owns relatively stable platform resources whose desired state belongs in reviewable code. It is weaker for rapidly changing objects controlled by Kubernetes controllers, deployment pipelines, or operators. This lesson teaches how the Kubernetes and Helm providers work, where their state comes from, and how to design configurations that do not fight the cluster.

How Terraform Talks to Kubernetes

The Kubernetes provider is a Terraform provider plugin that translates resource blocks into Kubernetes API calls. During planning it reads existing objects from the API server and compares selected fields with Terraform state and configuration. During apply it creates, patches, replaces, or deletes objects by using the Kubernetes REST API. The provider does not talk to kubelet, etcd, or container runtimes directly; the API server remains the enforcement point for authentication, authorization, admission, defaults, and validation.

A Terraform resource address such as kubernetes_namespace.training is bound in state to a remote object identified by API group, kind, namespace when applicable, and name. Kubernetes itself also has object identity fields such as metadata.uid, resourceVersion, labels, annotations, and owner references. Terraform normally cares about stable desired fields, while Kubernetes continuously mutates operational fields such as status, managed fields, defaulted values, and generated timestamps. A good Terraform configuration avoids trying to own fields that the control plane or another controller is expected to change.

The Helm provider uses Helm libraries to install or upgrade a release. A helm_release resource is not one Kubernetes object; it is a release record plus the rendered manifests created from a chart, values, and chart dependencies. Terraform stores the release settings in state, while Helm stores release history in the cluster. With wait = true, Terraform waits for the resources that Helm recognizes as part of the release to become ready before marking the apply successful.

Provider Configuration Anatomy

Both providers need Kubernetes credentials. For local labs this is often a kubeconfig file and context. In CI for an AWS EKS cluster, the provider is commonly configured from the EKS endpoint, certificate authority data, and a short-lived authentication token. The same Terraform run can use the AWS provider to build the cluster and then pass cluster connection details into the Kubernetes and Helm providers, but that creates a timing dependency: the Kubernetes API must be reachable before those providers can refresh or apply resources.

The first example creates only a namespace. It demonstrates the smallest useful Terraform-owned Kubernetes object: a named container for later resources, with labels that make ownership visible.

terraform {
  required_providers {
    kubernetes = {
      source = "hashicorp/kubernetes"
    }
  }
}

provider "kubernetes" {
  config_path    = "~/.kube/config"
  config_context = "kind-platform"
}

resource "kubernetes_namespace" "training" {
  metadata {
    name = "tf-training"
    labels = {
      owner      = "terraform"
      curriculum = "aws-kubernetes"
    }
  }
}

When planned against an empty cluster, Terraform should propose one create action for kubernetes_namespace.training. After apply, kubectl get namespace tf-training --show-labels should show the namespace with owner=terraform and curriculum=aws-kubernetes. Reapplying without configuration changes should produce no changes. That no-op result is important evidence that Terraform state, configuration, and the live Kubernetes object agree.

Worked Example 2: A ConfigMap Dependency

Kubernetes objects often depend on a namespace. Terraform captures that dependency when one resource references another resource’s attribute. The following ConfigMap uses the namespace returned by the namespace resource, so Terraform knows it must create the namespace first and delete the ConfigMap before deleting the namespace.

resource "kubernetes_config_map" "app_settings" {
  metadata {
    name      = "web-settings"
    namespace = kubernetes_namespace.training.metadata[0].name
  }

  data = {
    LOG_LEVEL       = "info"
    FEATURE_BANNER  = "enabled"
    UPSTREAM_REGION = "us-east-1"
  }
}

The expected live object is a ConfigMap named web-settings in tf-training with three keys. If someone edits LOG_LEVEL manually to debug, the next plan should show Terraform changing it back to info. That is not a bug; it is Terraform enforcing ownership. If application teams need to change this data independently, the ConfigMap should move to their deployment process, or Terraform should own only a different platform-level object.

Worked Example 3: Installing a Helm Chart

Helm charts bundle many Kubernetes resources behind a smaller release interface. Terraform sees the helm_release as one resource, but a chart can create Deployments, Services, ServiceAccounts, ConfigMaps, Secrets, Jobs, or custom resources. This example installs an NGINX chart into the namespace created earlier and forces an internal ClusterIP service for a lab-friendly result.

terraform {
  required_providers {
    helm = {
      source = "hashicorp/helm"
    }
    kubernetes = {
      source = "hashicorp/kubernetes"
    }
  }
}

provider "helm" {
  kubernetes {
    config_path    = "~/.kube/config"
    config_context = "kind-platform"
  }
}

resource "helm_release" "nginx" {
  name             = "training-nginx"
  repository       = "https://charts.bitnami.com/bitnami"
  chart            = "nginx"
  namespace        = kubernetes_namespace.training.metadata[0].name
  create_namespace = false
  wait             = true
  timeout          = 300

  set {
    name  = "service.type"
    value = "ClusterIP"
  }
}

With an available chart repository and a healthy cluster, the plan should show one Helm release create. After apply, helm status training-nginx -n tf-training should report a deployed release, and kubectl get service -n tf-training should include a ClusterIP service for the chart. The deterministic part is the Terraform graph and configured values; exact pod names are generated by Kubernetes and should not be hard-coded in tests.

Worked Example 4: Values as a Contract

For non-trivial charts, values are the contract between Terraform and the chart. Inline set blocks are convenient for small scalar overrides, but nested configuration is easier to review when encoded as YAML from structured Terraform values. Replace the previous release block with the next version when you want the chart settings to be reviewed as one structured value.

resource "helm_release" "nginx" {
  name       = "training-nginx"
  repository = "https://charts.bitnami.com/bitnami"
  chart      = "nginx"
  namespace  = kubernetes_namespace.training.metadata[0].name
  wait       = true
  timeout    = 300

  values = [yamlencode({
    replicaCount = 2
    service = {
      type = "ClusterIP"
    }
    resources = {
      requests = {
        cpu    = "100m"
        memory = "128Mi"
      }
      limits = {
        cpu    = "250m"
        memory = "256Mi"
      }
    }
  })]
}

The expected behavior is an upgraded Helm release with two desired replicas and resource requests and limits on the chart’s workload, assuming the chart supports those values. This example also illustrates a trade-off: Terraform can validate HCL structure, but it cannot guarantee that every chart value is meaningful unless the chart schema defines and enforces it. A misspelled value key may plan and apply successfully while changing nothing. Always verify rendered or live manifests for important chart settings.

Design Choices and Trade-offs

Use Terraform for cluster add-ons and platform baseline resources when reviewability, dependency ordering, and stateful reconciliation are more valuable than rapid per-commit deployment. Use Kubernetes-native tools, GitOps controllers, or application pipelines when objects change many times per day, depend on image promotion workflows, or are already reconciled by another controller.

Prefer typed Kubernetes resources, such as kubernetes_namespace or kubernetes_config_map, when the provider supports the kind and the schema is stable. Typed resources give clearer plans and input validation. Use manifest-style resources only when you need unsupported or custom resource kinds, and be more careful about server-side defaults and field ownership. For Helm, pin chart repositories and chart versions in real environments, review chart changes before upgrade, and understand whether a chart installs custom resource definitions. CRDs often need special handling because the API type must exist before custom resources of that type can be created.

State is another design choice. Terraform state can include Kubernetes object data and Helm values. Secrets are especially sensitive: a Kubernetes Secret managed by Terraform places secret material in state unless you design around that. Store state in a protected remote backend with locking and encryption, restrict read access, and avoid using Terraform as a general secret distribution mechanism when an external secret operator or cloud secret manager is the better owner.

Failure Modes and Troubleshooting

A common symptom is dial tcp, connection refused, or no such host during plan. The cause is usually that the Kubernetes API endpoint is unreachable from the machine or runner executing Terraform. Check the active kubeconfig context, network path, private endpoint access, VPN, and security groups. Correct the runner placement or provider configuration before retrying; changing resource definitions will not fix an unreachable API server.

If apply fails with Forbidden, authentication succeeded but authorization did not. Diagnose with the same identity Terraform uses, for example by checking whether it can create namespaces or install resources in the target namespace. Correct the Role, ClusterRole, RoleBinding, or cloud identity mapping. Avoid granting broad cluster-admin permissions just to make the error disappear; inventory the verbs and resource types the configuration really needs.

If a Helm release times out while pods show ImagePullBackOff, CrashLoopBackOff, or unschedulable events, Terraform is reporting the symptom that Helm waited for readiness and the workload never became ready. Inspect pods, events, image names, node capacity, pull secrets, and chart values. Correct the chart values or cluster prerequisites, then rerun apply. If the release is stuck in a pending or failed state, a targeted helm status, helm history, and sometimes helm rollback or release cleanup may be needed before Terraform can proceed cleanly.

If Terraform repeatedly wants to change fields you did not configure, Kubernetes defaulting or another controller may be mutating them. Compare the plan with kubectl get ... -o yaml, identify the field owner, and decide which system should control that field. The correction may be to remove the field from Terraform, add ignore_changes for a narrowly selected attribute, or stop the external mutation. Use lifecycle ignores sparingly because they hide future drift in that attribute.

Security, Reliability, and Performance

Kubernetes and Helm applies are powerful because they can change running workloads. Run Terraform with a dedicated identity, least privilege, short-lived credentials, and a clearly scoped kubeconfig. Protect plans and logs because provider diagnostics may include object names, chart values, and sometimes sensitive data. Review any chart before installation; a chart can create RBAC bindings, privileged workloads, webhooks, or jobs that run during install and upgrade.

Reliability depends on separating cluster creation from in-cluster configuration when the dependency chain becomes fragile. For small labs, one configuration is convenient. For production, many teams apply the EKS cluster first, wait for API stability and node readiness, then apply Kubernetes and Helm layers. This reduces refresh failures and makes rollback easier. Performance concerns are usually API-server pressure and slow chart readiness rather than Terraform CPU. Large numbers of manifest resources can make plans noisy and applies slower, so group stable add-ons carefully and avoid using Terraform as a high-frequency deployment controller.

Hands-on Lab

Prerequisites: Terraform installed, a disposable Kubernetes cluster such as kind or a non-production EKS cluster, kubeconfig access to that cluster, Helm access to the public chart repository used in the example, and permissions to create a namespace, ConfigMap, and namespaced workloads. Do not run the lab against a shared production namespace.

  1. Create a new empty Terraform directory and add the namespace example. Run terraform init, then terraform plan. Verify that the plan proposes one namespace create.
  2. Apply the namespace. Verify with kubectl get namespace tf-training --show-labels and confirm the two labels are present.
  3. Add the ConfigMap example. Plan and apply. Verify with kubectl get configmap web-settings -n tf-training -o yaml and confirm the three data keys.
  4. Manually change LOG_LEVEL to debug with kubectl, then run terraform plan. Verify that Terraform detects drift and proposes restoring info.
  5. Add the Helm provider and release example, run terraform init again if the Helm provider is new, then plan and apply. Verify with helm status training-nginx -n tf-training and kubectl get pods,svc -n tf-training.
  6. Cleanup by running terraform destroy from the lab directory. If a Helm release remains failed or pending, inspect Helm history and remove only the lab release and namespace after confirming no unrelated objects are inside.

Assessment Exercises

  1. A platform team and an application team both want to update the same ConfigMap. Which team should own it in Terraform, and what evidence would you use to decide?
  2. A Helm chart upgrade succeeds from Terraform’s perspective, but a value key was misspelled and the workload did not change. What verification step would catch this before production?
  3. Your Terraform runner can describe pods but cannot create RoleBindings. Explain how this appears during apply and how to correct it without granting cluster-admin.
  4. An EKS cluster and its Helm add-ons are in one Terraform configuration, and plans fail whenever the cluster endpoint is temporarily unavailable. What restructuring would make the workflow more reliable?
  5. Choose one field from a Kubernetes object that Terraform should usually not own. Explain which controller owns it and what drift symptoms you would expect.

Summary

Terraform manages Kubernetes resources by reconciling provider schemas, state, configuration, and live API objects through the Kubernetes API server. It manages Helm charts by treating a Helm release as a Terraform resource whose rendered objects live in the cluster. The strongest designs use Terraform for stable platform-owned resources, keep application deployment loops separate when needed, protect state, verify live objects after apply, and troubleshoot from the actual boundary: provider credentials, API reachability, RBAC, Helm release status, and Kubernetes workload events.