Upgrades, Rollbacks, GitOps, and Environment Promotion

Upgrades, rollbacks, GitOps, and environment promotion are the release engineering loop for Helm charts. The outcome is not merely running helm upgrade; it is moving a known chart version and a known values set through clusters, detecting whether Kubernetes accepted and rolled out the change, and having a repeatable way to return to the previous release when the new desired state is wrong.

In this course, earlier lessons built charts, templates, values, hooks, and dependency boundaries. This chapter puts those pieces into an operator workflow. You will learn what Helm records during an upgrade, how rollback chooses an older revision, how GitOps controllers apply rendered or referenced charts, and how promotion keeps development, staging, and production different without letting them drift accidentally.

Release State and the Helm Upgrade Mechanism

A Helm release is a named installation of a chart in a namespace. Each time Helm installs, upgrades, or rolls back that release, it creates a numbered revision. By default, Helm stores release metadata in Kubernetes Secrets in the same namespace. That record includes the chart, values, rendered manifest, status, and revision history. Kubernetes still owns the live objects, but Helm owns the mapping from chart plus values to the manifest it last submitted.

helm upgrade renders templates using the candidate chart and values, compares ownership annotations on existing objects, and sends create, patch, or delete operations to the Kubernetes API. The Deployment controller, StatefulSet controller, Job controller, and other controllers then reconcile those API objects. Helm does not replace Kubernetes rollout logic; it waits for selected resources only when you ask it to with flags such as --wait. Without waiting, a command can succeed even while a Deployment later fails to become available.

The main upgrade flags express release engineering choices. --install makes the command idempotent for first deployment. --namespace scopes both objects and release history. --values layers environment configuration. --set is useful for short overrides but weak for review because shell history and CI logs can expose sensitive or ambiguous values. --atomic rolls back automatically when an upgrade fails, and it implies waiting. --timeout defines how long Helm waits before declaring failure. --history-max limits stored revision history, which matters because rollback cannot target a revision that has been pruned.

GitOps and Promotion Anatomy

In GitOps, Git is the reviewed source of desired state, and an in-cluster controller applies that desired state. The controller may render a Helm chart directly from an application definition, or it may apply manifests that CI rendered from Helm. Either pattern can work. The important distinction is where rendering happens and which identity has permission to mutate the cluster.

Environment promotion means advancing an artifact through environments without rebuilding it differently each time. For Helm, the promoted artifact is usually a chart version, an image digest or immutable image tag, and a values file or overlay. Development may use one replica and relaxed resources; production may use more replicas, stricter disruption budgets, and external endpoints. Those are legitimate differences. Drift appears when the chart version, image, template behavior, or operational policy changes in one environment without being represented as a reviewed promotion.

Example 1: A Reviewed Upgrade

Start with a small chart whose image and replica count come from values. The chart below is intentionally ordinary: a Deployment template consumes .Values.image.repository, .Values.image.tag, and .Values.replicaCount. During rendering, Helm substitutes those values and emits a Kubernetes Deployment.

apiVersion: v2
name: catalogue
version: 0.3.0
appVersion: "1.4.2"
type: application
replicaCount: 2
image:
  repository: registry.example.com/catalogue
  tag: "1.4.2"
resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    memory: 256Mi
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "catalogue.fullname" . }}
  labels:
    app.kubernetes.io/name: {{ include "catalogue.name" . }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app.kubernetes.io/name: {{ include "catalogue.name" . }}
  template:
    metadata:
      labels:
        app.kubernetes.io/name: {{ include "catalogue.name" . }}
    spec:
      containers:
        - name: catalogue
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          ports:
            - containerPort: 8080
          resources:
{{ toYaml .Values.resources | indent 12 }}

A disciplined upgrade renders and reviews before it mutates the cluster. The deterministic part is local rendering: with the values shown, the container image in the manifest is registry.example.com/catalogue:1.4.2 and replicas is 2. The live upgrade then asks Kubernetes to converge on that manifest.

helm lint ./charts/catalogue
helm template catalogue ./charts/catalogue -f values/staging.yaml
helm upgrade --install catalogue ./charts/catalogue \
  --namespace staging --create-namespace \
  -f values/staging.yaml \
  --wait --timeout 5m --history-max 20

Expected behavior: lint catches chart-structure and template problems; template output can be diffed in review; upgrade creates a new revision for catalogue. If the Deployment reaches its availability condition before the timeout, Helm marks the release deployed. If Pods cannot start, the release can be left failed unless --atomic is used.

Example 2: Rollback After a Bad Image

Rollback is not magic undo for the entire cluster. Helm selects a previous release revision, reuses the chart and values stored in that revision, renders the old manifest, and applies it as a new revision. Revision numbers keep increasing. If revision 8 is bad and you roll back to revision 7, Helm creates revision 9 whose manifest is based on revision 7.

helm history catalogue --namespace staging
helm rollback catalogue 7 --namespace staging --wait --timeout 5m
kubectl rollout status deployment/catalogue --namespace staging

Expected behavior: history shows deployed, superseded, or failed revisions; rollback applies the older desired state; rollout status reports success only when the Deployment controller observes enough available Pods. A rollback will not reverse external database migrations, messages already published, or data deleted by a Job. For that reason, charts that include migrations need a forward-compatible schema plan, not only a Helm rollback plan.

Example 3: Promotion with Environment Values

Promotion becomes reviewable when each environment references the same chart version and changes only the intended values. Here development and production share image 1.4.2, but production increases replicas and resources. The expected output is a Deployment with different capacity, not different application code.

replicaCount: 1
image:
  repository: registry.example.com/catalogue
  tag: "1.4.2"
service:
  type: ClusterIP
replicaCount: 4
image:
  repository: registry.example.com/catalogue
  tag: "1.4.2"
service:
  type: ClusterIP
podDisruptionBudget:
  minAvailable: 3

A GitOps repository might store these files under environments/dev/catalogue-values.yaml and environments/prod/catalogue-values.yaml. Promoting from staging to production becomes a pull request that changes the chart version or image tag in the production path after staging has passed verification. The controller then reconciles production from Git. Operators diagnose desired state by reading Git first, then the controller status, then Kubernetes objects.

Design Choices and Trade-offs

Choose imperative Helm from CI when you want the pipeline to be the release actor and you can protect cluster credentials tightly. Choose GitOps when you want all cluster changes to flow through pull requests and an in-cluster reconciler. GitOps improves auditability, but it adds another controller whose sync policy, pruning behavior, and secret access must be understood.

Choose --atomic for workloads where a failed rollout should automatically return to the previous manifest. Avoid assuming it solves every failure: a chart hook that mutates external state, a non-idempotent migration, or an overloaded dependency may still need human judgment. Use immutable image tags or digests for promotion. Reusing latest means the same Git commit can produce different Pods at different times, which breaks rollback analysis.

Keep values layered but not scattered. A common base plus one environment file is easy to review. Many small override files can hide effective configuration, especially when the same key appears in several places. Secrets should come from a secret manager, sealed secret workflow, or external secret controller rather than plain values files.

Failure Modes and Troubleshooting

Symptom: Helm upgrade succeeds, but users see errors. Cause: the command did not wait for rollout, or readiness probes accepted traffic before the application was truly ready. Diagnose with helm status, kubectl rollout status, kubectl describe pod, and recent application logs. Correct by adding meaningful readiness probes and using --wait with a timeout that matches real startup behavior.

Symptom: upgrade fails with an ownership error. Cause: the chart is trying to manage an object that already exists without Helm release annotations for this release and namespace. Diagnose by inspecting the object’s labels and annotations. Correct by importing ownership deliberately only when appropriate, renaming the object, or separating shared infrastructure from the application chart.

Symptom: GitOps keeps reverting a manual hotfix. Cause: the controller is reconciling the Git version as designed. Diagnose by checking the application sync status and comparing live object fields with Git. Correct by committing the hotfix to the environment path or pausing sync according to your controller’s runbook before making emergency changes.

Symptom: rollback completes, but the app still fails. Cause: the earlier manifest cannot run against changed external state, such as a migrated database schema. Diagnose by comparing release history with migration logs and application errors. Correct by using backward-compatible migrations, separating migration approval from app rollout, and testing rollback in a staging environment with realistic data shape.

Security, Reliability, and Performance Implications

The release actor should have the narrowest Kubernetes permissions needed for the namespaces and resources it manages. A GitOps controller with cluster-wide write access is powerful infrastructure, not a convenience account. Protect chart repositories, values repositories, and image registries because each can change production behavior. Sign or verify artifacts when your supply-chain policy requires it, and prefer image digests when exact reproducibility matters.

Reliability comes from making rollouts observable. Track release revision, chart version, image identifier, rollout duration, unavailable replicas, restart count, and error rate. Performance changes should be promoted like code changes: a values-only change to CPU, memory, concurrency, or replica count can alter latency and cost as much as a new image.

Hands-on Lab: Promote and Roll Back a Chart

Prerequisites: a test Kubernetes cluster, Helm, kubectl, and a chart you can safely deploy into a disposable namespace. Do not run this lab against a shared production namespace.

  1. Create or choose a namespace named helm-promotion-lab.
  2. Render the chart with development values and save the output for review.
  3. Install the release with --wait and confirm the Deployment is available.
  4. Change only the image tag or a harmless environment variable in the staging values file, then run helm upgrade.
  5. Inspect helm history and identify the previous deployed revision.
  6. Roll back to that revision and verify that Kubernetes converges.
  7. Clean up the release and namespace when finished.
kubectl create namespace helm-promotion-lab
helm template catalogue ./charts/catalogue -f values/dev.yaml
helm upgrade --install catalogue ./charts/catalogue \
  --namespace helm-promotion-lab -f values/dev.yaml --wait --timeout 5m
kubectl rollout status deployment/catalogue --namespace helm-promotion-lab
helm history catalogue --namespace helm-promotion-lab
helm rollback catalogue 1 --namespace helm-promotion-lab --wait --timeout 5m
helm uninstall catalogue --namespace helm-promotion-lab
kubectl delete namespace helm-promotion-lab

Verification: helm history should show at least one deployed revision before rollback and another revision after rollback. kubectl rollout status should report the Deployment as successfully rolled out. Cleanup removes both the Helm release record and the namespace-scoped Kubernetes objects.

Assessment Exercises

  1. A team uses helm upgrade without --wait and pages only after customers report errors. Design a safer command and name the Kubernetes condition you would verify.
  2. A production hotfix was made with kubectl edit, but the field changed back ten minutes later. Explain the GitOps behavior and write the operational correction.
  3. You need to promote staging to production, but production has four replicas and staging has one. Which fields should be identical, which may differ, and how would you review that difference?
  4. A rollback to the previous Helm revision succeeds, but the application cannot read the database. What does this reveal about the release design, and what migration strategy would reduce the risk?
  5. Compare using mutable image tags with immutable tags or digests for Helm rollbacks. Focus on what evidence is available during an incident.

Summary

Helm upgrades create revisioned release records and submit rendered manifests to Kubernetes. Rollbacks apply an older release state as a new revision, which restores Kubernetes objects but not necessarily external side effects. GitOps moves the release actor into a reconciler that applies reviewed Git state, while environment promotion advances the same chart and application artifact through controlled values differences. Treat every upgrade as a testable change to desired state: render, review, apply, verify, observe, and rehearse rollback before the incident.