Pods and Container Lifecycle

A Kubernetes Pod is the smallest workload unit the scheduler places on a node. It is not just a wrapper around a container. A Pod gives one or more containers a shared network identity, shared volumes, common scheduling fate, and a lifecycle managed by the kubelet on the selected node. In a Helm course, this matters because most chart templates eventually render Pods indirectly through Deployments, StatefulSets, Jobs, or DaemonSets. Good Helm values are easier to design when you understand what Kubernetes will actually do after the manifest is accepted.

By the end of this lesson, you should be able to predict how init containers, app containers, probes, hooks, restart policy, graceful termination, and controller replacement interact. The practical outcome is simple: you can write a chart whose Pods start only after prerequisites are prepared, receive traffic only when ready, shut down without dropping avoidable requests, and expose enough status to diagnose failures.

Pod Lifecycle In Plain Language

A Pod begins as an API object. The scheduler assigns it to a node. The kubelet on that node creates its sandbox, attaches volumes, runs init containers in order, starts app containers, reports conditions, restarts containers when policy allows it, and eventually terminates the Pod. Controllers such as Deployments do not repair a container directly; they maintain a desired number of Pods by creating replacements when Pods disappear or become superseded by a rollout.

The main Pod phases are Pending, Running, Succeeded, Failed, and Unknown. These phases are deliberately coarse. Most useful troubleshooting comes from container states, Pod conditions, events, and probe results. A Pod can be Running while its container is not ready, restarting repeatedly, or blocked by an image pull error.

How Kubernetes Runs A Pod

After scheduling, the kubelet asks the container runtime to create the Pod sandbox. All containers in the Pod share the same Pod IP and can reach each other through localhost. Volumes declared at Pod scope are mounted into whichever containers request them. This is why a sidecar can write a generated file into an emptyDir volume and the main application can read it without crossing a Service boundary.

Init containers run before normal containers. Each init container must complete successfully before the next one starts. They are useful for deterministic setup, such as rendering configuration from a mounted Secret or checking that a schema migration Job has finished. They are a poor fit for long-running helpers because they block the Pod from ever becoming available.

App containers then start together, unless their own startup is delayed by image pulls or runtime errors. Kubernetes tracks each container as Waiting, Running, or Terminated. A terminated container has an exit code and reason. With restartPolicy: Always, common in Deployments, the kubelet restarts failed app containers with backoff. Init containers are retried until success unless the Pod restart policy prevents that.

Configuration Anatomy

The lifecycle-related fields live under spec and each container entry. At Pod level, restartPolicy controls whether containers are restarted after exit. terminationGracePeriodSeconds sets the time Kubernetes gives processes to stop after sending SIGTERM before it sends SIGKILL. volumes define shared storage. initContainers define ordered setup tasks. Under each app container, readinessProbe controls endpoint membership, livenessProbe tells Kubernetes when to restart a stuck container, and startupProbe protects slow-starting applications from premature liveness failures. The lifecycle field supports hooks such as postStart and preStop, but hooks are not a substitute for application signal handling.

Example 1: Init Container And Shared Volume

This Pod writes a file from an init container, then reads it from the app container. The expected log line is prepared. The Pod may not become Ready because the BusyBox process exits after sleeping and there is no readiness probe; that is acceptable for demonstrating ordered startup.

apiVersion: v1
kind: Pod
metadata:
  name: lifecycle-basic
  labels:
    app.kubernetes.io/name: lifecycle-basic
spec:
  restartPolicy: Never
  initContainers:
    - name: prepare
      image: busybox:1.36
      command: ["sh", "-c", "echo prepared > /work/status"]
      volumeMounts:
        - name: work
          mountPath: /work
  containers:
    - name: app
      image: busybox:1.36
      command: ["sh", "-c", "cat /work/status; sleep 20"]
      volumeMounts:
        - name: work
          mountPath: /work
  volumes:
    - name: work
      emptyDir: {}

The mechanism is the important part: prepare must finish before app starts. If prepare exits non-zero, the app container never starts. In a Helm chart, this pattern should be parameterized carefully. For example, make setup commands explicit values only for trusted chart users, because templating arbitrary shell commands into Pods is a privilege boundary decision.

Example 2: Readiness, Liveness, And Graceful Stop

This Deployment creates two nginx Pods. The readiness probe checks whether each Pod should receive Service traffic. The liveness probe checks whether the kubelet should restart the container. The preStop hook sleeps briefly, giving endpoint updates time to propagate before the process exits.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: lifecycle-web
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: lifecycle-web
  template:
    metadata:
      labels:
        app.kubernetes.io/name: lifecycle-web
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: web
          image: nginx:1.27-alpine
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: 80
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 10
            periodSeconds: 10
          lifecycle:
            preStop:
              exec:
                command: ["sh", "-c", "sleep 5"]

When applied, you should expect two desired replicas and, after image pull and startup, two ready Pods. During deletion or rolling update, Kubernetes marks a Pod for termination, runs preStop, sends SIGTERM, waits up to the grace period, and removes the Pod from active use. The trade-off is latency during rollout: a longer grace period protects in-flight work but slows replacement and node drain.

Example 3: Helm Template For Lifecycle Defaults

A chart usually renders Pod lifecycle settings inside a controller template rather than a raw Pod. This fragment shows where values normally enter the Pod template. It is marked as a fragment because Helm functions and values require a chart context.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "demo.fullname" . }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app.kubernetes.io/name: {{ include "demo.name" . }}
  template:
    metadata:
      labels:
        app.kubernetes.io/name: {{ include "demo.name" . }}
    spec:
      containers:
        - name: web
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          readinessProbe:
            httpGet:
              path: {{ .Values.probes.readinessPath | quote }}
              port: 80

The key design choice is which fields chart consumers may change. Image repository and tag are usually values. Probe paths often should be values because applications expose different health endpoints. Selector labels should be stable helper output, not casual values, because changing a Deployment selector can make upgrades fail or orphan existing Pods.

Design Choices And Trade-Offs

Use a single container when the application can own its setup, health reporting, logging, and shutdown. Add an init container when preparation must finish before startup and can be retried safely. Add a sidecar only when a separate long-running process shares fate with the app, such as a local proxy or log shipper. Sidecars increase resource requests and make termination ordering more important.

Readiness probes should answer whether the Pod can serve real traffic now. They should fail during dependency warm-up, cache loading, or graceful shutdown. Liveness probes should answer whether restarting this container is likely to repair it. A liveness probe that checks a downstream database can cause restart storms during a database outage. Startup probes are best for applications that need a long initial load but should be held to stricter liveness checks afterward.

For Helm charts, avoid hiding lifecycle behavior behind surprising defaults. A chart that silently adds aggressive liveness probes may make a stable but slow application crash-loop. A chart that omits resource requests leaves scheduling less predictable because the scheduler lacks CPU and memory intent. Values should make the common path easy while still rendering explicit Kubernetes objects that operators can inspect with helm template.

Failure Modes And Troubleshooting

ImagePullBackOff: the Pod stays Pending or containers show Waiting with image pull errors. The cause is usually a wrong image name, missing tag, private registry authentication failure, or node network problem. Diagnose with kubectl describe pod and read the Events section. Correct the image reference, add the needed image pull Secret, or fix registry access.

CrashLoopBackOff: the container starts and then exits repeatedly. The cause may be a bad command, missing configuration, failed dependency, or application panic. Diagnose with kubectl logs --previous, inspect exit codes in kubectl describe pod, and compare rendered Helm values with expected environment variables and mounts. Correct the command, configuration, Secret, or dependency assumption.

Pod Running But Not Ready: the process is alive but readiness fails. Symptoms include a Deployment with available replicas lower than desired and a Service that does not send traffic to the new Pods. Diagnose the readiness endpoint from inside the Pod if possible, inspect probe path and port, and check application startup logs. Correct the probe to match the real serving endpoint or fix the app so it reports readiness only after initialization.

Slow Or Stuck Termination: Pods remain in Terminating. Common causes are long grace periods, processes ignoring SIGTERM, blocking preStop hooks, or finalizers on related objects. Diagnose timestamps in kubectl describe pod and application shutdown logs. Correct signal handling, shorten hook work, or move cleanup into a controller or Job designed for retry.

Security, Performance, And Reliability

Lifecycle settings affect more than availability. Init containers and hooks run with the permissions you grant them, so avoid broad ServiceAccounts or shell snippets that can read every mounted Secret. Probes create ongoing traffic; set periods and timeouts that detect failures without adding material load. Resource requests influence scheduling and eviction behavior. A Pod with no memory limit can pressure a node; a Pod with a limit below normal startup memory can be killed before it ever becomes ready.

Reliability comes from aligning Kubernetes behavior with application behavior. Applications should handle SIGTERM, stop accepting new work, finish or checkpoint current work, and exit before the grace period expires. Controllers should use enough replicas and a rollout strategy that keeps capacity during updates. Helm releases should be reviewed as rendered manifests so probe, hook, and resource changes are visible before upgrade.

Hands-On Lab

Prerequisites: a Kubernetes cluster reachable with kubectl, permission to create a namespace, and a local file named lifecycle-basic.yaml containing the first example. The lab uses public BusyBox images and creates only temporary namespace-scoped objects.

kubectl create namespace lifecycle-lab
kubectl apply -n lifecycle-lab -f lifecycle-basic.yaml
kubectl wait -n lifecycle-lab --for=condition=Ready pod/lifecycle-basic --timeout=30s || true
kubectl logs -n lifecycle-lab lifecycle-basic
kubectl describe pod -n lifecycle-lab lifecycle-basic
kubectl delete namespace lifecycle-lab

Step 1 creates an isolated namespace. Step 2 applies the Pod. Step 3 intentionally allows the wait command to fail because this short-lived Pod is not a web server with a readiness endpoint. Step 4 verifies deterministic behavior: logs should include prepared. Step 5 shows events, init container status, app container status, and exit information. Step 6 removes the namespace and all lab objects.

For deeper verification, reapply the manifest after changing the init command to exit 1. The expected behavior is that the app container never prints prepared, and the init container status shows repeated failure or a failed Pod depending on policy. Roll back by restoring the original command and recreating the Pod, because most Pod spec fields are immutable after creation.

Assessment Exercises

  1. A Deployment has Pods in Running, but the Service has no endpoints. Identify two likely lifecycle causes and the commands you would run first.
  2. Design probes for an application that takes two minutes to load a model and then serves HTTP traffic. Explain which probe should tolerate the slow start and which should be strict afterward.
  3. A chart exposes deployment.selectorLabels as user-editable values. Explain why that is risky during upgrades and propose a safer values interface.
  4. A container must flush a queue before shutdown. Describe the Kubernetes termination sequence and what the application, readiness probe, and grace period each need to do.
  5. Given a CrashLoopBackOff after a Helm upgrade, list the evidence needed to decide whether to roll back values, change the image, or fix a missing Secret.

Summary

Pods combine scheduling, shared runtime context, and container lifecycle into the unit Kubernetes can place and replace. Init containers gate startup, probes control traffic and restarts, hooks and grace periods shape shutdown, and controllers create replacement Pods rather than repairing old ones in place. In Helm-managed workloads, lifecycle fields should be explicit, reviewable, and parameterized only where chart users need real control.