Requests, Limits, Affinity, Taints, and Topology

Kubernetes does not place Pods at random. The scheduler receives a Pod specification, compares it with the current cluster state, filters impossible nodes, scores acceptable nodes, and binds the Pod to one node. Requests, limits, affinity, taints, tolerations, and topology spread constraints are the main levers you give that scheduler and the kubelet. In a Helm chart, these fields become reusable placement policy: the chart author exposes safe defaults, and the operator adjusts them for a cluster without rewriting the workload.

The practical outcome is predictable scheduling. A Pod should request enough CPU and memory to reserve capacity, avoid nodes that cannot run it, tolerate only the taints it is meant to tolerate, spread replicas across failure domains, and fail visibly when the cluster cannot satisfy those promises. That is different from merely making a manifest valid; a valid manifest can still overcommit a node, concentrate all replicas in one zone, or let an ordinary web Pod land on a node dedicated to a stateful database.

Scheduler and Kubelet Mechanics

Requests are scheduling inputs. If a container requests 500 millicores of CPU and 1 GiB of memory, the scheduler subtracts those amounts from each candidate node’s allocatable capacity when deciding whether the Pod fits. CPU is compressible: a busy container can be throttled when it reaches its limit. Memory is not compressible in the same way: if a container exceeds its memory limit, the kernel can terminate it and Kubernetes reports an OOMKilled container state.

Limits are enforcement inputs for the runtime and kubelet. A CPU limit creates a ceiling and may introduce throttling even when spare CPU exists for short bursts. A memory limit creates a hard boundary. A Pod with requests but no CPU limit is common for latency-sensitive services because it reserves capacity while allowing opportunistic CPU bursts; a Pod with no memory limit can consume enough memory to harm node stability, so memory limits are usually more important.

Affinity changes where a Pod may or should run. Node affinity matches labels on nodes. Pod affinity and anti-affinity match labels on already running Pods and compare a topology key such as kubernetes.io/hostname or topology.kubernetes.io/zone. Required rules are filters: if no node satisfies them, the Pod remains Pending. Preferred rules are scores: the scheduler tries to honor them but can choose another node.

Taints repel Pods from nodes. A node tainted with dedicated=payments:NoSchedule refuses Pods unless they carry a matching toleration. Tolerations do not attract Pods; they only allow scheduling onto a tainted node. To intentionally use dedicated nodes, combine a toleration with node affinity or a node selector. The NoExecute effect can also evict running Pods that do not tolerate the taint.

Topology spread constraints ask the scheduler to keep matching Pods balanced across a topology domain. The scheduler counts existing matching Pods in each domain, calculates skew, and checks whether placing the new Pod would exceed maxSkew. With DoNotSchedule, the Pod stays Pending rather than making the distribution worse. With ScheduleAnyway, the scheduler prefers balance but can violate it to keep work moving.

Manifest and Helm Anatomy

The resource fields live under each container at spec.template.spec.containers[].resources. Placement fields live at Pod spec level: affinity, nodeSelector, tolerations, and topologySpreadConstraints. This distinction matters in Helm templates. Container resources often vary by component, while placement policy usually applies to the whole Pod. A good chart keeps these values explicit in values.yaml and renders them without hiding strong scheduling rules behind surprising defaults.

Use required rules for constraints that are objectively necessary, such as GPU nodes for a GPU workload or architecture labels for a native binary. Use preferred rules for resilience or cost preferences that should not block deployments during maintenance. Keep label keys stable and owned. Cloud provider labels, custom node pool labels, and zone labels can all work, but the chart should document which labels the cluster operator must maintain.

Example 1: Reserve and Bound a Service

This Deployment reserves 250 millicores and 256 MiB for each replica. Two replicas therefore require 500 millicores and 512 MiB of allocatable cluster capacity before overhead. The CPU limit allows each replica to use up to one core, while the memory limit caps it at 512 MiB. Expected behavior: if nodes have enough unreserved capacity, both Pods schedule; if a container grows beyond 512 MiB, the container can be terminated with reason OOMKilled.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: checkout
  template:
    metadata:
      labels:
        app.kubernetes.io/name: checkout
    spec:
      containers:
        - name: app
          image: example.invalid/checkout:1.0.0
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 512Mi

The trade-off is burst control versus latency. The CPU limit protects neighbors from sustained CPU use, but it can throttle a service during legitimate bursts. The memory limit is less optional: without it, a leak can pressure the whole node instead of being isolated to the container.

Example 2: Require a Node Class and Prefer Separation

The reporting workload below must run on nodes labelled node.kubernetes.io/instance-type=compute-large. It also prefers not to place two reporting Pods on the same hostname. Expected behavior: on a three-node compute pool, three replicas normally land one per node. If only one matching node exists, all replicas may still schedule there because the anti-affinity is preferred, not required.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: reporting
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: reporting
  template:
    metadata:
      labels:
        app.kubernetes.io/name: reporting
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node.kubernetes.io/instance-type
                    operator: In
                    values:
                      - compute-large
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 80
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: reporting
                topologyKey: kubernetes.io/hostname
      containers:
        - name: app
          image: example.invalid/reporting:1.0.0
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              memory: 2Gi

This pattern is useful when a service benefits from a node class but should degrade gracefully if the cluster is temporarily small. Making the anti-affinity required would improve separation but could block a rollout when a node is drained. The scheduler’s wording is literal: requiredDuringSchedulingIgnoredDuringExecution is checked when placing the Pod, and the Pod is not evicted later just because labels or neighbors change.

Example 3: Use Dedicated Nodes and Zone Spread

The payments workload is allowed onto nodes tainted for payments, requires the matching node label, and spreads replicas across zones with maximum skew of one. Expected behavior: with four replicas and two eligible zones, the scheduler aims for a 2 and 2 distribution. With three eligible zones, the distribution can be 2, 1, and 1. If only one eligible zone is available and existing matching Pods make the skew too high, new Pods remain Pending because whenUnsatisfiable is DoNotSchedule.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments
spec:
  replicas: 4
  selector:
    matchLabels:
      app.kubernetes.io/name: payments
  template:
    metadata:
      labels:
        app.kubernetes.io/name: payments
    spec:
      tolerations:
        - key: dedicated
          operator: Equal
          value: payments
          effect: NoSchedule
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: payments
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: dedicated
                    operator: In
                    values:
                      - payments
      containers:
        - name: app
          image: example.invalid/payments:1.0.0
          resources:
            requests:
              cpu: 300m
              memory: 384Mi
            limits:
              memory: 768Mi

The important detail is that toleration and attraction are separate. The toleration passes the taint check; node affinity ensures the Pod actually targets the dedicated pool. The topology rule then balances among the eligible nodes, not across every node in the cluster.

Helm Values Shape

A chart can expose the same policy as values. Keep the values close to Kubernetes field names so operators can transfer knowledge from native manifests to Helm. Avoid inventing one Boolean such as productionScheduling: true; it hides decisions that need different answers for development, batch, web, and regulated workloads.

resources:
  requests:
    cpu: 300m
    memory: 384Mi
  limits:
    memory: 768Mi
nodeSelector:
  dedicated: payments
tolerations:
  - key: dedicated
    operator: Equal
    value: payments
    effect: NoSchedule
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule

In templates, render these maps only when provided, and use schema validation where possible to prevent misspelled keys from silently producing weak scheduling policy. A values schema cannot prove every node label exists, but it can catch type errors such as a string where a list of tolerations is required.

Design Choices and Trade-offs

Requests should reflect measured steady-state need plus reasonable headroom. Too-low requests make nodes appear emptier than they are and invite eviction or throttling under load. Too-high requests waste capacity and can leave Pods Pending even when the cluster has unused resources. Limits should reflect failure containment. CPU limits are sometimes omitted for interactive services to avoid throttling artifacts, while memory limits are commonly set because memory exhaustion has harsher node-level consequences.

Prefer soft placement for goals that improve availability but should not block urgent fixes. Prefer hard placement for hardware, compliance, licensing, or data-locality requirements. Topology spread is usually clearer than hand-written anti-affinity when the intent is even distribution across zones. Anti-affinity is still useful when the exact relationship between workloads matters, such as separating replicas of the same cache shard.

Failure Modes and Troubleshooting

  • Pending Pods with FailedScheduling events: symptoms include Pods stuck in Pending and events saying insufficient cpu, insufficient memory, unmatched node affinity, or untolerated taint. The cause is that scheduler filters removed every node. Diagnose with kubectl describe pod, then compare requests, node labels, taints, and allocatable capacity. Correct by lowering unrealistic requests, adding capacity, fixing labels, or changing hard rules to preferred rules when they are not mandatory.
  • OOMKilled containers: symptoms include restarts, exit code 137, and a last state reason of OOMKilled. The cause is memory use exceeding the container limit. Diagnose with container metrics and recent changes in traffic or cache size. Correct by fixing the memory growth, raising the limit with matching request adjustments, or splitting work into smaller Pods.
  • Unexpected placement on dedicated nodes: symptoms include general workloads running on nodes intended for a special pool. The cause is usually a missing taint, a broad toleration, or no node affinity on the dedicated workload. Diagnose node taints with kubectl describe node and inspect rendered Pod tolerations. Correct by applying precise taints and removing catch-all tolerations from charts.
  • Topology spread blocks rollouts: symptoms include one or more new replicas Pending during a zone outage or node drain. The cause is a strict DoNotSchedule spread rule that cannot maintain skew. Diagnose events and count matching Pods per topology domain. Correct by restoring capacity, temporarily using ScheduleAnyway, reducing replicas, or selecting a topology key that matches available infrastructure.

Security, Performance, and Reliability

Scheduling policy is part of workload isolation. Taints and affinity can keep sensitive or noisy workloads on known node pools, but they are not a substitute for runtime isolation, network policy, or access control. Anyone allowed to create Pods with arbitrary tolerations may bypass node dedication, so admission policy should restrict high-risk tolerations and node selectors in shared clusters.

Performance depends on honest resource requests and careful CPU limits. Under-requested services compete unpredictably; over-limited services show throttling even when average CPU appears low. Reliability improves when topology rules match real failure domains, such as zones or hosts, and when hard constraints are reserved for conditions the cluster can normally satisfy.

Hands-on Lab

Prerequisites: a test Kubernetes cluster, Helm, kubectl access to a namespace you can modify, and a chart containing a Deployment template that accepts resources, tolerations, node selection, and topology spread values. Do not run this against a production namespace unless the chart, labels, and taints are already approved.

  1. Create a values file named values-scheduling.yaml using the Helm values shape above.
  2. Render the chart and inspect the Deployment. Verify that resources appear under the container and placement fields appear under the Pod spec.
  3. Run server-side dry run before applying. This catches schema errors against the cluster API.
  4. Apply the rendered manifest in the test namespace.
  5. Verify placement with kubectl get pods -o wide and inspect scheduler events with kubectl describe pod.
  6. Change whenUnsatisfiable from DoNotSchedule to ScheduleAnyway, render again, and compare the diff. The expected difference is only the spread rule behavior, not container resources or selectors.
  7. Cleanup by deleting the rendered manifest or uninstalling the Helm release.
helm template scheduling-demo ./chart --values values-scheduling.yaml > rendered.yaml
kubectl apply --dry-run=server -f rendered.yaml
kubectl get pods -l app.kubernetes.io/name=payments -o wide
kubectl describe pod -l app.kubernetes.io/name=payments
kubectl delete -f rendered.yaml

Verification is successful when the rendered YAML contains the intended fields, the dry run succeeds, scheduled Pods appear only on eligible nodes, and describe output has no unresolved FailedScheduling events. If your cluster lacks matching labels or taints, keep the exercise at dry-run and template-inspection level, or adjust the labels to match a disposable node pool.

Assessment Exercises

  1. A service has a 200 millicore request, no CPU limit, and a 512 MiB memory limit. Explain how the scheduler and kubelet each use those values, and name one benefit and one risk.
  2. You add a toleration for dedicated=analytics:NoSchedule, but Pods still land on ordinary nodes. What Kubernetes field would you add, and why?
  3. A three-replica Deployment uses required pod anti-affinity on hostname, and the cluster temporarily has two eligible nodes. Predict the rollout behavior and propose a less blocking alternative.
  4. A chart exposes resources but not topologySpreadConstraints. Design a values interface that preserves Kubernetes terminology while allowing a team to spread replicas by zone.
  5. During a zone outage, strict topology spread leaves new Pods Pending. List the diagnostic command, the likely event category, and two corrections with different trade-offs.

Summary

Requests reserve capacity for scheduling; limits bound runtime consumption; affinity selects or prefers locations; taints repel Pods unless tolerated; topology spread balances replicas across failure domains. In a Kubernetes and Helm workflow, the goal is to encode these choices as clear chart values, render them predictably, and verify them with scheduler events and real placement. Strong rules protect correctness, but every hard rule can also block a rollout when the cluster cannot satisfy it.