Deploy Model Services on Kubernetes

Deploy Model Services on Kubernetes means packaging an inference server as a container, describing how Kubernetes should run it, and exposing it through stable network objects so clients can call the model without knowing which pod currently holds it. In this lesson the outcome is concrete: you should be able to read a model-serving manifest, explain how requests reach a replica, choose rollout and scaling settings, and diagnose the failures that commonly appear when an otherwise valid model is put behind a Kubernetes Service.

In an MLOps system, Kubernetes is not the model registry, training pipeline, or evaluation gate. It is the runtime scheduler and control plane for the serving layer. The model artifact, image tag, environment variables, resource requests, health probes, and routing rules together define the deployed decision service. Treat that deployment as a versioned release of software plus model assets.

How Kubernetes Runs an Inference Service

The smallest useful mental model has four objects. A Deployment declares the desired number of identical pods and the pod template. A Pod runs one or more containers, such as a FastAPI, BentoML, KServe runtime, Triton, or TorchServe process. A Service gives the changing pod set a stable DNS name and virtual IP. An Ingress or gateway accepts external traffic and forwards it to the Service.

The Deployment controller continuously compares desired state with observed state. If a node fails or a pod exits, the controller creates a replacement. The Service does not point at pod names; it selects pods by labels. That is why labels are part of the API contract. A selector such as app: fraud-model and version: v3 decides which pods receive traffic.

Kubernetes also separates scheduling from serving. The scheduler places pods on nodes that can satisfy CPU, memory, accelerator, taint, affinity, and volume requirements. The kubelet on each node pulls the image, starts the container, and runs probes. EndpointSlice objects keep track of ready pod IPs for a Service. Only pods passing readiness checks should receive production requests.

Manifest Anatomy

A model service manifest usually contains a Deployment and a Service. Important Deployment fields include metadata.labels, spec.replicas, spec.selector.matchLabels, spec.template.metadata.labels, container image, ports, env, resources, and probes. The selector and pod labels must match, and a Deployment selector is effectively part of the object identity after creation.

Resource requests reserve schedulable capacity; limits cap runtime usage. For inference, requests should reflect warmed model memory and representative CPU usage, not an idle container. Readiness probes protect clients from cold starts and failed model loads. Liveness probes should be conservative because killing a slow but healthy model server can turn latency into an outage.

Example 1: One HTTP Model Behind a Service

This example serves a classifier image on port 8080. It uses two replicas, a readiness endpoint, and a ClusterIP Service. Inside the cluster, clients call http://sklearn-iris.default.svc.cluster.local/predict or simply http://sklearn-iris/predict from the same namespace. Kubernetes load balances across ready pods.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sklearn-iris
  labels:
    app: sklearn-iris
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sklearn-iris
  template:
    metadata:
      labels:
        app: sklearn-iris
    spec:
      containers:
        - name: server
          image: ghcr.io/example/sklearn-iris:2026-09-06
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            periodSeconds: 5
          resources:
            requests:
              cpu: "250m"
              memory: "512Mi"
            limits:
              cpu: "1"
              memory: "1Gi"
---
apiVersion: v1
kind: Service
metadata:
  name: sklearn-iris
spec:
  selector:
    app: sklearn-iris
  ports:
    - port: 80
      targetPort: 8080

Expected behavior: kubectl get deploy sklearn-iris eventually reports two available replicas, and kubectl get endpoints sklearn-iris shows two pod IPs after both readiness probes pass. If the model takes 30 seconds to load, traffic waits until /ready succeeds rather than hitting a half-initialized process.

Example 2: Call the Service and Interpret the Response

The service contract should be explicit about request shape and response shape. This minimal client sends feature values to the in-cluster DNS name. It also applies a timeout because inference services can saturate under concurrent load.

import json
import urllib.request

payload = json.dumps({"features": [[5.1, 3.5, 1.4, 0.2]]}).encode("utf-8")
request = urllib.request.Request(
    "http://sklearn-iris/predict",
    data=payload,
    headers={"Content-Type": "application/json"},
    method="POST",
)

with urllib.request.urlopen(request, timeout=2) as response:
    body = json.loads(response.read().decode("utf-8"))

print(body["class_name"])

For a deterministic iris demo model trained on the common sample data, the expected class is usually setosa. In a real lesson lab, verify the exact response against the model artifact you built. The important Kubernetes behavior is DNS resolution to the Service, Service forwarding to a ready endpoint, and timeout-bounded client behavior.

Example 3: Rolling Out a New Model Version

A model update should change the pod template, commonly by changing the image tag, model URI, or configuration checksum. The Deployment controller then creates a new ReplicaSet and gradually replaces old pods according to rollout settings.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sklearn-iris
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: sklearn-iris
        model-version: v2
    spec:
      containers:
        - name: server
          image: ghcr.io/example/sklearn-iris:2026-09-06-v2
          env:
            - name: MODEL_VERSION
              value: v2

This fragment is not a complete manifest because it shows only the rollout-specific fields. With maxUnavailable: 0, Kubernetes keeps the old version serving while the new pods become ready. Expected behavior: kubectl rollout status deployment/sklearn-iris blocks until the new ReplicaSet is available, and kubectl rollout undo deployment/sklearn-iris returns to the previous pod template if validation fails.

Example 4: Scaling for Latency

Inference services usually scale on CPU, concurrent requests, queue depth, or custom accelerator metrics. A HorizontalPodAutoscaler changes Deployment replicas; it does not make a single pod faster. Your container must expose enough replicas, readiness, and resource requests for the autoscaler to act sensibly.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: sklearn-iris
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sklearn-iris
  minReplicas: 2
  maxReplicas: 8
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65

Expected behavior: when average CPU across ready pods stays above the target, Kubernetes raises desired replicas up to eight. This helps CPU-bound preprocessing or small models. For GPU models, CPU utilization can be misleading; use runtime metrics such as queue latency, inflight requests, or GPU duty cycle through a metrics adapter.

Design Choices and Trade-offs

Choose a plain Deployment when you own a conventional HTTP or gRPC server and need simple scaling. Choose a model-serving framework such as KServe, Seldon, or BentoML on Kubernetes when you need model-specific abstractions like canary traffic, scale-to-zero, transformer containers, inference graphs, or model repository integration. These frameworks still create Kubernetes objects underneath, so understanding Deployments and Services remains useful.

Embedding the model inside the image makes rollbacks simple because the image digest identifies code and artifact together. Pulling the model from object storage at startup makes images smaller and allows a common server image, but startup now depends on storage credentials, network availability, and artifact integrity checks. A sidecar can handle model downloads or telemetry, but it increases pod complexity and resource accounting.

Use more small replicas for lower blast radius and smoother rolling updates. Use fewer larger replicas when model memory dominates and loading many copies is too expensive. For large language or vision models, consider node selectors, tolerations, and GPU resource requests so the scheduler places pods on nodes with the required accelerator.

Failure Modes and Troubleshooting

Pods stay Pending. Symptom: kubectl get pods shows Pending. Cause: no node can satisfy CPU, memory, GPU, affinity, or volume requirements. Diagnose with kubectl describe pod and read scheduler events. Correct by lowering requests, adding suitable nodes, fixing node labels, or requesting the correct accelerator resource.

Pods restart after loading the model. Symptom: CrashLoopBackOff or repeated restarts after high memory usage. Cause: the process exceeds its memory limit or exits when artifact loading fails. Diagnose previous logs with kubectl logs POD --previous and inspect termination reason. Correct by increasing memory, reducing model footprint, validating the model URI, or failing readiness instead of exiting for temporary dependencies.

Service has no endpoints. Symptom: clients get connection errors, and kubectl get endpoints sklearn-iris is empty. Cause: Service selector does not match pod labels, or readiness never passes. Diagnose labels with kubectl get pods --show-labels and probe failures with kubectl describe pod. Correct the selector, pod labels, readiness path, or server binding address.

Rollout hangs. Symptom: kubectl rollout status never completes. Cause: new pods are not becoming ready, commonly due to a bad image tag, missing secret, schema mismatch, or slow cold start. Diagnose ReplicaSets and pod events. Correct the image or configuration, extend startup expectations with a startup probe, or undo the rollout.

Security, Reliability, and Performance

Run the serving container with a dedicated ServiceAccount and only the permissions it needs. Many inference pods need no Kubernetes API access at all. Store registry credentials, object-store tokens, and database passwords in Secrets or an external secret manager, then mount or inject them at runtime. Never bake credentials into model images.

Reliability depends on probes, disruption settings, and rollback discipline. Use readiness for traffic admission, startup probes for long model initialization, and PodDisruptionBudgets when voluntary maintenance would otherwise evict too many replicas. Pin image digests for audited releases, and record the model version in labels or annotations so logs and metrics can be joined to the deployed artifact.

Performance tuning starts inside the pod. Set worker counts, batch size, model thread pools, and BLAS or runtime environment variables deliberately. Kubernetes can add replicas, but it cannot fix a server that queues every request behind one oversized batch. Measure p50, p95, p99 latency, error rate, saturation, and cold-start time under representative payloads.

Hands-On Lab

Prerequisites: a Kubernetes cluster such as kind, minikube, or a managed development namespace; kubectl configured for that cluster; an inference image that exposes /ready and /predict; permission to create Deployments and Services in a test namespace.

Create a namespace named model-lab and apply the Example 1 manifest there, replacing the image with your test server if needed.
Run kubectl -n model-lab rollout status deployment/sklearn-iris and wait for completion.
Verify endpoints with kubectl -n model-lab get endpoints sklearn-iris. Expect at least one endpoint IP and port.
Start a temporary curl pod in the same namespace and POST a known payload to http://sklearn-iris/predict. Record the response and latency.
Change the image tag or MODEL_VERSION value and apply the rollout fragment as part of the full Deployment. Watch kubectl -n model-lab rollout status deployment/sklearn-iris.
Break readiness intentionally by changing the readiness path to /not-ready. Confirm that the new pods do not enter endpoints and that rollout does not complete.
Rollback with kubectl -n model-lab rollout undo deployment/sklearn-iris and verify endpoints return.
Cleanup with kubectl delete namespace model-lab when the lab is finished.

Assessment Exercises

A model server loads successfully but every client request times out during deploy. Which Kubernetes objects and events would you inspect first, and what evidence would distinguish routing failure from slow inference?
Your team wants to pull the model from object storage at pod startup instead of baking it into the image. List two operational benefits and three new failure cases you must test.
Given a Service selector app: risk-score and pod labels app: risk-score, model-version: v4, design a blue-green routing approach that avoids sending traffic to both versions accidentally.
An HPA based on CPU does not scale a GPU-backed model even though users see high latency. Explain why and propose a better metric.
Write a rollback criterion for a model rollout that includes both service health and model output behavior.

Summary

Kubernetes deploys model services by reconciling desired pod replicas, routing Services to ready endpoints, and rolling pod templates forward or back. The MLOps work is to bind that machinery to model identity, resource reality, request contracts, rollout evidence, and fast recovery. A good deployment manifest makes the model reachable, observable, scalable, and reversible without hiding how traffic actually reaches the code that performs inference.