Health Checks, Smoke Tests, and Rollbacks

Shipping a new version is only half the job. A deployment step that exits 0 tells you the deploy command ran, not that the application is actually serving traffic correctly. This lesson covers the three checks that close that gap: health checks that confirm the process is alive and ready, smoke tests that confirm the critical paths still work, and rollback strategies that get you back to a known-good state quickly when either of those checks fails.

Overview / How it works

A health check answers a narrow question: is the running instance responding at all? It usually hits a lightweight endpoint (/healthz, /ready) that checks the process is up, dependencies like the database are reachable, and the app isn’t still starting up. A smoke test answers a broader question: do the things users actually depend on still work? That might mean logging in, creating a record, or hitting the three most important API routes. Smoke tests are not a replacement for your full test suite — they run against the live deployed environment, so they need to be fast, idempotent, and safe to run against production.

Rollback is the safety net when either check fails. In continuous delivery, a human decides when to promote after checks pass. In continuous deployment, the pipeline promotes automatically, which makes automated rollback essential — there’s no human in the loop to notice a broken release before customers do. The goal is to minimize mean time to recovery (MTTR): the faster your pipeline can detect a bad deploy and revert it, the smaller the blast radius.

Syntax or workflow structure

A deploy-then-verify-then-rollback pipeline is usually modeled as separate jobs connected with needs, so each stage’s success or failure is explicit and visible in the run summary:

  • A deploy job that ships the new artifact and records what was running before, so there’s something concrete to roll back to.
  • A health check step (often inside the deploy job) that polls the new deployment with retries and a timeout, rather than checking once.
  • A smoke-test job that runs after health checks pass, using needs: deploy so it only runs against a service that’s already confirmed to be up.
  • A rollback job guarded by if: failure(), so it only runs when an earlier job in the chain failed.

This lesson’s examples use minimal permissions — contents: read to check out the repo, and deployments: write only when a job needs to update GitHub’s Deployment API. Broader permissions like write access to packages or issues aren’t needed for verification or rollback and should stay out of the token entirely.

Examples

Example 1: A deploy with a retrying health check. This is the minimum viable safety net — deploy, then wait for the service to actually respond before declaring success.

name: Deploy and Health Check

on:
  push:
    branches: [main]

permissions:
  contents: read
  deployments: write

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - name: Deploy new version
        run: ./scripts/deploy.sh "ghcr.io/example/app@${{ needs.build.outputs.digest }}"

      - name: Wait for service to become healthy
        run: ./scripts/health-check.sh https://api.example.com/healthz

Expected behavior: the deploy step ships the new image, then health-check.sh polls the endpoint every few seconds until it returns HTTP 200 or the retry budget runs out, at which point the job fails loudly instead of reporting a false success.

Example 2: Adding a smoke-test job. Health checks confirm the process is alive; smoke tests confirm the product works. They run as a separate job so their logs and pass/fail status are distinct from the deploy step.

name: Deploy, Health Check, and Smoke Test

on:
  push:
    branches: [main]

permissions:
  contents: read
  deployments: write

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - name: Deploy new version
        run: ./scripts/deploy.sh "ghcr.io/example/app@${{ needs.build.outputs.digest }}"

      - name: Wait for service to become healthy
        run: ./scripts/health-check.sh https://api.example.com/healthz

  smoke-test:
    needs: deploy
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository
        uses: actions/checkout@v4

      - name: Run critical-path smoke tests
        run: ./scripts/smoke-test.sh https://api.example.com

Expected behavior: smoke-test only starts once deploy succeeds. smoke-test.sh exercises a handful of real user flows against the live URL — for example, creating and then deleting a throwaway resource — and exits non-zero if any of them fail, without needing to know about the underlying image digest.

Example 3: Automatic rollback on failure. This is the full picture: record what’s currently running before deploying, and if health checks or smoke tests fail, redeploy that recorded version automatically.

name: Deploy with Automatic Rollback

on:
  push:
    branches: [main]

permissions:
  contents: read
  deployments: write

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    outputs:
      previous_digest: ${{ steps.record.outputs.previous_digest }}
    steps:
      - name: Record currently running image digest
        id: record
        run: |
          current=$(./scripts/current-digest.sh)
          echo "previous_digest=$current" >> "$GITHUB_OUTPUT"

      - name: Deploy new image
        run: ./scripts/deploy.sh "ghcr.io/example/app@${{ needs.build.outputs.digest }}"

      - name: Wait for service to become healthy
        run: ./scripts/health-check.sh https://api.example.com/healthz

      - name: Run smoke tests
        run: ./scripts/smoke-test.sh https://api.example.com

  rollback:
    needs: deploy
    if: failure()
    runs-on: ubuntu-latest
    environment: production
    steps:
      - name: Roll back to previous image digest
        run: ./scripts/deploy.sh "ghcr.io/example/app@${{ needs.deploy.outputs.previous_digest }}"

      - name: Verify rollback health
        run: ./scripts/health-check.sh https://api.example.com/healthz

Expected behavior: if the health check or smoke-test step fails, the rest of the deploy job stops, but the previous_digest output — set earlier by the step that already succeeded — is still available. The rollback job then redeploys that exact digest and re-checks health, so the pipeline actively recovers instead of just alerting someone.

Not every failure should trigger an automatic rollback without a human ever knowing. Pair this with a notification step (Slack, email, an issue comment) so the team is aware a rollback happened, even though the pipeline handled the immediate recovery.

Step by step

  1. Before deploying, record the identifier of whatever is currently live — ideally an immutable image digest, not a tag.
  2. Deploy the new artifact to the target environment.
  3. Poll a lightweight health endpoint with retries and a timeout, not a single check, since services need time to start.
  4. Once healthy, run a small set of smoke tests against real critical paths, not just the homepage.
  5. If any step fails, run a rollback job gated with if: failure() that redeploys the recorded previous digest.
  6. Re-run the health check after rollback to confirm the previous version is actually serving traffic again.
  7. Also provide a manual rollback entry point for cases the pipeline can’t detect on its own, such as a slow data-correctness bug reported after the fact.

A manual rollback workflow gives an operator a controlled way to revert without needing repository write access or direct server access:

name: Manual Rollback

on:
  workflow_dispatch:
    inputs:
      image_digest:
        description: "Digest of the image to roll back to, e.g. sha256:abc123..."
        required: true
        type: string

permissions:
  contents: read
  deployments: write

jobs:
  rollback:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - name: Roll back to specified digest
        run: ./scripts/deploy.sh "ghcr.io/example/app@${{ inputs.image_digest }}"

      - name: Verify rollback health
        run: ./scripts/health-check.sh https://api.example.com/healthz

Because this targets the production environment, any required reviewers or wait timers configured on that environment still apply, so a manual rollback can’t bypass the same protection rules that guard a normal deploy.

Here is a working health-check.sh that both examples above call. It retries with a fixed delay and fails closed if the service never becomes healthy:

#!/usr/bin/env bash
set -euo pipefail

url="$1"
max_attempts=10
delay=10

for attempt in $(seq 1 "$max_attempts"); do
  status=$(curl -s -o /dev/null -w "%{http_code}" "$url")
  if [ "$status" = "200" ]; then
    echo "Healthy after $attempt attempt(s)"
    exit 0
  fi
  echo "Attempt $attempt/$max_attempts returned $status, retrying in ${delay}s"
  sleep "$delay"
done

echo "Service did not become healthy in time"
exit 1

Common Mistakes

Mistake 1: Checking health once, immediately after the deploy command returns. Deploy scripts often return before the new process has finished starting, so an immediate check races the app’s own startup time and produces false negatives — or worse, false positives if the old instance is still answering while the new one boots.

    - name: Deploy new version
      run: ./scripts/deploy.sh "$IMAGE"

    - name: Check health
      run: |
        status=$(curl -s -o /dev/null -w "%{http_code}" https://api.example.com/healthz)
        if [ "$status" != "200" ]; then
          exit 1
        fi

Fix: poll with retries and a delay, giving the service a real window to finish starting before declaring failure.

    - name: Deploy new version
      run: ./scripts/deploy.sh "$IMAGE"

    - name: Wait for service to become healthy
      run: ./scripts/health-check.sh https://api.example.com/healthz

Mistake 2: Rolling back to a floating tag instead of a recorded digest. If a rollback step redeploys app:latest, that tag may have already been overwritten by a newer, possibly also-broken build by the time the rollback runs — tags are mutable pointers, not fixed versions.

    - name: Roll back
      run: ./scripts/deploy.sh "ghcr.io/example/app:latest"

Fix: capture the exact digest that was running before the new deploy, and roll back to that specific, immutable reference.

    - name: Roll back
      run: ./scripts/deploy.sh "ghcr.io/example/app@${{ needs.deploy.outputs.previous_digest }}"

Best Practices

  • Prefer image digests over tags for both deploys and rollbacks — a digest always points to the exact same content, while a tag can move.
  • Give health checks a bounded retry window with backoff instead of a single check or an unbounded loop that can hang a runner.
  • Keep smoke tests small and idempotent: create-then-delete patterns, read-only checks on critical routes, nothing that leaves junk data behind on every deploy.
  • Gate rollback jobs with if: failure() so they only run when something upstream actually broke, and route their result to wherever your team gets alerts.
  • Use protected environments with required reviewers on production, but keep the automated rollback path fast and reviewer-free — recovery should not wait on a human approval that the original deploy already required.
  • Remember that rolling back application code does not roll back database migrations. Prefer backward-compatible, forward-only migrations so an old app version can still run correctly against a newer schema during a rollback.
  • Grant workflows only the permissions the job needs — contents: read for checkout, deployments: write only where the Deployment API is actually updated.
  • Set a short bake time after smoke tests pass before considering the deploy fully done, so slow-to-surface errors have a chance to trip monitoring before the pipeline moves on.

Practice Exercises

  • Write a health-check step that polls an endpoint up to 8 times with a 5-second delay, and fails the job with a clear message if none of the attempts return HTTP 200.
  • Add a smoke-test job that runs only after a successful deploy job, and have it check at least two different endpoints instead of one.
  • Extend a deploy job to record the previously running image digest as a job output, then write a rollback job that only runs when the deploy job fails, redeploying that digest.
  • Add a workflow_dispatch workflow that accepts an image digest as input and performs a manual rollback to it, respecting the same protected environment as your automatic deploy.
  • Explain, in a comment on your workflow file, why redeploying a previous digest is safer for rollback than redeploying a branch or tag reference.

Summary

A deployment isn’t verified until something checks that it’s actually working. Health checks confirm the process is alive and ready; smoke tests confirm the features people rely on still function; and a rollback path — ideally automatic, always available manually — gets you back to a known-good, digest-pinned version fast when either check fails. Treat these as one connected pipeline stage, not optional extras bolted on after the real work of deploying is done.