Docker Layer Caching in CI

Docker layer caching in CI means reusing work from previous image builds so every commit does not reinstall the same dependencies from scratch. It matters because CI runners are often fresh machines: without an external cache, a build that is fast on your laptop can become slow and expensive in every pipeline run. Good caching is mostly about writing Dockerfiles whose stable steps come first, then exporting BuildKit cache somewhere the next runner can pull it.

Overview: How Docker Layer Caching Works in CI

A Docker image is made from read-only layers stacked by a union filesystem. Each Dockerfile instruction such as FROM, COPY, and RUN contributes metadata and often a filesystem diff. During a build, BuildKit checks whether it has already produced the result for a step with the same inputs. If the instruction text, parent layer, build arguments that affect the step, copied files, or mounted build inputs change, that step misses the cache and every later dependent step must run again.

On a developer machine, the cache usually lives in the local Docker builder storage, so repeated docker build commands feel fast. CI is different. Many CI systems create a new runner for each job. That runner has an empty builder cache unless you restore one. Pulling the final image, such as myapp:main, can help with classic image-layer reuse, but modern BuildKit cache metadata is richer than a final image alone. BuildKit can export cache records to a registry reference, then a later job can import them with --cache-from.

The most important Dockerfile rule is to put stable layers before volatile layers. For a Node.js app, copy package.json and package-lock.json, run npm ci, and only then copy the rest of the source. A source code edit then invalidates the app-copy layer, but not the dependency-install layer. If you do COPY . . before npm ci, every source change changes the input to dependency installation and the expensive install layer is rebuilt.

Registries store image manifests, layer blobs, and optional cache manifests. When you use docker buildx build --cache-to type=registry,ref=..., BuildKit pushes cache data under the registry reference you choose. The next CI run uses --cache-from type=registry,ref=... to pull enough metadata and layer content to skip steps that still match. The cache reference is usually separate from the runtime image tag; for example, push the application image to registry.example.com/team/web:main and the cache to registry.example.com/team/web:buildcache.

Syntax

The common BuildKit form for CI is:

docker buildx build [OPTIONS] PATH
Option Meaning
-t name:tag Names the image that will be produced or pushed.
--file Dockerfile Chooses a Dockerfile when it is not named Dockerfile in the context root.
--push Pushes the resulting image to its registry. CI normally uses this for shared images.
--load Loads the image into the local Docker Engine instead of pushing. Useful for local testing, not for multi-platform registry output.
--cache-from type=registry,ref=IMAGE Imports BuildKit cache metadata and reusable layers from a registry reference.
--cache-to type=registry,ref=IMAGE,mode=max Exports cache data to a registry reference. mode=max preserves more intermediate cache records than the default minimal export.
--build-arg NAME=value Passes a build argument. Changing a build arg used by a step can invalidate that step.
. The build context. Docker sends files from this directory to the builder, excluding paths in .dockerignore.

These examples assume a Docker daemon is available, BuildKit/buildx is installed with modern Docker, and you are logged in to any private registry you push to.

Examples

Example 1: A Cache-Friendly Node.js Dockerfile

FROM node:20-alpine
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY . .
EXPOSE 3000
CMD ["node", "server.js"]

Output:

# First build:
#8 [4/6] RUN npm ci --omit=dev
#8 DONE 18.4s
#9 [5/6] COPY . .
#9 DONE 0.2s

# After editing server.js only:
#8 CACHED
#9 [5/6] COPY . .
#9 DONE 0.2s

This Dockerfile uses the pinned base image tag node:20-alpine instead of node or latest, which makes builds more reproducible. The dependency files are copied before the source code, so npm ci is cached until package.json or package-lock.json changes. EXPOSE 3000 is only metadata documenting the container port; it does not publish the port to the host. Publishing still requires docker run -p 3000:3000 ... or Compose ports:.

Example 2: Export and Import a Registry Cache in CI

docker buildx build \
  --file Dockerfile \
  --tag registry.example.com/team/web:main \
  --cache-from type=registry,ref=registry.example.com/team/web:buildcache \
  --cache-to type=registry,ref=registry.example.com/team/web:buildcache,mode=max \
  --push \
  .

Output:

#12 importing cache manifest from registry.example.com/team/web:buildcache
#12 DONE 1.1s
#8 [4/6] RUN npm ci --omit=dev
#8 CACHED
#15 exporting cache to registry
#15 DONE 4.3s
#16 pushing registry.example.com/team/web:main
#16 DONE 2.8s

The command imports cache records from a registry reference before building, then exports updated cache records after the build. The application image tag and the cache reference are separate on purpose: the image tag is what you deploy, while the cache reference is build infrastructure. On the first run, the cache import may say the reference was not found; that is normal because nothing has been exported yet. Later runs can reuse matching layers even when the CI runner starts empty.

Example 3: A GitHub Actions Job Using Buildx Cache

name: image
on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: registry.example.com
          username: ${{ secrets.REGISTRY_USER }}
          password: ${{ secrets.REGISTRY_PASSWORD }}
      - uses: docker/build-push-action@v6
        with:
          context: .
          file: ./Dockerfile
          push: true
          tags: registry.example.com/team/web:main
          cache-from: type=registry,ref=registry.example.com/team/web:buildcache
          cache-to: type=registry,ref=registry.example.com/team/web:buildcache,mode=max

Output:

writing image sha256:8fb2c7...
pushing registry.example.com/team/web:main
exporting cache manifest to registry.example.com/team/web:buildcache

This workflow checks out the repository, prepares a Buildx builder, logs in to the registry, then builds and pushes with a shared registry cache. The secrets are referenced from the CI system; they are not written into the Dockerfile and are not baked into image layers.

How It Works Step by Step

  1. The CI runner checks out your repository. Docker treats the chosen directory, usually ., as the build context.
  2. Docker applies .dockerignore before sending files to the builder. Ignoring node_modules, .git, coverage reports, and local build output keeps the context small and avoids accidental cache busting.
  3. BuildKit reads the Dockerfile and starts from the base image layer. If the base image tag resolves to the same digest as before, the parent input is stable.
  4. For each instruction, BuildKit computes whether the instruction and its inputs match an existing cache record. A COPY package.json package-lock.json ./ step depends only on those two files, not the rest of the source tree.
  5. When a step matches, BuildKit reuses the cached filesystem snapshot. When a step misses, it executes the instruction and all later dependent instructions are evaluated on top of the new result.
  6. At the end, BuildKit creates an image manifest for the runtime image and, when configured, a separate cache manifest for future builds.

Common Mistakes

Copying Everything Before Installing Dependencies

FROM node:20-alpine
WORKDIR /app
COPY . .
RUN npm ci --omit=dev
CMD ["node", "server.js"]

This is valid Dockerfile syntax, but it is poor caching. Any source edit changes the COPY . . layer, so npm ci has to run again. The fix is to copy dependency manifests first, install dependencies, and then copy the source.

Letting the Build Context Change Constantly

node_modules
.git
coverage
dist
*.log

A missing .dockerignore sends unnecessary files to the builder. Large contexts slow uploads to remote builders and files such as logs or generated output can change on every run, invalidating COPY . .. The block above is a typical .dockerignore for a Node.js service.

Baking Secrets Into Cached Layers

FROM node:20-alpine
WORKDIR /app
ENV NPM_TOKEN="changeme"
RUN npm config set //registry.npmjs.org/:_authToken "$NPM_TOKEN"
COPY package.json package-lock.json ./
RUN npm ci --omit=dev

Do not put real tokens in ENV, ARG, or files copied into image layers. Even if a later instruction removes the file, the secret may remain in an earlier layer or cache record. Use your CI secret store with BuildKit secret mounts or your registry/package-manager integration instead.

Best Practices

  • Use pinned base image tags such as node:20-alpine; avoid latest for CI builds because it can change without a code change.
  • Order Dockerfile instructions from least-changing to most-changing: base image, system packages, dependency manifests, dependency install, then application source.
  • Commit a focused .dockerignore so the build context contains only files needed by the image.
  • Use docker buildx build with --cache-from and --cache-to for ephemeral CI runners.
  • Store cache in a registry reference your CI jobs can read and write. Keep it separate from deployable image tags.
  • Use mode=max for CI cache exports when you want intermediate build stages to be reusable.
  • Do not treat cache as correctness. Builds must still work from an empty cache; cache only makes correct builds faster.
  • Do not bake secrets into layers. Use CI secrets and BuildKit secret features so credentials do not become part of image history or cache.
  • Prune or rotate cache references if registry storage grows beyond your retention policy.

Practice Exercises

  1. You have a Python app whose Dockerfile does COPY . . before pip install -r requirements.txt. Rewrite the order so changing app.py does not reinstall dependencies. Hint: copy requirements.txt first.
  2. Add a registry cache to a CI build for registry.example.com/team/api:main. Use a separate cache reference ending in :buildcache and include both import and export settings.
  3. Create a .dockerignore for a Node.js project that excludes dependencies, Git metadata, logs, coverage, and generated build output. Expected end state: the build context is smaller and source-only changes are easier to reason about.

Summary

  • Docker caches builds per layer, and a changed layer invalidates the layers after it.
  • CI runners often start without local Docker cache, so registry cache export/import is the reliable BuildKit-era solution.
  • Dependency-first Dockerfile ordering is the biggest practical improvement for application builds.
  • .dockerignore protects both performance and cache stability by keeping noisy files out of the build context.
  • Cache speeds up builds, but it must not hide broken Dockerfiles or contain secrets.