Reproducible Environments and Dependency Locking

A reproducible MLOps environment is an execution context that can be rebuilt later with the same interpreter, dependency graph, operating system assumptions, and configuration that produced an experiment or model artifact. Dependency locking is the practice of recording the resolved package versions, and often their hashes, so that training, evaluation, batch scoring, and serving do not silently change when a package index changes.

The outcome is practical: when a model metric moves, you can decide whether the cause was data, code, parameters, or the environment. Without a lock, a rerun may install a newer transitive package, such as a numerical library or serialization dependency, and the comparison is no longer clean. In this course section on experiments and artifacts, the locked environment becomes part of the artifact lineage alongside the dataset snapshot, source revision, parameters, metrics, and produced model file.

What Gets Locked

An environment has layers. At the bottom is the operating system and CPU or accelerator runtime. Above that are the language runtime, build tools, system libraries, Python packages, and application configuration. A lock file usually records the Python dependency graph, not the entire machine. For full reproducibility, teams often combine a language lock file with a container image digest and a small manifest that names the data snapshot and code revision.

Dependency resolution starts from direct requirements, such as scikit-learn, then expands to transitive requirements, such as numpy, scipy, threadpoolctl, and joblib. A resolver selects versions that satisfy every declared constraint and environment marker. The lock captures the selected graph. If the lock includes file hashes, the installer can also reject a package file that has a matching name and version but different bytes.

Locks are not magic time machines. They do not pin external services, unversioned data files, GPU drivers, locale settings, random seeds, or CPU instruction behavior. They narrow the environment variable so that remaining differences are easier to diagnose.

Anatomy of a Lock

The simplest lock is a fully pinned requirements file. Each line names exactly one package version. More advanced tools add hashes, supported platforms, dependency groups, source indexes, and metadata about the resolver. Conda-style locks may include native libraries. Poetry and uv-style locks represent packages as structured records rather than one line per package. Docker and OCI images add another layer by freezing the filesystem built from those package choices.

numpy==1.26.4
pandas==2.2.2
scikit-learn==1.5.1
joblib==1.4.2

This example is readable, but intentionally incomplete for a real scikit-learn environment because it omits transitive dependencies. A production lock should be resolver generated, reviewed, and committed. Directly hand-editing only top-level packages creates a false sense of control because the installer still has freedom to choose the packages below them.

Example 1: From Intent to Resolved Graph

The first progressive step is separating human intent from machine resolution. A training project might declare that it needs pandas and scikit-learn, while the lock records the exact graph selected today. Developers update the intent file when the project needs a new capability. Automation regenerates the lock in a clean environment, runs tests, and shows the dependency diff.

The expected behavior is that two installs from the same lock produce the same package versions. If a teammate installs next week and the package index now contains newer releases, the installer should still choose the locked versions. If the lock is absent and only broad constraints are present, the same command may choose a different transitive dependency and make old experiment metrics hard to compare.

Example 2: Fingerprinting the Lock

A lock file is useful only if you record which lock produced an artifact. A common pattern is to compute a stable digest of the normalized lock text and store that digest in experiment metadata. The model registry entry can then say that model version 42 came from code revision X, data snapshot Y, and environment lock digest Z.

import hashlib
from pathlib import Path

lock_text = Path("requirements.lock").read_text(encoding="utf-8")
normalized = "\n".join(
    line.strip()
    for line in lock_text.splitlines()
    if line.strip() and not line.startswith("#")
)
print(hashlib.sha256(normalized.encode("utf-8")).hexdigest())

When run beside a requirements.lock file, this prints a 64-character SHA-256 hexadecimal digest. If any pinned version changes, the digest changes. The normalization removes blank lines and comments, so adding a comment does not create a new environment identity. That is a design choice: some teams want comments to affect the digest because they treat the whole file as evidence; others want only installable content to count.

Example 3: Capturing the Environment Manifest

The next step is joining the dependency lock with runtime context. The manifest below is small enough to inspect but has the fields needed to connect a model artifact to a rebuild procedure.

import json
import sys

manifest = {
    "python": sys.version.split()[0],
    "lock_sha256": "replace-with-output-from-fingerprint-step",
    "training_image": "local/mlops-lock-lab:demo",
    "data_snapshot": "demo-csv-v1"
}
print(json.dumps(manifest, sort_keys=True, indent=2))

The deterministic part of the output is the sorted JSON shape: it contains data_snapshot, lock_sha256, python, and training_image. The Python version depends on the interpreter running the script. In an experiment tracker, this manifest should be attached to the run before the model is promoted. During incident review, it lets you rebuild the candidate environment before blaming model code or data drift.

Example 4: Freezing the Installer Boundary

A container image turns the lock into an executable boundary. The Dockerfile below installs from the lock before copying the training script, which lets the dependency layer be cached until the lock changes.

FROM python:3.12-slim
WORKDIR /app
COPY requirements.lock .
RUN pip install --no-cache-dir -r requirements.lock
COPY train.py .
CMD ["python", "train.py"]

The expected behavior is that changing train.py rebuilds only the application layer, while changing requirements.lock rebuilds the dependency layer. For stronger rebuild evidence, production pipelines usually refer to a base image by digest and record the built image digest, not only the tag. Tags are convenient names; digests identify bytes.

Design Choices and Trade-offs

Strict pins maximize repeatability but slow routine upgrades. Broad constraints make upgrades easier but reduce experiment comparability. Hash checking improves supply-chain integrity but requires every resolved file to be known for each target platform. Single lock files are simple for homogeneous Linux training jobs; platform-specific locks are better when developers use macOS, CI uses Linux, and serving uses a slim container.

There is also a cadence trade-off. Updating locks on every commit creates noise and can mask code changes. Updating them only during incidents leaves known vulnerabilities and incompatibilities in place. A workable policy is to regenerate locks in a scheduled dependency update job, run the training and serving test suite, review the diff, and promote the new environment as its own change.

For notebooks, the trap is hidden state. A notebook that imports packages from a developer laptop is not a reproducible experiment unless the kernel environment is built from the same lock. Store the lock and manifest with the notebook run, not merely in a separate repository directory.

Failure Modes and Troubleshooting

Symptom: a rerun of an old training job produces a different score although the code and data snapshot look unchanged. Cause: the job installed from broad constraints and picked a newer transitive numerical dependency. Diagnose: compare pip freeze or the experiment manifest from both runs, then check resolver logs. Correct: regenerate a full lock from the older known-good environment if possible, record the digest, and make future jobs install from the lock.

Symptom: CI fails with "no matching distribution" for a locked package. Cause: the lock was generated on a platform or Python version that differs from CI. Diagnose: inspect the package markers, wheel tags, and interpreter version. Correct: generate locks for the actual deployment platform or use a tool that records platform-specific resolutions.

Symptom: a container rebuild succeeds but the deployed model still behaves differently. Cause: the lock pinned Python packages, but the base image tag moved or an external data file was overwritten. Diagnose: compare image digests, manifest fields, dataset checksums, and environment variables. Correct: pin the base image by digest, version the data snapshot, and store both values with the experiment run.

Security, Performance, and Reliability

Locks reduce accidental dependency drift and make vulnerability response targeted: you can search for every model trained with a vulnerable package version. Hashes make package substitution harder. Private package indexes should be explicit in build configuration, and credentials should be injected by CI rather than committed.

Performance can change when numerical libraries, BLAS implementations, or GPU runtimes change. Treat an environment update like a model-affecting change: run representative training and inference benchmarks. Reliability improves because rollback can restore not just old code but the old dependency graph and image digest.

Hands-on Lab: Rebuild and Verify a Locked Run

Prerequisites: Python, a virtual environment tool, Docker if you want the container step, and a small repository with train.py. Use a throwaway branch or directory.

  1. Create requirements.lock with pinned package versions generated by your resolver. For a quick local exercise, start with lightweight packages that install on your platform.
  2. Install into a fresh virtual environment using only the lock. Do not install from broad requirement ranges.
  3. Run the fingerprint script from Example 2 and save the digest in a run metadata file.
  4. Run training and write a manifest using the fields from Example 3.
  5. Delete the virtual environment, recreate it from the same lock, rerun training, and compare package versions plus the manifest fields.
  6. Optional: build the Dockerfile from Example 4 and record the resulting image digest.

Verification: the package list matches after reinstall, the lock digest is unchanged, and the run metadata points to the same lock, code revision, and data snapshot. If the model metric is nondeterministic, verify that it remains within the expected tolerance and that the environment evidence is identical.

Cleanup: remove the virtual environment, delete temporary manifests, and remove the local demo image if you built one. Roll back by restoring the previous lock file and rebuilding from that file, not by manually downgrading individual packages.

Assessment Exercises

  1. A model registry entry has code revision and dataset version but no environment digest. Explain what incident question remains unanswerable and how you would fix the metadata.
  2. Your team wants one lock for laptop development, Linux CI, and GPU serving. Identify two risks and propose a locking strategy.
  3. A dependency update improves security but changes validation metrics by 0.7 percent. Describe the tests and evidence needed before promotion.
  4. Given a Docker image tag and a requirements lock, explain why the build may still be non-reproducible and what additional identifier you need.
  5. Design a rollback plan for a bad environment update that does not roll back model data or application code.

Summary

Reproducible environments make experiment comparisons defensible by turning dependencies and runtime context into recorded artifacts. A good MLOps lock strategy separates intent from resolution, stores a digest with each run, pins the build boundary, and treats environment updates as reviewable changes. The goal is not to freeze software forever; it is to know exactly what changed when model behavior changes.