MLOps Lifecycle and System Boundaries

The MLOps lifecycle is the route by which an idea becomes a monitored decision system and, later, is retired or replaced. System boundaries are the lines that say which component owns which state: raw data, features, training code, model artifacts, online predictions, human approvals, and business actions. In this lesson, the outcome is concrete: you should be able to draw an MLOps system as stages and handoffs, name the artifact that crosses each handoff, and decide what must be versioned, tested, approved, and observed before a model affects users.

Purpose and Operating Outcome

Traditional software usually ships code. An ML system ships a coupled bundle of code, data assumptions, fitted parameters, feature definitions, thresholds, runtime dependencies, and policy decisions. MLOps exists because any one of those pieces can change system behavior. A lifecycle gives the team a repeatable route from problem framing to training, evaluation, registration, deployment, monitoring, retraining, and rollback. Boundaries prevent the route from becoming a blur of notebooks, scripts, services, and dashboards where nobody can tell which version made a particular prediction.

In this MLOps foundations course, the lifecycle is the map you will reuse when studying experiment tracking, model registries, feature stores, CI/CD, monitoring, and deployment patterns. Each later tool solves one boundary problem: tracking captures evidence, registries control promotion, deployment systems isolate runtime change, and monitoring detects when production diverges from evaluation.

How the Lifecycle Works Internally

A dependable MLOps lifecycle usually has eight internal states. First, problem framing defines the decision, target variable, risk, and acceptance criteria. Second, data acquisition records source tables, extraction time, consent constraints, and labeling rules. Third, feature engineering turns raw facts into model inputs and records schema, transformations, and freshness expectations. Fourth, training produces candidate artifacts from an immutable data snapshot, code revision, environment, and parameter set. Fifth, evaluation compares the candidate against statistical, fairness, latency, and business gates. Sixth, registration stores the candidate, its lineage, and its stage such as candidate, approved, production, archived, or rejected. Seventh, deployment binds a model version to a serving surface: batch job, online API, streaming consumer, embedded model, or human review queue. Eighth, operation monitors service health, data quality, prediction distributions, outcomes, and incidents.

The important mechanism is the controlled transition between stages. A transition should accept one immutable input and produce one auditable output. A training run should not silently read today’s mutable table if the evaluation report names last week’s table. A deployment should not pull whatever model is latest if the release ticket approved a specific registry version. A retraining workflow should not overwrite the production model until the new candidate has passed the same gates that allowed the existing one.

Boundary Anatomy

Every boundary in an MLOps system has a contract. The contract names inputs, outputs, owner, allowed callers, version identifiers, validation rules, side effects, and rollback behavior. A data boundary might say that training consumes a partitioned snapshot named by date and checksum. A feature boundary might say that the online service accepts only schema features:v4 with non-null tenure, region, and balance fields. A model boundary might say that deployment receives a registry URI, image digest, threshold, and explanation mode, not a loose file copied from a notebook.

The practical syntax varies by platform, but the same fields recur: run_id, dataset_version, feature_schema, model_version, metric_name, metric_value, stage, approved_by, deployed_at, and rollback_target. These identifiers let an operator answer, “Which code and data produced the model serving this request?” If that question cannot be answered from records rather than memory, the boundary is too weak.

Example 1: Promotion Gate From Training to Registry

The first example models the boundary between evaluation and registration. A candidate can move forward only if it carries lineage and satisfies both quality and fairness gates. The function returns a deterministic decision string. The expected output is promote: churn-v12 from customers-2026-08-31 at git:91e7c2.

from dataclasses import dataclass

@dataclass(frozen=True)
class ModelCandidate:
    name: str
    data_snapshot: str
    code_revision: str
    validation_auc: float
    bias_delta: float


def promotion_decision(candidate: ModelCandidate, min_auc: float, max_bias_delta: float) -> str:
    missing = [field for field in (candidate.name, candidate.data_snapshot, candidate.code_revision) if not field]
    if missing:
        return "reject: incomplete lineage"
    if candidate.validation_auc < min_auc:
        return "reject: accuracy gate failed"
    if candidate.bias_delta > max_bias_delta:
        return "reject: fairness gate failed"
    return f"promote: {candidate.name} from {candidate.data_snapshot} at {candidate.code_revision}"

candidate = ModelCandidate("churn-v12", "customers-2026-08-31", "git:91e7c2", 0.914, 0.018)
print(promotion_decision(candidate, min_auc=0.90, max_bias_delta=0.03))

This captures the real control point. Promotion is not “the notebook looked good.” It is a rule over a named candidate. If the data snapshot is blank, the candidate is rejected even with a high metric because the team could not reproduce or audit it. If the AUC falls below the gate, the registry should record rejection rather than hiding the run. If the fairness delta exceeds the limit, the model does not advance even if aggregate accuracy is attractive.

Example 2: Deployment Contract at the Serving Boundary

The second example represents the boundary between model registry and serving system. The serving service should receive a compact contract that names the feature schema, model version, decision threshold, and owning team. The fingerprint is deterministic for the same contract, so it can be logged with every prediction and used during incident review. For this input the first printed line is e617065a91a2; the second line prints the contract fields.

from dataclasses import dataclass
from hashlib import sha256
import json

@dataclass(frozen=True)
class BoundaryContract:
    feature_schema_version: str
    model_version: str
    decision_threshold: float
    owner: str

    def fingerprint(self) -> str:
        payload = json.dumps(self.__dict__, sort_keys=True, separators=(",", ":"))
        return sha256(payload.encode("utf-8")).hexdigest()[:12]

contract = BoundaryContract("features:v4", "model:churn-v12", 0.67, "risk-ml")
print(contract.fingerprint())
print(contract)

A fingerprint is not a replacement for full lineage, but it is cheap enough to include in API logs, batch output, and dashboards. When a customer complaint arrives, support can group decisions by fingerprint, then use the registry to expand that fingerprint into the complete model and configuration record.

Example 3: Operational Drift at the Monitoring Boundary

The third example checks whether a categorical feature at serving time still resembles the distribution used for training. It compares regional proportions and reports categories whose absolute change is greater than eight percentage points. The expected output is ['east: -9.0%'].

from collections import Counter

training_regions = Counter({"north": 4200, "south": 3800, "west": 2100, "east": 1900})
serving_regions = Counter({"north": 210, "south": 170, "west": 95, "east": 35})


def proportions(counts: Counter) -> dict[str, float]:
    total = sum(counts.values())
    return {key: value / total for key, value in counts.items()}

train = proportions(training_regions)
serve = proportions(serving_regions)
alerts = []
for region in sorted(train):
    delta = serve.get(region, 0.0) - train[region]
    if abs(delta) > 0.08:
        alerts.append(f"{region}: {delta:+.1%}")

print(alerts or ["no material region drift"])

This is not a full drift detector. It is a boundary alarm: the live prediction stream has moved far enough from the evaluated population that somebody should investigate. The correction might be retraining, a data pipeline fix, a product launch annotation, or a temporary routing rule. The lifecycle matters because the response should create a new candidate and evaluation record, not patch the live model without evidence.

Design Choices and Trade-Offs

The first design choice is boundary size. A single end-to-end pipeline is easier to reason about, but it can become slow and hard to change. Many small pipelines support ownership and reuse, but they create more contracts to version. Early teams often start with a training pipeline and a deployment script, then split feature computation, model evaluation, registry promotion, and serving as volume and risk increase.

The second choice is synchronous versus asynchronous handoff. Online prediction has a tight latency boundary and needs precomputed or fast features. Batch scoring can tolerate slower joins and heavier validation, but stale outputs may drive decisions for hours or days. Human approval adds accountability for high-risk domains, yet it also creates queues and ambiguous ownership unless the approval state is recorded as part of the lifecycle.

The third choice is automatic retraining. Scheduled retraining is simple and predictable, while trigger-based retraining reacts to drift or performance decay. Both are unsafe if they bypass evaluation. Automatic promotion should be reserved for low-risk systems with strong tests, stable data contracts, and rollback. In higher-risk systems, automation should prepare the candidate and evidence, while a human approves promotion.

Failure Modes and Troubleshooting

Symptom: offline validation is excellent, but live predictions are poor. Likely cause: training and serving features are computed differently, or serving uses a different schema version. Diagnostics: compare feature schema identifiers in training runs, registry metadata, and prediction logs; replay a small request through both transformation paths; inspect null rates and categorical levels. Correction: centralize shared transformations or version duplicated implementations, block deployment when schema versions do not match, and add a canary check that scores known records before release.

Symptom: the team cannot explain which model produced a disputed decision. Likely cause: model files were copied directly to serving storage without registry lineage or deployment fingerprints. Diagnostics: inspect serving logs for model version, image digest, threshold, and deployment time; compare those values with registry entries and release tickets. Correction: require deployments to reference immutable registry versions, log the deployment contract fingerprint with every prediction, and archive the untracked artifact rather than promoting it further.

Symptom: retraining runs fail intermittently or produce inconsistent metrics. Likely cause: the pipeline reads mutable source data while labels are still arriving or source backfills are in progress. Diagnostics: check extraction timestamps, snapshot checksums, row counts by partition, and upstream incident history. Correction: train only from sealed snapshots, record checksums, and delay training until upstream data quality checks pass.

Security, Performance, and Reliability Implications

Security boundaries matter because ML pipelines often touch sensitive raw data while serving systems should not. Training jobs may need read access to labeled histories; online prediction usually needs only features for the current subject and the approved model artifact. Split credentials by stage. The deployment service should not have permission to rewrite training data, and the notebook environment should not be able to replace a production model without registry approval.

Performance boundaries decide where computation happens. Feature generation can be done at ingestion, on demand, in a batch scoring job, or inside the online service. Moving work earlier improves prediction latency but risks staleness. Moving work into the request path improves freshness but increases tail latency and dependency failures. Reliability comes from making these choices explicit and testing each boundary under expected volume, missing features, delayed labels, and dependency timeouts.

Hands-On Lab: Draw and Test a Small Lifecycle

Prerequisites: Python 3, a terminal, and permission to create a temporary working directory. No external service is required.

  1. Create a temporary directory and save the three examples above as promotion.py, contract.py, and drift.py.
  2. Run python promotion.py. Verify that the output says the candidate is promoted from the named data snapshot and code revision.
  3. Edit the candidate AUC to 0.70 and rerun the file. Verify that the output changes to reject: accuracy gate failed. Restore the original value afterward.
  4. Run python contract.py twice. Verify that the fingerprint is identical across runs. Change only the threshold from 0.67 to 0.70 and verify that the fingerprint changes.
  5. Run python drift.py. Verify that east is flagged. Change the serving count for east from 35 to 70 and rerun; the alert should disappear if no region exceeds the threshold.
  6. Cleanup by deleting the temporary directory, or rollback by restoring the edited values from the code shown in this lesson.

Assessment Exercises

  1. A batch model is retrained every night from a table that upstream systems can backfill for seven days. Design the data boundary so yesterday’s run can be reproduced exactly.
  2. An online model needs a user balance feature that changes throughout the day. Decide whether to compute it at ingestion, in a feature store, or in the request path, and defend the reliability trade-off.
  3. A candidate improves aggregate accuracy but worsens performance for a regulated customer segment. Specify the promotion gate and registry metadata that should prevent an unsafe release.
  4. Your incident review has request logs but no model version. Propose the minimum deployment contract fields that future logs must include.
  5. Define one rollback test for a model served through an API and one for a model used in a daily batch scoring job.

Summary

The MLOps lifecycle turns model delivery into a sequence of auditable transitions: problem, data, features, training, evaluation, registration, deployment, monitoring, and retirement. System boundaries make each transition explicit by naming the artifact, owner, version, validation rule, and rollback path. The goal is demanding: any production prediction should be traceable to the data, code, model, configuration, and approval that produced it, and every replacement should pass through the same controlled route.