Cloud MLOps Reference Architecture

A cloud MLOps reference architecture is a repeatable blueprint for moving a model from experimentation to governed production service. Its outcome is not a diagram alone. The architecture must make it clear which artifact is running, which data and code produced it, who approved it, how traffic reaches it, how behavior is measured, and how the system returns to a known-good version when something fails.

In this course, the reference architecture ties together earlier lessons on training pipelines, registries, deployment, monitoring, and rollback. A useful cloud design treats models as versioned release units, not as loose files copied from a notebook. The central design question is: what must be immutable, what may change by configuration, and where should automated gates stop unsafe changes?

Reference Architecture Flow

A typical cloud MLOps architecture has six cooperating planes. The data plane stores curated training data, feature tables, validation reports, and sometimes streaming features. The build plane creates reproducible training and packaging runs. The registry plane stores model versions, metadata, signatures, evaluation results, and lifecycle state. The release plane promotes candidates through test, staging, canary, and production. The serving plane exposes online endpoints, batch scoring jobs, or event consumers. The operations plane observes service health, data drift, prediction quality, cost, and incidents.

The strongest design choice is to pass an immutable model package between planes. That package normally includes serialized weights or model binaries, preprocessing code or a feature contract, dependency metadata, input and output schema, evaluation metrics, and a pointer to training lineage. Runtime configuration can select traffic percentage, instance size, secrets, and endpoint policy, but it should not silently change the model contents.

Mechanisms and Internals

Internally, the architecture works by separating run evidence from release intent. A training pipeline produces evidence: dataset version, source revision, container digest, parameters, metrics, and artifacts. A registry records that evidence under a model version. A promotion gate evaluates the evidence against rules such as minimum accuracy, maximum latency, fairness bounds, schema compatibility, and required approvals. A deployment controller then maps an approved model version to serving infrastructure.

Cloud services differ in names, but the primitives are similar. Object storage holds artifacts. A workflow orchestrator runs steps with retry and dependency rules. A container registry stores trainer and server images by digest. A feature store or warehouse provides point-in-time training data and online features. A model registry holds lifecycle state such as candidate, staging, approved, deployed, archived, or rejected. A deployment target may be a managed endpoint, Kubernetes service, serverless function, batch job, or stream processor.

Lineage is the mechanism that makes the architecture auditable. A deployed endpoint should answer: model version, artifact digest, training data snapshot, code revision, evaluation report, approver, deployment time, and previous production version. Without that link, incident response becomes guesswork. With it, teams can compare the current model against the exact candidate that passed validation.

Configuration Anatomy

A practical reference architecture can be described as a manifest. The manifest names artifact sources, gates, environments, deployment strategy, and telemetry requirements. It should be reviewed like application release configuration because it controls production behavior.

manifest = {
    "model_name": "customer_churn",
    "version": "2026-09-06.1",
    "lineage": {
        "data_version": "dataset:churn-2026-08",
        "code_revision": "git:8ac421f",
        "trainer_image": "registry/ml-trainer@sha256:111"
    },
    "serving": {
        "mode": "shadow",
        "traffic_percent": 0,
        "input_schema": "schema/customer-v4.json"
    }
}

required = {"data_version", "code_revision", "trainer_image"}
missing = sorted(required - set(manifest["lineage"]))
if missing:
    raise ValueError(f"missing lineage fields: {missing}")

lineage_key = "|".join(manifest["lineage"][name] for name in sorted(required))
print(lineage_key)
print(manifest["serving"]["mode"])

This first example validates a minimal release manifest before any cloud deployment happens. The deterministic output is a stable lineage key followed by the serving mode. The expected behavior is refusal if any required lineage field is absent, because an untraceable model should not enter a governed release path.

Promotion Gates

Promotion gates convert evaluation reports into release decisions. A gate should check more than one score. For example, a churn model may need enough AUC to be useful, low enough prediction latency for the customer workflow, and a bounded bias gap across monitored groups. The gate is a policy decision, so keep thresholds explicit and versioned.

import json

candidate_metrics = {
    "auc": 0.914,
    "p95_ms": 38,
    "bias_gap": 0.027
}

thresholds = {
    "min_auc": 0.900,
    "max_p95_ms": 50,
    "max_bias_gap": 0.030
}

checks = {
    "auc": candidate_metrics["auc"] >= thresholds["min_auc"],
    "latency": candidate_metrics["p95_ms"] <= thresholds["max_p95_ms"],
    "fairness": candidate_metrics["bias_gap"] <= thresholds["max_bias_gap"]
}

print(json.dumps(checks, sort_keys=True))
print("PROMOTE" if all(checks.values()) else "BLOCK")

The output is {"auc": true, "fairness": true, "latency": true} and PROMOTE. If the bias gap were 0.041, the same mechanism would print BLOCK. In a cloud architecture, this logic is usually implemented in a pipeline step, policy engine, CI job, or registry transition hook. The important property is that the gate runs before the deployment controller can route traffic.

Progressive Deployment

After approval, release strategy controls exposure. Shadow deployment sends production requests to the new model without using its predictions. Canary deployment sends a small percentage of real traffic to the new model. Blue-green deployment prepares a full replacement environment and switches routing when checks pass. Batch scoring often uses a staged output location and only publishes results after validation.

requests = [
    {"id": "r001", "risk": 0.12},
    {"id": "r002", "risk": 0.83},
    {"id": "r003", "risk": 0.48},
    {"id": "r004", "risk": 0.77}
]

canary_percent = 25
canary_count = round(len(requests) * canary_percent / 100)

for index, request in enumerate(requests):
    route = "new-model" if index < canary_count else "current-model"
    decision = "review" if request["risk"] >= 0.75 else "standard"
    print(f"{request['id']} -> {route} -> {decision}")

The output routes only r001 to new-model; the remaining requests use current-model. The example uses deterministic ordering for teaching, but production canaries usually use a stable hash of tenant, account, or request identifier. Stable routing prevents the same user from receiving alternating model behavior during one release window.

Design Choices and Trade-Offs

Managed cloud ML platforms reduce infrastructure burden and provide integrated registries, endpoints, workflow logs, and identity controls. They can also make portability harder and may hide low-level deployment details. Kubernetes-based MLOps gives more control over networking, custom runtimes, and multi-cloud patterns, but it raises operational load. Serverless inference can be economical for bursty workloads, while always-on endpoints fit low-latency, steady traffic.

Feature handling is another major trade-off. Embedding preprocessing in the model package improves reproducibility, but shared feature stores reduce duplicated logic across models. Online feature stores support low-latency decisions but add freshness, backfill, and consistency concerns. Batch architectures are simpler and easier to audit, but they cannot support interactive decisions.

Promotion policy also deserves care. Strict gates protect production but can slow urgent fixes. Manual approvals add accountability but may become rubber stamps if the evidence is hard to inspect. The best architecture makes approvals evidence-based: evaluators should see metric deltas, data coverage, schema changes, known limitations, and rollback target.

Failure Modes and Troubleshooting

Symptom: a deployed endpoint returns schema errors for normal traffic. Cause: the model was trained with customer-v5 features but the production endpoint still sends customer-v4. Diagnostic steps: inspect the endpoint model version, registry signature, request validation logs, and feature pipeline deployment history. Correction: roll back routing to the previous model or deploy the compatible feature contract, then add schema compatibility as a promotion gate.

Symptom: canary metrics look healthy, but complaints rise from one customer segment. Cause: traffic sampling was request-random rather than tenant-stable, so the canary underrepresented a high-volume segment or split user experience. Diagnostic steps: group canary traffic by tenant, geography, plan, or channel and compare prediction distribution against baseline. Correction: use stable segment-aware routing and require slice metrics before expanding traffic.

Symptom: retraining produces an apparently better model that fails after deployment. Cause: training used future information, duplicate labels, or a feature backfill unavailable online. Diagnostic steps: compare event timestamps, label cutoff times, feature materialization times, and online feature null rates. Correction: enforce point-in-time joins, add leakage tests, and block registry promotion when online-offline skew exceeds tolerance.

Security, Reliability, and Performance

Security starts with identity boundaries between pipeline runners, artifact registries, feature stores, and serving workloads. A trainer usually needs read access to curated data and write access to artifact storage. A serving workload needs read access to approved artifacts and runtime features, not write access to training data. Registry state transitions should require a service identity or reviewer group rather than broad developer credentials.

Reliability depends on idempotent pipeline steps and atomic promotion. If a training job retries, it should create a new run or safely resume the same run without overwriting another candidate. Deployment should keep the previous production version addressable until the new version has passed health and behavior checks. Performance budgets should include model load time, cold starts, feature lookup latency, prediction latency, queue depth, and cost per prediction or scored record.

Hands-On Lab

Prerequisites: Python 3, a shell, and a temporary working directory. No cloud account is required because the lab simulates the control-plane logic that cloud services implement.

  1. Create a file named architecture_gate.py and place the three Python examples in it in order.
  2. Run python architecture_gate.py. Verify that the lineage key prints, the promotion decision is PROMOTE, and only one of four requests routes to new-model.
  3. Change bias_gap to 0.041 and rerun the script. Verify that the promotion decision changes to BLOCK.
  4. Remove trainer_image from the manifest and rerun. Verify that the script raises a lineage validation error before deployment routing matters.
  5. Restore the original values. This cleanup returns the simulated architecture to an approved candidate state.

In a real cloud implementation, the same lab maps to separate services: the manifest is release configuration, the gate is a CI or workflow step, and the routing block is endpoint traffic policy. The verification principle remains the same: inspect the evidence before traffic moves.

Assessment Exercises

  1. A model has excellent aggregate accuracy but worse recall for one regulated customer segment. Which gate would you add, and where should it run in the architecture?
  2. Your team wants to update preprocessing code without changing the model artifact. What must be versioned so rollback is still reliable?
  3. A batch scoring job writes partial output before failing halfway through. Design a publish mechanism that prevents downstream consumers from reading mixed results.
  4. Canary latency is normal, but cloud cost doubles. Which telemetry would you inspect to distinguish model compute cost from feature lookup cost?
  5. Two approved models are accidentally deployed to the same endpoint during overlapping releases. What invariant should the deployment controller enforce?

Summary

A cloud MLOps reference architecture is a control system for model releases. It records lineage, stores immutable artifacts, evaluates promotion gates, routes traffic deliberately, observes model and service behavior, and preserves rollback paths. The architecture succeeds when every production prediction can be traced to its approved evidence and every failed release can be diagnosed without reconstructing history from memory.