Shadow, Canary, and Blue-Green Deployments

Shadow, canary, and blue-green deployments are release patterns for changing a live model service without betting the whole business process on a single cutover. The outcome is simple: expose a candidate model to increasing operational reality while preserving evidence, comparison, and rollback. In this MLOps course, safer model delivery means code, feature transformations, model artifacts, thresholds, and serving routes all move under control.

Purpose and Outcome

A conventional one-step model replacement answers only one question: did the new service start? Shadow, canary, and blue-green deployments answer better questions. Shadowing asks, "What would the candidate have predicted on real traffic if it had not been allowed to affect users?" Canarying asks, "Can a small, controlled slice of users receive the candidate without harming service health or model outcomes?" Blue-green asks, "Can we keep two complete serving environments ready so routing can move forward or back quickly?"

These patterns are especially useful for models because offline metrics rarely cover all production conditions. Feature distributions shift, upstream services change serialization details, latency varies with real request bursts, and downstream policy rules may react badly to a changed score distribution. A delivery plan defines the traffic path, comparison signals, promotion rule, and rollback action before the candidate receives business-critical load.

How the Mechanisms Work

In a shadow deployment, the production request is copied after authentication and feature lookup. The primary model returns the response used by the product. The shadow model receives the same feature vector, produces a prediction, and writes telemetry, but its result is never returned to the user or downstream decision engine. The important internal detail is correlation: both predictions need the same request identifier, model version, feature schema version, and timestamp. Shadowing finds latency, schema, and score-distribution problems, but it cannot measure user behavior caused by the candidate because nobody acts on its output.

In a canary deployment, a router sends a small percentage or selected segment of live requests to the candidate. The router can be an API gateway, service mesh, load balancer, inference platform, or application-level dispatcher. Good canaries use stable assignment, such as hashing a user or account id, so a user does not bounce between models on every request. Promotion usually increases exposure in steps: one percent, five percent, twenty-five percent, then full traffic. At each step, gates compare request errors, latency, saturation, business guardrails, and model-specific signals such as score drift, calibration, approval rate, false-positive review volume, or override rate.

In a blue-green deployment, two complete environments exist side by side. Blue is currently active; green is prepared with the new model, container image, configuration, feature schema, and dependencies. A switch changes the production route from blue to green after health checks and smoke tests pass. The old environment stays warm for rollback until confidence is high. Blue-green is less about gradual exposure and more about atomic routing and fast recovery.

Configuration Anatomy

All three patterns share a few pieces of release anatomy. A candidate identifier should name the model artifact and serving configuration, not just the container image. Traffic selection must specify who can be exposed: all users, an internal tenant, a region, a risk tier, or a deterministic hash bucket. Metrics must be separated by model version and route, otherwise aggregate dashboards hide regressions. Gates should include a decision window, a minimum sample size, an allowed error budget impact, and an owner who can stop or advance the rollout.

A practical deployment record usually contains current_model, candidate_model, feature_schema, traffic_policy, guardrail_metrics, promotion_steps, and rollback_target. For model services, add data-quality checks at the boundary: missing feature counts, category unknown rate, numeric range violations, and transformation version.

Example 1: Shadow Comparison

This example sends the same requests to a current model and a candidate model. Only the current score is returned. The shadow score is logged with a delta so the team can inspect distribution changes without affecting users.

from dataclasses import dataclass
from typing import Dict, List

@dataclass(frozen=True)
class Request:
    request_id: str
    features: Dict[str, float]

class ToyModel:
    def __init__(self, name: str, bias: float):
        self.name = name
        self.bias = bias

    def predict(self, features: Dict[str, float]) -> float:
        return round(features["amount"] * 0.01 + features["risk"] + self.bias, 3)

def run_shadow(primary: ToyModel, shadow: ToyModel, requests: List[Request]) -> None:
    for request in requests:
        primary_score = primary.predict(request.features)
        shadow_score = shadow.predict(request.features)
        print(f"{request.request_id}: returned={primary_score} shadow={shadow_score} delta={round(shadow_score - primary_score, 3)}")

requests = [
    Request("r1", {"amount": 40, "risk": 0.20}),
    Request("r2", {"amount": 70, "risk": 0.35}),
]
run_shadow(ToyModel("current", 0.00), ToyModel("candidate", 0.05), requests)

The deterministic output is r1: returned=0.6 shadow=0.65 delta=0.05 and r2: returned=1.05 shadow=1.1 delta=0.05. In a real service, the next step would be aggregating deltas by segment. A candidate that is only slightly higher overall may still be unacceptable if it shifts scores for one region, merchant category, or high-value customer tier.

Example 2: Stable Canary Routing

A canary must avoid random per-request flipping. The following dispatcher hashes a user id into one hundred buckets and sends only the first thirty buckets to the candidate. The same user keeps the same assignment as long as the percentage and identifier are unchanged.

from hashlib import sha256
from typing import Dict

def stable_bucket(user_id: str, percent: int) -> bool:
    if percent < 0 or percent > 100:
        raise ValueError("percent must be between 0 and 100")
    bucket = int(sha256(user_id.encode("utf-8")).hexdigest()[:8], 16) % 100
    return bucket < percent

def choose_model(user_id: str, canary_percent: int) -> str:
    return "candidate" if stable_bucket(user_id, canary_percent) else "current"

for user in ["ana", "bo", "cy", "di", "eli"]:
    print(f"{user}: {choose_model(user, 30)}")

For this user list and a thirty percent canary, the expected output is ana: current, bo: candidate, cy: current, di: current, and eli: current. If the canary is increased to fifty percent, some additional users may enter the candidate group, but users already assigned to the candidate remain there when the bucket rule is monotonic.

Example 3: Blue-Green Route Switch

Blue-green release logic should refuse to point production traffic at an environment that has failed health checks. This small example models the route switch and the rollback guard.

from dataclasses import dataclass

@dataclass
class Environment:
    name: str
    model_version: str
    healthy: bool

@dataclass
class Router:
    active: str

    def switch_to(self, target: Environment) -> None:
        if not target.healthy:
            raise RuntimeError(f"refusing to route to unhealthy {target.name}")
        self.active = target.name

blue = Environment("blue", "fraud-model-v12", True)
green = Environment("green", "fraud-model-v13", True)
router = Router(active=blue.name)
print(f"before: {router.active}")
router.switch_to(green)
print(f"after: {router.active}")
green.healthy = False
try:
    router.switch_to(green)
except RuntimeError as exc:
    print(str(exc))

The output is before: blue, after: green, and refusing to route to unhealthy green. The same rule belongs in production automation: route changes should be explicit, auditable, and blocked when the target serving environment is not ready.

Design Choices and Trade-offs

Shadowing has the lowest user risk because the candidate cannot change the product decision. Its cost is duplicate inference, extra telemetry volume, and the danger of overconfidence: shadow metrics prove technical behavior, not user reaction. It is best before canarying a model with a changed architecture, feature set, or latency profile.

Canarying gives the strongest evidence under real use, but it exposes real users to candidate behavior. The hardest design choice is the traffic unit. Request-level canaries collect samples quickly but can create inconsistent experiences. User-level, account-level, or entity-level canaries are slower but cleaner for analysis. Canary gates also need enough time to observe delayed labels. A fraud model may show immediate latency and approval-rate changes, while chargeback labels arrive much later, so early promotion must rely on proxy guardrails.

Blue-green deployments offer fast switch and rollback, but they require two compatible environments. If the candidate needs a non-backward-compatible feature schema or writes new state that blue cannot read, rollback may only appear easy. For model serving, blue-green works best when feature contracts are backward compatible or when a migration plan allows both environments to operate during the release window.

Failure Modes and Troubleshooting

Shadow predictions are missing. The symptom is normal primary traffic with sparse candidate telemetry. Common causes are asynchronous queue drops, timeout budgets that cancel the shadow call, or sampling rules that were enabled only for one route. Diagnose by tracing a single request id through the gateway, feature service, primary prediction, shadow prediction, and metrics sink. Correct by making shadow dispatch observable, bounding retries, and emitting a counter for every skipped shadow request with a reason.

The canary looks healthy but complaints rise. The symptom is acceptable aggregate latency and error rate while support tickets or manual overrides increase. The cause is often a segment hidden by averages: a geography, tenant, language, device, or risk band received worse predictions. Diagnose by breaking guardrails down by stable segment and comparing candidate exposure against the current model for the same time window. Correct by rolling back or narrowing the canary, then adding segment-specific gates before the next promotion.

Rollback after blue-green does not restore behavior. The symptom is traffic routed back to blue while predictions or errors remain changed. Causes include a shared feature transformation updated in place, a cache warmed with candidate outputs, or a downstream schema change made during the release. Diagnose by comparing active route, model version header, feature schema version, cache keys, and recent configuration changes. Correct by rolling back shared dependencies or clearing incompatible cache entries, then separate environment-specific configuration from global state.

Security, Reliability, and Performance

Shadow deployments must protect sensitive data because they intentionally duplicate real requests. The shadow path should use the same authorization boundary as the primary path, avoid logging raw payloads, and store only identifiers and bounded feature summaries when possible. Canary controls should be restricted to release operators; otherwise an accidental percentage change can become a production incident.

Performance cost is not theoretical. Shadowing can double model inference load. Canarying can overload the candidate if autoscaling is based on total service traffic instead of per-version traffic. Blue-green can hide cold-start behavior if green is tested only with synthetic probes. Reliability improves when health checks include model loading, feature lookup, prediction, and a representative post-processing path, not merely an HTTP 200 from the container.

Hands-on Lab

Prerequisites: Python 3, a shell, and permission to create a temporary local directory. Create a file named rollout_lab.py with the three examples above, or run each block separately. Step 1: run the shadow example and confirm that the returned score always comes from the current model. Step 2: change the candidate bias from 0.05 to 0.50 and verify that shadow deltas grow while returned values do not change. Step 3: run the canary example twice and verify that each user receives the same assignment both times. Step 4: change the canary percentage from 30 to 0 and confirm that every user is routed to current. Step 5: run the blue-green example and confirm that the router refuses an unhealthy target.

Verification is the printed output from each step plus one written observation: which release pattern found the issue earliest and which pattern would have exposed users? Cleanup is deleting rollout_lab.py and any temporary directory. In a real platform, rollback is setting canary exposure to zero or switching the route back to the previous environment, then confirming metrics are tagged with the previous model version.

Assessment Exercises

1. A recommendation model has excellent offline precision but uses a new feature generated by a slow service. Which pattern would you use first, and which metrics would decide whether it can advance?
2. A canary shows no aggregate regression, but one enterprise tenant reports changed decisions. Explain how stable assignment and segmented metrics should have been configured.
3. Design a blue-green rollout for a model whose feature schema adds one optional field. What must be true for rollback to remain valid?
4. Why is shadowing insufficient evidence for a pricing model whose output changes customer behavior?
5. Write a promotion gate for a ten percent canary that includes both platform health and model behavior. Include sample size and rollback conditions.

Summary

Shadow, canary, and blue-green deployments are complementary controls for model release risk. Shadowing compares candidate predictions on real traffic without user impact. Canarying exposes a stable, limited population and promotes only when guardrails hold. Blue-green keeps complete environments ready so routing can move forward or back deliberately. The MLOps discipline is tying those routing choices to model versions, feature schemas, segmented metrics, and practiced rollback.