Serving Models and Monitoring Drift

Serving a model means turning a trained artifact into a bounded prediction service, then watching whether the world that service sees still resembles the data used to build it. In this lesson, the outcome is a small but realistic serving design: validate request features, keep latency predictable, measure input and performance drift, and decide when to retrain or roll back.

In the production part of this machine learning course, this topic sits between offline evaluation and operational response. A high validation score is only a starting point. The serving path can change feature values, delay labels, hide distribution shifts, or amplify a bad model version before anyone notices. The goal is to make those mechanisms explicit enough that failures are diagnosable.

Serving And Drift In Plain Terms

An online model service usually receives a request, builds the same feature vector used during training, runs preprocessing, calls the model, applies a decision threshold, and returns a prediction with metadata. Monitoring drift means comparing production traffic and outcomes against a reference window, such as the training set, a validation set, or last month of healthy traffic.

There are several distinct drift signals. Feature drift, also called covariate shift, means input distributions change. Prediction drift means the model’s scores or classes change distribution. Concept drift means the relationship between features and target changes, so the same feature vector now implies a different outcome. Data quality drift covers nulls, invalid categories, timestamp skew, duplicate requests, and unit changes. These are related, but they do not imply one another. Feature drift can be harmless, and severe concept drift may not be visible until labels arrive.

Internal Mechanism

A dependable serving system treats the trained model as one versioned component inside a larger contract. The contract includes the model artifact, preprocessing code, feature schema, reference statistics, decision threshold, timeout budget, and rollback target. Each prediction should record a model version and feature schema version so later label joins and drift investigations use the exact context that produced the decision.

The online path normally has five stages. First, a request parser validates required fields and converts external units into canonical units. Second, feature retrieval joins online store values such as recent counts or account age. Third, a transformer applies the fitted encoders, imputers, scalers, and column order from training. Fourth, the model returns a score or class. Fifth, policy code converts that score into an action and emits metrics.

Drift monitoring runs beside this path rather than inside the critical request latency budget. The service logs compact prediction events: timestamp, version, selected features, prediction, decision, latency, and request status. A batch or streaming monitor aggregates those events into windows. It compares current feature statistics with reference statistics using tests such as population stability index for binned numeric values, Jensen-Shannon divergence for categorical distributions, missing-rate deltas, quantile movement, and score distribution changes. When labels arrive, delayed performance monitoring computes metrics such as precision, recall, AUC, log loss, calibration error, or business loss by model version and segment.

API And Configuration Anatomy

A serving configuration should be boring and explicit. It names the model artifact, schema, threshold, latency budget, and monitoring rules. The threshold belongs in configuration because retraining is not required to change the operating point of a probabilistic classifier. Drift thresholds should be tied to an action, not just a chart.

from dataclasses import dataclass

@dataclass(frozen=True)
class ServingConfig:
    model_version: str
    schema_version: str
    threshold: float
    timeout_ms: int
    max_missing_rate: float
    psi_warn: float
    psi_fail: float

config = ServingConfig(
    model_version="fraud-model-2026-09-01",
    schema_version="fraud-features-v3",
    threshold=0.72,
    timeout_ms=80,
    max_missing_rate=0.02,
    psi_warn=0.10,
    psi_fail=0.25,
)
print(config.model_version, config.threshold)

This prints the active model version and threshold. In a real service, the same values would be attached to prediction logs and dashboards. The important design point is that model behavior can be explained after the fact.

Example 1: Validate Online Features

The first serving failure is often not the model. It is a mismatch between training columns and online request fields. This example validates a tiny feature schema before prediction. It rejects missing values, wrong types, and unknown plans.

from dataclasses import dataclass

VALID_PLANS = {"free", "team", "enterprise"}

@dataclass(frozen=True)
class CustomerRequest:
    account_age_days: int
    monthly_spend: float
    plan: str

def validate_request(payload: dict) -> CustomerRequest:
    required = {"account_age_days", "monthly_spend", "plan"}
    missing = required - payload.keys()
    if missing:
        raise ValueError(f"missing fields: {sorted(missing)}")
    if not isinstance(payload["account_age_days"], int) or payload["account_age_days"] < 0:
        raise ValueError("account_age_days must be a non-negative integer")
    if not isinstance(payload["monthly_spend"], (int, float)) or payload["monthly_spend"] < 0:
        raise ValueError("monthly_spend must be a non-negative number")
    if payload["plan"] not in VALID_PLANS:
        raise ValueError("plan is not recognized")
    return CustomerRequest(payload["account_age_days"], float(payload["monthly_spend"]), payload["plan"])

request = validate_request({"account_age_days": 40, "monthly_spend": 29.5, "plan": "team"})
print(request)

The deterministic output is CustomerRequest(account_age_days=40, monthly_spend=29.5, plan='team'). The model never sees malformed input. This also makes bad clients visible as validation errors instead of silently converting them into strange predictions.

Example 2: Score With A Latency Budget

A model endpoint should have a bounded response path. The timeout should cover feature lookup, transformation, and model inference, not just the final predict call. This example simulates a model and blocks a slow request before it can violate the service budget.

import time

class TinyChurnModel:
    def predict_proba(self, features: list[float]) -> float:
        account_age_days, monthly_spend, is_enterprise = features
        score = 0.65 - min(account_age_days, 365) / 1000 - monthly_spend / 500 + is_enterprise * 0.10
        return max(0.01, min(0.99, score))

def score_with_budget(model: TinyChurnModel, features: list[float], timeout_ms: int) -> tuple[float, int]:
    started = time.perf_counter()
    score = model.predict_proba(features)
    elapsed_ms = int((time.perf_counter() - started) * 1000)
    if elapsed_ms > timeout_ms:
        raise TimeoutError(f"prediction exceeded {timeout_ms} ms")
    return score, elapsed_ms

model = TinyChurnModel()
score, elapsed_ms = score_with_budget(model, [40.0, 29.5, 0.0], timeout_ms=80)
print(round(score, 3), elapsed_ms <= 80)

The output is a churn score and True for the latency check, for example 0.551 True. The exact elapsed milliseconds vary, so tests should check the budget property rather than an exact time.

Example 3: Detect Input Drift With PSI

Population stability index compares how many current observations fall into reference bins. It is common in risk and marketing models because it is easy to explain, but it depends on sensible bins and enough volume. Values near zero indicate similar distributions; larger values indicate movement.

import math

def population_stability_index(reference_counts: list[int], current_counts: list[int]) -> float:
    if len(reference_counts) != len(current_counts) or not reference_counts:
        raise ValueError("aligned non-empty bins are required")
    ref_total = sum(reference_counts)
    cur_total = sum(current_counts)
    if ref_total == 0 or cur_total == 0:
        raise ValueError("bin totals must be positive")
    psi = 0.0
    for ref, cur in zip(reference_counts, current_counts):
        ref_pct = max(ref / ref_total, 0.0001)
        cur_pct = max(cur / cur_total, 0.0001)
        psi += (cur_pct - ref_pct) * math.log(cur_pct / ref_pct)
    return psi

reference = [250, 250, 250, 250]
current = [100, 200, 300, 400]
psi = population_stability_index(reference, current)
print(round(psi, 3))

The expected output is 0.208. With the earlier configuration, this would exceed the warning threshold but not the fail threshold. The right response might be investigation, segment analysis, or shadow evaluation, not immediate retraining.

Example 4: Join Delayed Labels To Predictions

Performance drift requires labels, and labels often arrive later than predictions. Store a prediction identifier and model version at serving time, then join labels when they become available. This example computes accuracy for one model version.

predictions = [
    {"id": "p1", "model_version": "fraud-model-2026-09-01", "prediction": 1},
    {"id": "p2", "model_version": "fraud-model-2026-09-01", "prediction": 0},
    {"id": "p3", "model_version": "fraud-model-2026-09-01", "prediction": 1},
]
labels = {"p1": 1, "p2": 1, "p3": 1}

def accuracy_for_version(rows: list[dict], labels_by_id: dict[str, int], version: str) -> float:
    matched = [row for row in rows if row["model_version"] == version and row["id"] in labels_by_id]
    if not matched:
        raise ValueError("no labeled predictions for version")
    correct = sum(row["prediction"] == labels_by_id[row["id"]] for row in matched)
    return correct / len(matched)

print(round(accuracy_for_version(predictions, labels, "fraud-model-2026-09-01"), 2))

The expected output is 0.67. This is a toy metric, but the mechanism is production-shaped: preserve prediction identity, join delayed truth, and evaluate by version.

Design Choices And Trade-Offs

Batch serving is simpler and cheaper when predictions can wait, because feature generation and scoring can be scheduled together. Online serving is necessary for interactive decisions, but it adds latency budgets, dependency failures, and consistency problems between offline and online features. Streaming sits between them: it can update features or scores continuously, but it increases operational complexity.

Feature stores reduce training-serving skew by sharing definitions and materializing online values, but they are not magic. They still require timestamp correctness, backfill discipline, and tests that compare offline and online transformations. Embedding all preprocessing inside a model artifact can improve portability; keeping preprocessing in service code can improve transparency and rollback speed. Either way, column order, missing-value policy, categorical handling, and units must be versioned.

Monitoring thresholds also involve trade-offs. Sensitive drift alerts catch problems earlier but create noise during seasonal changes, campaigns, or product launches. Loose thresholds reduce alert fatigue but may let a degraded model make many bad decisions. A useful alert includes the affected feature, segment, model version, sample size, recent deploys, and a recommended operator action.

Failure Modes And Troubleshooting

Symptom: prediction scores suddenly cluster near one value. The likely cause is a feature pipeline returning defaults, such as zeros after an online store timeout. Diagnose by comparing missing rates, default rates, and feature-store latency before and after the change. Correct by failing closed for high-impact decisions, adding a circuit breaker for the feature dependency, and backfilling affected predictions for analysis.

Symptom: offline validation is strong, but online labels show poor recall. The cause may be training-serving skew or a threshold copied from an old class balance. Diagnose by replaying production requests through the offline pipeline and diffing transformed feature vectors column by column. Correct the inconsistent transformation, recalibrate scores, and choose a threshold from recent labeled traffic.

Symptom: drift alerts fire every Monday. The cause may be normal weekly seasonality rather than model damage. Diagnose by comparing the current window to the same day-of-week reference and checking business events. Correct by using seasonal baselines, segment-specific thresholds, or alert suppression rules that still preserve severe anomaly detection.

Symptom: dashboards show healthy input drift while business loss increases. This can be concept drift. The inputs look familiar, but the target relationship has changed. Diagnose with delayed labels, calibration plots, and segment-level error analysis. Correct with retraining data that includes the new regime, temporary threshold changes, or a rollback to a simpler policy if labels are too sparse.

Reliability, Security, And Performance

Reliability starts with bounded behavior. Use explicit timeouts, bounded retries, idempotent prediction identifiers, and a tested rollback route. Separate validation errors from internal failures so dashboards do not hide broken infrastructure inside client error rates. Log enough feature metadata to debug drift, but avoid storing raw personal data unless retention, access control, and purpose are clear.

Security matters because model endpoints are data-processing APIs. Validate payload size, reject unexpected fields when appropriate, authenticate callers, and avoid returning raw feature values or internal model details to untrusted clients. Performance work should focus on the whole request path: feature lookup fanout, serialization, model loading, vectorization, and cold starts. Caching can help stable features, but cached values must have freshness guarantees.

Hands-On Lab: Build A Drift Check

Prerequisites: Python 3, a terminal, and no external packages. Create a temporary file named serve_drift_lab.py. Add the PSI function from Example 3, then add two reference windows and one current window for a numeric feature such as transaction amount. Choose four bins: low, medium, high, and extreme.

1. Run the script with a stable current distribution such as [240, 260, 255, 245] against [250, 250, 250, 250]. Verify that PSI rounds close to zero.
2. Change the current distribution to [100, 200, 300, 400]. Verify that PSI prints approximately 0.208.
3. Add a decision rule: below 0.10 prints ok, from 0.10 to below 0.25 prints investigate, and 0.25 or above prints stop or shadow.
4. Record the feature name, window start, window end, sample count, and decision in the output. This makes the alert actionable rather than just numeric.
5. Cleanup by deleting the temporary file, or keep it as a local regression check and add more features.

Verification is successful when stable data produces ok, moderate movement produces investigate, and an extreme distribution such as [20, 80, 250, 650] produces the stop or shadow decision. If results differ, check that bin counts are aligned, totals are positive, and smoothing is applied before the logarithm.

Assessment Exercises

• A feature’s PSI jumps to 0.32, but labeled performance is not available yet. What should you inspect before deciding to retrain?
• Design a prediction log schema that supports delayed performance evaluation without storing unnecessary personal data.
• Explain why changing a classifier threshold can be safer than immediately retraining after a recall drop.
• Given an online feature value and an offline recomputed value for the same request, list the steps you would use to locate training-serving skew.
• Choose one business segment where global drift metrics could hide a serious model problem, and define a segment-level alert.

Summary

Serving models and monitoring drift is the discipline of keeping prediction behavior bounded, explainable, and measurable after deployment. The core mechanics are schema validation, versioned preprocessing, latency budgets, prediction logging, reference-window comparisons, delayed label joins, and action-oriented alerts. Drift is not automatically a retraining command; it is evidence that should trigger a defined investigation, threshold change, shadow test, rollback, or retraining workflow.