Capstone: Productionize a Churn Model

This capstone turns a churn model into a production decision system. The outcome is not merely a saved model file. It is a repeatable path from customer data to a scored retention action, with evidence that the score was produced by the expected data snapshot, feature definitions, model artifact, threshold, and service version.

In this MLOps course, earlier lessons covered training, packaging, deployment, monitoring, and governance separately. Here those pieces are joined around one concrete use case: predicting whether an active customer is likely to cancel service. A productionized churn model must answer three operational questions at any time: which customers were scored, why they received their probabilities, and what operators should do if the model or its data becomes unreliable.

Purpose and Outcome

A churn system usually supports retention teams, lifecycle marketing, or account management. The model estimates churn probability from recent customer behavior, billing history, support interactions, subscription tenure, and contract attributes. The production pipeline turns that probability into a ranked list, an API response, or a message sent to another system.

The target architecture for this capstone has six named parts: an ingestion job that freezes a scoring snapshot, a feature transformation step, a model artifact in a registry, a scoring interface, a promotion gate, and model monitoring. Each part owns a different risk. Ingestion protects against changing data during scoring. Feature transformation keeps training and inference logic aligned. The registry identifies the exact model. Scoring exposes the prediction. Promotion gates prevent weak candidates from replacing stronger ones. Monitoring detects data, service, and outcome degradation after release.

How the System Works Internally

Internally, a churn model is a function over a feature vector. A row such as customer tenure, monthly charges, support ticket count, and contract type is normalized into numeric inputs. The model then emits a probability, for example 0.8549, meaning the fitted model ranks this customer as high risk under the training objective. That number is not a guarantee that the individual customer will leave. It is a calibrated estimate only if the training data, target definition, and current population remain compatible.

The most important internal boundary is the target window. A common definition is: churn equals cancellation within the next 30 days, measured after an observation date. Features must come from data available on or before that observation date. If a feature includes information created after the observation date, the model learns from the future and will fail in production even when offline metrics look strong.

The second boundary is the feature contract. Each feature needs a name, type, null policy, valid range, encoding rule, and owner. For example, contract_type might accept only month_to_month, one_year, and two_year. If a new product plan sends trial without a contract update, a one-hot encoder may create an all-zero category and silently change predictions.

The third boundary is artifact identity. A deployable package should identify the model bytes, training data snapshot, feature code revision, dependency image, threshold policy, and evaluation report. Without those identifiers, an operator cannot distinguish a model regression from a feature pipeline regression.

API and Configuration Anatomy

A practical churn deployment normally has a batch scoring job, a synchronous scoring API, or both. Batch scoring is common when retention teams need a daily prioritized queue. An API is useful when a customer-facing or agent-facing application needs a current risk score. The configuration should name the artifact URI, feature schema version, score threshold, maximum batch size, output destination, and rollback candidate.

A minimal scoring response should include customer_id, churn_probability, risk_band, model_version, and scored_at. Avoid returning raw sensitive features unless the consumer has a specific need. For explanation, prefer reviewed reason codes such as high support volume or short tenure over unrestricted dumps of feature values.

Example 1: Validate the Feature Contract

The first example catches invalid customer rows before they enter training or scoring. The expected output shows two errors for the second customer: negative tenure and an unknown contract type.

from dataclasses import dataclass
from typing import Iterable

@dataclass(frozen=True)
class CustomerRow:
    customer_id: str
    tenure_months: int
    monthly_charges: float
    support_tickets_90d: int
    contract_type: str

ALLOWED_CONTRACTS = {"month_to_month", "one_year", "two_year"}

def validate_rows(rows: Iterable[CustomerRow]) -> list[str]:
    errors: list[str] = []
    for row in rows:
        if not row.customer_id:
            errors.append("customer_id is required")
        if row.tenure_months < 0:
            errors.append(f"{row.customer_id}: tenure_months must be nonnegative")
        if row.contract_type not in ALLOWED_CONTRACTS:
            errors.append(f"{row.customer_id}: unknown contract_type")
    return errors

sample = [
    CustomerRow("C001", 14, 72.10, 1, "month_to_month"),
    CustomerRow("C002", -1, 39.20, 0, "trial"),
]
print(validate_rows(sample))

This example is intentionally small, but the production pattern is the same. Validation belongs before expensive training, before writing score tables, and before publishing campaign audiences. The correction is not to coerce every bad value. Some values should be rejected so the upstream data owner sees the break.

Example 2: Serve a Deterministic Score

The second example represents the serving layer. It applies a fixed transformation and model formula to one customer. The deterministic output is 0.8549.

from math import exp

def sigmoid(value: float) -> float:
    return 1 / (1 + exp(-value))

def churn_score(features: dict[str, float]) -> float:
    logit = -1.2
    logit += 0.035 * features["monthly_charges"]
    logit += 0.42 * features["support_tickets_90d"]
    logit -= 0.055 * features["tenure_months"]
    logit += 0.80 * features["is_month_to_month"]
    return round(sigmoid(logit), 4)

request_features = {
    "monthly_charges": 72.10,
    "support_tickets_90d": 1,
    "tenure_months": 14,
    "is_month_to_month": 1,
}
print(churn_score(request_features))

A real model may be a tree ensemble or neural network rather than this simple formula, but the serving requirements are identical: stable feature names, stable numeric meaning, bounded latency, and consistent preprocessing between training and inference. If the API receives missing support_tickets_90d, it should fail with a typed client error or apply a documented imputation rule, not guess differently per service instance.

Example 3: Gate Model Promotion

The third example models release control. The candidate must pass global AUC, calibration, segment performance, and lineage requirements. The expected output is promote to staging.

from dataclasses import dataclass

@dataclass(frozen=True)
class CandidateModel:
    name: str
    auc: float
    calibration_error: float
    min_segment_auc: float
    data_snapshot: str


def promotion_decision(candidate: CandidateModel) -> str:
    if not candidate.data_snapshot:
        return "reject: missing lineage"
    if candidate.auc < 0.78:
        return "reject: auc below gate"
    if candidate.calibration_error > 0.06:
        return "reject: probability calibration too weak"
    if candidate.min_segment_auc < 0.70:
        return "reject: weak segment performance"
    return "promote to staging"

model = CandidateModel("churn-xgb-2026-09", 0.812, 0.041, 0.724, "warehouse://snapshots/churn/2026-09-01")
print(promotion_decision(model))

The segment gate matters because a model with strong average AUC can still perform poorly for a region, product tier, or customer age band. Promotion should create an immutable staging entry, not overwrite production in place. The production alias can later move to the approved artifact after smoke tests and stakeholder review.

Example 4: Monitor Input Drift

The fourth example computes a population stability index for tenure buckets. The expected output is 0.0954, a moderate signal that should be investigated but may not require immediate rollback by itself.

from math import log

def population_stability_index(expected: list[float], actual: list[float]) -> float:
    if len(expected) != len(actual):
        raise ValueError("bucket counts must have the same length")
    total_expected = sum(expected)
    total_actual = sum(actual)
    psi = 0.0
    for exp_count, act_count in zip(expected, actual):
        exp_share = max(exp_count / total_expected, 0.0001)
        act_share = max(act_count / total_actual, 0.0001)
        psi += (act_share - exp_share) * log(act_share / exp_share)
    return round(psi, 4)

training_tenure_buckets = [1200, 980, 760, 430]
production_tenure_buckets = [900, 870, 810, 790]
print(population_stability_index(training_tenure_buckets, production_tenure_buckets))

Drift is a symptom, not a diagnosis. Tenure distribution may shift because acquisition strategy changed, a billing migration loaded old customers incorrectly, or a feature job filtered accounts differently. A useful alert links the drifted feature to the data source, recent deployments, and the model versions affected.

Design Choices and Trade-Offs

Batch scoring is easier to audit and cheaper at high volume, but the scores age between runs. API scoring is fresher and integrates into interactive products, but it introduces latency budgets, availability requirements, and more complex feature retrieval. Many churn systems use batch scores for campaigns and API scores for account pages.

Threshold choice is also a business decision. A low threshold catches more potential churners but may waste retention incentives on customers who would have stayed. A high threshold concentrates effort on the riskiest customers but misses borderline cases. Track precision, recall, treatment capacity, and intervention cost together.

Retraining can be scheduled or triggered. Scheduled retraining is predictable and easier to govern. Triggered retraining reacts to drift or metric decay but can amplify bad data if the trigger is caused by an upstream incident. In both cases, retraining must pass the same evaluation and promotion gates as the first release.

Failure Modes and Troubleshooting

Symptom: daily churn scores suddenly rise for nearly every customer. Likely cause: a feature default changed, such as missing contract type encoded as month-to-month. Diagnostics: compare feature null rates, category counts, and score distributions against the last healthy run. Check the feature schema version in the score table. Correction: stop publishing the affected run, restore the previous feature transformation, backfill scores, and add a contract test for the missing category.

Symptom: offline AUC improves but campaign conversion drops. Likely cause: target leakage or an evaluation split that allowed future information. Diagnostics: inspect feature timestamps relative to the observation date and rebuild evaluation with a time-based split. Correction: remove leaking features, retrain from a clean snapshot, and block promotion unless temporal validation passes.

Symptom: the scoring API times out during peak account-service traffic. Likely cause: online feature lookup depends on slow warehouse queries or cold model loading per request. Diagnostics: inspect p95 latency by stage: request parsing, feature fetch, model inference, and response serialization. Correction: cache stable features, pre-load the model at process start, set bounded timeouts, and degrade to the last batch score when the online path is unhealthy.

Security, Performance, and Reliability

Churn features often contain sensitive commercial behavior, billing state, and support history. Use service identities with read access only to approved feature sources and write access only to approved score destinations. Logs should include customer identifiers only when policy permits it, and should avoid raw free-text support notes.

For performance, measure records per minute for batch scoring and p95 latency for online scoring with realistic feature retrieval. Model inference is often not the bottleneck; joins, network calls, serialization, and warehouse contention are frequent causes. Reliability depends on idempotent scoring runs. A rerun for the same snapshot and model version should replace or reproduce the same partition instead of appending duplicate scores.

Hands-On Lab

Prerequisites: Python 3, a local folder for the lab, and permission to create temporary files. No external service is required.

  1. Create a small CSV with five customers containing customer_id, tenure_months, monthly_charges, support_tickets_90d, and contract_type.
  2. Implement the validation from Example 1 and reject any row with invalid tenure or contract type.
  3. Transform contract_type into is_month_to_month, then score valid rows with Example 2.
  4. Write a score file with customer id, probability, risk band, model version, and scoring timestamp.
  5. Record lineage in a separate JSON file: input filename, row count, model version, feature schema version, and code revision placeholder.
  6. Run the promotion gate from Example 3 for a candidate model and save the decision beside the lineage file.

Verification: confirm that invalid rows are excluded, every valid row has exactly one score, probabilities are between 0 and 1, the model version appears in every output row, and the lineage row count matches the number of scored customers. Re-run the same input and confirm the same customer probabilities. Cleanup: delete the temporary CSV, score file, and lineage file, or keep them as evidence if this lab is being reviewed.

Assessment Exercises

  1. A retention team asks to lower the high-risk threshold from 0.80 to 0.55. Which metrics and capacity constraints would you inspect before approving the change?
  2. A new contract type appears in production data. Design the safest short-term behavior for scoring and the longer-term change needed in training.
  3. Your candidate model has higher AUC but worse calibration. Explain when you would reject it, recalibrate it, or change the downstream decision rule.
  4. Design a rollback plan for a batch churn scoring job that already published bad scores to a campaign table.
  5. Choose batch scoring, online scoring, or both for an account-management dashboard, and justify the trade-off using freshness, latency, and auditability.

Summary

Productionizing a churn model means controlling the full path from observation date to retention action. The model artifact matters, but so do feature contracts, temporal validation, promotion gates, score identity, monitoring, and rollback. A dependable churn system rejects bad inputs early, promotes only traceable candidates, measures data and outcome changes after release, and gives operators a precise recovery path when scores are wrong or unavailable.