Module 8 · ML in Production

Capstone: Build a Churn Prediction System

Lesson 24 of 24 12 min read Updated

A churn prediction system estimates which active customers are likely to cancel within a defined future window so the business can choose a timely retention action. The outcome of this capstone is not just a classifier. It is a repeatable workflow that defines churn, builds customer snapshots, trains a leakage-safe model, chooses an intervention threshold, and verifies that the system can be monitored after release.

In the ML in Production section, churn is useful because it makes production trade-offs concrete. The prediction affects who receives discounts, support outreach, or product education. A technically accurate model can still fail if the target window is wrong, if features use data from after the decision time, or if the threshold creates more outreach than the team can handle.

Purpose and Decision Outcome

Start with the decision, not the algorithm. A practical churn system asks: given what we know about a customer at an as_of time, should we intervene before the customer cancels in the next N days? The answer usually becomes a ranked list, a probability, and an action policy. Accounts above 0.70 probability might receive a customer-success call, while accounts between 0.40 and 0.70 receive an educational email.

The main entities are customer_id, as_of, features observed at or before that time, a future churn label, a fitted model, a score, and a threshold. The target horizon must match the intervention. A 7-day horizon fits urgent outreach. A 90-day horizon may fit renewal planning but is noisier for low-cost subscriptions.

How the System Works Internally

The mechanism is a time-indexed supervised learning problem. First, create snapshots. Each snapshot represents one customer at one decision time. Second, attach a label by looking forward from that snapshot: did cancellation happen within the horizon? Third, compute features using only history available up to the snapshot. Fourth, split and validate in time order so evaluation resembles future scoring. Fifth, convert calibrated probabilities into an intervention policy.

Leakage is the central danger. If the feature table includes days_until_cancel, a refund code created during offboarding, or a customer status updated after the horizon begins, the model may look excellent offline and fail in production. The strict rule is: every feature must be reproducible at scoring time before the action is chosen.

A typical pipeline has a snapshot builder, feature transformations, estimator, threshold policy, and monitoring job. Transformations handle missing values, one-hot encode categorical fields such as plan, and scale numeric fields when the estimator benefits from it. The threshold policy converts probability to action. Monitoring compares current feature distributions, score distributions, and delayed churn outcomes against the training baseline.

API and Configuration Anatomy

A minimal batch scoring interface accepts a table with required columns and returns customer_id, as_of, churn_probability, recommended_action, and model_version. The training interface needs historical snapshots and labels. The deployment artifact should include the fitted preprocessing steps with the estimator, because scoring must use the same category encoding and numeric scaling learned during training.

Important configuration values are the horizon, snapshot cadence, feature list, split strategy, evaluation metrics, threshold, and capacity limit. If the retention team can call 500 customers per week, a top-500 policy may be more useful than a fixed threshold that sometimes selects 200 customers and sometimes 5,000.

Example 1: Create a Future Churn Label

This example labels each customer snapshot by checking whether cancellation occurred one to thirty days after the as_of date. Customer B is labeled churned because cancellation happened within the window. Customer C is not labeled churned for this target because the cancellation is later than the horizon.

import pandas as pd

snapshots = pd.DataFrame([
    {"customer_id": "A", "as_of": "2026-01-31", "plan": "basic", "tenure_days": 20, "tickets_30d": 0, "active_days_30d": 18, "monthly_spend": 29, "cancel_date": None},
    {"customer_id": "B", "as_of": "2026-01-31", "plan": "pro", "tenure_days": 180, "tickets_30d": 4, "active_days_30d": 3, "monthly_spend": 99, "cancel_date": "2026-02-12"},
    {"customer_id": "C", "as_of": "2026-01-31", "plan": "basic", "tenure_days": 75, "tickets_30d": 1, "active_days_30d": 12, "monthly_spend": 29, "cancel_date": "2026-03-25"},
])

snapshots["as_of"] = pd.to_datetime(snapshots["as_of"])
snapshots["cancel_date"] = pd.to_datetime(snapshots["cancel_date"])
horizon_days = 30
snapshots["churn_30d"] = (
    snapshots["cancel_date"].notna()
    & ((snapshots["cancel_date"] - snapshots["as_of"]).dt.days.between(1, horizon_days))
).astype(int)

print(snapshots[["customer_id", "as_of", "cancel_date", "churn_30d"]].to_string(index=False))

The deterministic output is a three-row table where customer A has churn_30d equal to 0, customer B has 1, and customer C has 0. This label is valid only if the prediction would truly be made on 2026-01-31 and if every feature in the same row was known by then.

Example 2: Train a Leakage-Safe Pipeline

The next example keeps preprocessing inside the model pipeline. Categories and scaling statistics are learned from training data and then reused consistently at scoring time. handle_unknown="ignore" prevents scoring from crashing when a new plan value appears, although monitoring should still report that new category.

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

train = pd.DataFrame([
    {"plan": "basic", "tenure_days": 20, "tickets_30d": 0, "active_days_30d": 18, "monthly_spend": 29, "churn_30d": 0},
    {"plan": "pro", "tenure_days": 180, "tickets_30d": 4, "active_days_30d": 3, "monthly_spend": 99, "churn_30d": 1},
    {"plan": "basic", "tenure_days": 75, "tickets_30d": 1, "active_days_30d": 12, "monthly_spend": 29, "churn_30d": 0},
    {"plan": "enterprise", "tenure_days": 300, "tickets_30d": 6, "active_days_30d": 2, "monthly_spend": 499, "churn_30d": 1},
    {"plan": "pro", "tenure_days": 40, "tickets_30d": 0, "active_days_30d": 22, "monthly_spend": 99, "churn_30d": 0},
    {"plan": "basic", "tenure_days": 15, "tickets_30d": 3, "active_days_30d": 1, "monthly_spend": 29, "churn_30d": 1},
])

features = ["plan", "tenure_days", "tickets_30d", "active_days_30d", "monthly_spend"]
X = train[features]
y = train["churn_30d"]
preprocess = ColumnTransformer([
    ("plan", OneHotEncoder(handle_unknown="ignore"), ["plan"]),
    ("numbers", StandardScaler(), ["tenure_days", "tickets_30d", "active_days_30d", "monthly_spend"]),
])
model = Pipeline([("preprocess", preprocess), ("classifier", LogisticRegression(max_iter=1000))])
model.fit(X, y)
new_accounts = pd.DataFrame([
    {"plan": "pro", "tenure_days": 35, "tickets_30d": 2, "active_days_30d": 4, "monthly_spend": 99},
    {"plan": "basic", "tenure_days": 90, "tickets_30d": 0, "active_days_30d": 20, "monthly_spend": 29},
])
print([round(p, 3) for p in model.predict_proba(new_accounts)[:, 1]])

The printed probabilities show relative churn risk for two new accounts. The first account should score higher because it has low recent activity and support friction, while the second should score lower because it is active and has no recent tickets.

Example 3: Choose an Intervention Threshold

A churn probability is not yet a decision. The threshold determines how many customers are contacted and how many true churners are missed. Lowering the threshold increases recall but may waste outreach on customers who were not going to churn. Raising it improves precision but leaves more churners untouched.

import pandas as pd
from sklearn.metrics import confusion_matrix, precision_score, recall_score

scores = pd.DataFrame({
    "customer_id": ["A", "B", "C", "D", "E", "F", "G", "H"],
    "probability": [0.91, 0.76, 0.64, 0.55, 0.43, 0.31, 0.22, 0.08],
    "churned": [1, 1, 0, 1, 0, 0, 1, 0],
})

for threshold in [0.30, 0.50, 0.70]:
    predicted = (scores["probability"] >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(scores["churned"], predicted, labels=[0, 1]).ravel()
    print(
        f"threshold={threshold:.2f} contacted={int(predicted.sum())} "
        f"precision={precision_score(scores['churned'], predicted):.2f} "
        f"recall={recall_score(scores['churned'], predicted):.2f} "
        f"tp={tp} fp={fp} fn={fn} tn={tn}"
    )

Expected output has three lines. At threshold 0.30, six customers are contacted, precision is 0.50, and recall is 0.75. At 0.50, four are contacted with the same precision and lower recall. At 0.70, two are contacted with precision 1.00 and recall 0.50. The right choice depends on intervention cost, customer experience, and team capacity.

Design Choices and Trade-offs

Logistic regression is transparent, quick to retrain, and easy to calibrate, but it may miss nonlinear patterns such as a sharp risk jump when activity falls below a product-specific threshold. Tree ensembles often improve ranking performance and handle nonlinear interactions, but they are harder to explain and can overfit small segments. Deep learning is rarely the first choice for tabular churn unless the feature set includes rich sequences or text.

The target definition is another trade-off. Account cancellation is easy to observe but may arrive too late for intervention. Inactivity is earlier but may confuse seasonal users with true churners. Subscription downgrades may matter economically even when the customer does not fully cancel. The model should predict the event that the business can act on and measure.

Finally, decide whether the output is a fixed threshold, a capacity-ranked list, or multiple action tiers. A fixed threshold is easy to explain. A ranked list respects limited retention capacity. Tiers allow different treatments, but they require careful testing so high-risk customers do not all receive expensive incentives that would have been unnecessary.

Failure Modes and Troubleshooting

Symptom: offline AUC is near perfect, but production saves few customers. Cause: leakage from cancellation status, refund events, or post-decision support activity. Diagnostics: inspect the top features, compare each feature timestamp to as_of, and rebuild a small row manually. Correction: enforce point-in-time joins and block any column whose source updates after the snapshot.

Symptom: the batch scoring job fails when a new plan launches. Cause: categorical encoding was fit outside the deployed pipeline or unknown categories are rejected. Diagnostics: check the failing row, list category values seen in training, and run the scorer with the new value in isolation. Correction: package preprocessing with the model and define a monitored fallback for unknown categories.

Symptom: outreach volume doubles after a marketing campaign. Cause: the score distribution shifted because many customers changed behavior, while the threshold stayed fixed. Diagnostics: compare weekly score histograms, feature drift, and campaign enrollment. Correction: use a capacity cap, recalibrate after outcomes mature, or adjust campaign-specific policy rules.

Reliability, Performance, and Responsible Use

Reliability depends on deterministic feature generation. Batch jobs should record the data extraction time, feature code version, model version, and row counts at each stage. Delayed labels mean dashboards must separate immediate operational metrics, such as scoring latency and outreach volume, from outcome metrics, such as retained revenue, that arrive weeks later.

Performance concerns are usually data related. A daily batch model can score millions of rows if features are precomputed and the model artifact is loaded once. Real-time scoring is useful when the product experience changes immediately, but it requires low-latency feature access and a clear fallback when features are unavailable.

Responsible use matters because churn scores can affect pricing, support priority, or incentives. Do not include sensitive attributes without reviewed justification. Check whether error rates and intervention rates differ across meaningful customer segments. Keep a human-review path for high-impact decisions.

Hands-on Lab: Build and Verify a Churn Prototype

Prerequisites: Python with pandas and scikit-learn installed, plus a CSV or DataFrame containing one row per customer snapshot. Required columns are customer_id, as_of, plan, tenure_days, tickets_30d, active_days_30d, monthly_spend, and cancel_date.

  1. Define the target horizon, such as 30 days, and write it in the project README or notebook header.
  2. Parse as_of and cancel_date as dates. Create churn_30d by looking forward from as_of, as shown in Example 1.
  3. Remove identifiers, raw dates, and any field that would not be known at scoring time from the feature list.
  4. Sort rows by as_of. Use earlier rows for training and later rows for validation, or use time-series cross-validation.
  5. Build one scikit-learn Pipeline containing preprocessing and the classifier. Fit it only on the training window.
  6. Score the validation window, compute ROC AUC and average precision, then print a threshold table like Example 3.
  7. Select an action policy: a threshold, a top-N list, or tiers. Record the expected weekly contact volume.
  8. Save the feature list, horizon, threshold, and model version next to the model artifact.

Verification: choose five validation rows and confirm manually that no feature uses information after as_of. Confirm the scorer returns one probability per input row and that unknown plan values do not crash scoring. Compare contacted volume under at least three thresholds.

Cleanup or rollback: delete temporary exported customer files, remove notebook outputs containing customer-level data, and keep only aggregated metrics or approved artifacts. If a deployed threshold causes excessive outreach, roll back to the previous threshold or switch to a top-N capacity policy while outcomes mature.

Assessment Exercises

  • You discover a feature called days_since_last_login. What questions must you ask before deciding whether it is leakage-safe?
  • A model with lower ROC AUC produces higher retained revenue in an experiment. Explain how that can happen and which metric should guide the rollout.
  • Design a threshold policy for a team that can call only 100 accounts per week while an email campaign can reach 10,000 accounts.
  • List three monitoring charts you would add for the first month after launch and explain what action each chart would trigger.
  • Given a spike in false positives for enterprise customers, outline the diagnostic steps before retraining the whole model.

Summary

A churn prediction system is a time-aware decision system. Its quality depends on a clear churn definition, point-in-time features, realistic validation, and an action policy matched to business capacity. The model ranks customers by risk, but the production system must also control leakage, handle new categories, monitor drift, and measure whether interventions actually reduce churn.

Progress is saved in this browser.