Model Interpretation

Model interpretation explains how a trained model converts input features into a prediction. The useful outcome is the ability to answer specific questions: which features moved this prediction, which features matter on held-out data, how a prediction changes when one feature is varied, and where an explanation should not be trusted.

In this responsible ML section, interpretation connects model quality to model accountability. Evaluation says whether a model is often right. Interpretation helps inspect whether it is right for defensible reasons, whether it relies on proxies, and whether a reviewer can reproduce the explanation from the same model, data, and preprocessing pipeline.

What Interpretation Measures

An interpretation method describes a fitted function, not the world itself. If a credit model gives high importance to postal code, the precise claim is that postal code changes the model’s predictions under the measured data distribution. It is not proof that postal code causes repayment behavior. Many methods perturb data in ways that may be unrealistic outside the training distribution.

Four families are especially useful in everyday Python machine learning. Coefficients describe parameters of linear and generalized linear models. Permutation importance measures how much a metric worsens when one feature is shuffled. Partial dependence estimates average predictions while one feature is forced to selected values. Local explanations decompose one prediction into feature-level effects around a single row or neighborhood.

Mechanism and Internals

A fitted model is a function from a feature vector to an output. Interpretation starts by deciding whether you inspect the function directly, as coefficients do, or probe it from the outside, as permutation and partial dependence do. Direct inspection is fast and exact for models with readable internals. External probing works across model types but depends on sampled rows, metrics, perturbations, and feature correlations.

For a linear model, the prediction is an intercept plus feature values multiplied by learned weights. With standardized features, coefficient magnitude is often comparable: a weight of -1.2 moves the score more than a weight of 0.08 for a one-standard-deviation change. Without standardization, units dominate magnitude; a coefficient on annual income in dollars can look tiny because one dollar is a tiny unit.

Permutation importance uses a validation set. First compute a baseline metric. Then shuffle one feature column, keeping every other column and the target fixed, and score again. If the metric degrades sharply, the model depended on that feature for that dataset and metric. The method captures interactions, but it can understate one of two correlated features when either can substitute for the other.

Partial dependence asks a counterfactual average question. For a grid value such as missed_payments = 2, copy the validation rows, replace that feature with 2 in every row, predict, and average the predictions. Repeating this over a grid gives a curve. The weakness is that it may create impossible rows, such as high balance with an account type that cannot hold that balance.

Local explanations focus on one prediction. For linear models, a feature contribution is value times coefficient, often relative to a baseline. For trees and ensembles, specialized methods trace how splits move a row away from an expected value. Model-agnostic local methods often fit a small surrogate model around perturbed copies of the row. The key design question is the neighborhood: explanations change when perturbations do not resemble valid nearby examples.

API Anatomy

A reliable interpretation workflow has five inputs: the trained model, the exact preprocessing used during training, a reference dataset, feature names after transformation, and the metric or output scale being explained. The output should say whether it is global or local, model-specific or model-agnostic, computed on raw or transformed features, and measured on score, probability, log-odds, loss, or class label.

In Python libraries, the ideas appear as methods such as coef_, feature_importances_, permutation_importance(...), PartialDependenceDisplay, and explainer objects with an explain or shap_values style of call. Interpret a pipeline or preserve the feature-name mapping from preprocessing. One-hot encoded columns should not be reported as if they were unchanged raw categorical fields.

Example 1: Coefficients and Local Contributions

This example treats a linear score as the model. The sign tells direction on the model’s score scale, and weight times feature value gives the local contribution for this row.

weights = {"income": 0.08, "debt_ratio": -1.20, "late_payments": -0.70}
features = {"income": 2.0, "debt_ratio": 0.5, "late_payments": 1.0}
intercept = -0.10
score = intercept + sum(weights[name] * features[name] for name in weights)
contributions = {name: weights[name] * features[name] for name in weights}
print(round(score, 2))
print(contributions)

The score is -1.24. Income adds 0.16, debt ratio subtracts 0.6, and late payments subtract 0.7. The largest downward movement comes from late payments because its learned weight and row value combine to make the largest contribution. Comparing coefficient magnitudes alone would require compatible feature scaling.

Example 2: Permutation Importance

Here the model predicts house price from rooms and repair status. The baseline error is low because the prediction rule nearly matches the targets. Shuffling rooms breaks the relationship between rooms and price while preserving the room values themselves.

def mean_absolute_error(actual, predicted):
    return sum(abs(a - p) for a, p in zip(actual, predicted)) / len(actual)

def predict_price(row):
    return 100 + 30 * row["rooms"] - 20 * row["needs_repair"]

rows = [
    {"rooms": 2, "needs_repair": 0, "price": 160},
    {"rooms": 3, "needs_repair": 0, "price": 191},
    {"rooms": 4, "needs_repair": 1, "price": 199},
    {"rooms": 5, "needs_repair": 1, "price": 229},
]
baseline = mean_absolute_error([r["price"] for r in rows], [predict_price(r) for r in rows])
permuted = [dict(r) for r in rows]
shuffled_rooms = [5, 2, 3, 4]
for row, value in zip(permuted, shuffled_rooms):
    row["rooms"] = value
permuted_error = mean_absolute_error([r["price"] for r in rows], [predict_price(r) for r in permuted])
print(round(baseline, 2))
print(round(permuted_error - baseline, 2))

The baseline mean absolute error is 0.75. After permuting rooms, the error increases by 44.0. That is the importance estimate for this validation sample, prediction function, and metric. It says the model’s validation error becomes much worse when room information is made uninformative; it does not say rooms cause prices to rise by 44.0.

Example 3: Partial Dependence and a Local Delta

This example computes average predicted risk while forcing missed_payments to 0, 1, and 2 for every row in a small population. It then compares one person’s risk with and without a missed payment.

def predict_risk(age, balance, missed_payments):
    score = -4.0 + 0.04 * age + 0.0003 * balance + 1.1 * missed_payments
    return 1 / (1 + 2.718281828 ** (-score))

population = [
    {"age": 25, "balance": 2000, "missed_payments": 0},
    {"age": 40, "balance": 5000, "missed_payments": 1},
    {"age": 60, "balance": 9000, "missed_payments": 0},
]
for value in [0, 1, 2]:
    changed = [dict(row, missed_payments=value) for row in population]
    average = sum(predict_risk(**row) for row in changed) / len(changed)
    print(value, round(average, 3))

person = {"age": 40, "balance": 5000, "missed_payments": 1}
base = predict_risk(40, 5000, 0)
local = predict_risk(**person)
print(round(local - base, 3))

The partial dependence values rise from 0.374 to 0.555 to 0.733, so the fitted function is monotonic over this grid. The local delta is 0.261, meaning this person’s probability is 0.261 higher than it would be with zero missed payments. The curve averages over the reference population; the delta describes one row.

Design Choices and Trade-Offs

Choose model-specific interpretation when the model family provides faithful internals: coefficients for linear models, split paths for decision trees, or additive components for generalized additive models. Choose model-agnostic interpretation when you need the same procedure across model classes or when the deployed object is only available through a prediction API. The cost is more computation and more assumptions about perturbations.

Choose the reference dataset deliberately. A training set reveals what the model learned, but a validation or recent monitoring sample better reflects current use. For fairness review, compute explanations separately for relevant groups because global averages can hide group-specific reliance on proxies. For correlated features, consider grouped permutation importance, conditional permutation, or domain-driven feature removal experiments.

Output scale is also a design choice. Explaining a logistic classifier on log-odds gives additive contributions, while explaining probabilities is more intuitive but nonlinear. For a thresholded class label, tiny score changes near the threshold can look huge and large score changes far from the threshold can look irrelevant. Record the scale directly in plots, tables, and review notes.

Failure Modes and Troubleshooting

Symptom: an importance table ranks an obviously leaky field first, such as approved_at in a loan model. Cause: target leakage, where the feature was created after, or too close to, the outcome. Diagnose by checking feature timestamps and recomputing importance after removing post-outcome fields. Correct it by rebuilding the dataset with only prediction-time information.

Symptom: a coefficient table shows unreadable names such as x17 or hundreds of one-hot columns. Cause: feature lineage was lost during preprocessing. Diagnose by inspecting transformed feature names and comparing them with raw schema names. Correct it by exporting feature names from preprocessing and aggregating one-hot or text-vector columns when the human question concerns the original field.

Symptom: a partial dependence curve shows behavior that domain experts say is impossible. Cause: correlated features are being varied independently. Diagnose by checking whether grid values create rows outside the joint distribution, using scatter plots, nearest-neighbor distances, or group min/max checks. Correct it with accumulated local effects, conditional partial dependence, valid slices, or domain-defined scenarios.

Symptom: a local explanation changes dramatically between runs. Cause: random perturbations, an unstable background sample, or a model served behind an API that changed. Diagnose by pinning random seeds, logging model and data versions, and repeating the explanation on the same row. Correct it by fixing the reference sample and versioning explainer inputs.

Security, Performance, and Reliability

Interpretation artifacts can expose sensitive data. Local explanations often include feature values from a person or account, and background samples can reveal rare combinations. Store them with the same access controls as predictions. When sharing externally, prefer aggregated global explanations and redact direct identifiers. In adversarial settings, revealing exact feature sensitivities may help users manipulate inputs.

Performance matters because model-agnostic methods call prediction many times. Permutation importance requires roughly one extra scoring pass per feature per repeat. Partial dependence requires one pass per grid value per feature. Local perturbation methods can require thousands of calls for one row. Use sampled reference data, grouped features, caching, and asynchronous report generation for large models. Keep the model version, preprocessing version, reference data hash, random seed, output scale, and metric with each report.

Hands-On Lab: Build an Interpretation Report

Prerequisites: Python, a small labeled tabular dataset, and a trained estimator or deterministic prediction function. If you use a library model, keep the preprocessing pipeline object available. The lab can be done with the examples above or with your own validation rows.

1. Select 20 to 200 validation rows that represent normal use. Keep the target column separate from feature columns.
2. Compute the baseline metric on those rows. For regression, mean absolute error is easy to inspect. For classification, choose a metric aligned to the decision, such as log loss, recall at a fixed threshold, or false positive rate.
3. For each candidate feature, copy the validation rows, shuffle only that feature, predict again, and record the metric change. Repeat several times if the sample is small.
4. Choose one important numeric or ordinal feature. Build a grid of plausible values, replace that feature across the reference rows, predict for each grid value, and average the predictions.
5. Pick one row for local review. Compute its prediction, then compute feature contributions if the model is additive, or compare predictions after controlled feature changes if it is not.
6. Write a report stating the model version, data sample, metric, output scale, top global findings, one local example, and known limitations.

Verification: the baseline metric should match your normal evaluation code on the same rows. Re-running with the same seed should reproduce the same rankings or stay within a documented tolerance. Top features should have plausible data lineage and be available at prediction time. Cleanup or rollback: remove exported row-level explanations that contain personal data, delete temporary shuffled datasets, and do not promote a report generated from a different preprocessing pipeline than the deployed model.

This diagnostic check catches a common reliability problem before you trust a local explanation: the row being explained may be outside the range of the reference data.

def predict_score(row):
    return 0.5 * row["income"] - 2.0 * row["debt"]

train_rows = [{"income": 40, "debt": 5}, {"income": 80, "debt": 20}]
explain_row = {"income": 120, "debt": 80}
train_max = {name: max(row[name] for row in train_rows) for name in train_rows[0]}
flags = [name for name, limit in train_max.items() if explain_row[name] > limit]
print(flags)
print(predict_score(explain_row))

The output flags both income and debt, so the local explanation is extrapolating beyond the small training reference. The prediction can still be computed, but the explanation should carry a warning or use a better reference sample.

Assessment Exercises

• A random forest ranks a duplicated feature and its original feature as moderately important, while dropping either one barely changes validation accuracy. Explain why grouped permutation may be more informative than single-feature permutation.
• You are asked to explain a logistic regression to nontechnical reviewers. Which output scale would you show, and what would you keep internally for auditability?
• A partial dependence plot says risk decreases as account age increases, but the youngest accounts all belong to a newer product line. What diagnostic would you run before presenting the curve?
• A local explanation says a customer’s region drove a denial. List two checks that distinguish legitimate dependency from proxy or leakage behavior.
• Design a minimal interpretation report for a release. Specify the reference data, metric, global method, local method, and one reproducibility field.

Summary

Model interpretation is a set of measurement tools for a fitted prediction function. Coefficients expose readable internals for additive models. Permutation importance measures metric damage when a feature is disrupted. Partial dependence averages predictions over controlled feature values. Local explanations describe one prediction relative to a baseline or neighborhood. Responsible interpretation states the question, data sample, output scale, and limitations so explanations improve decisions without pretending that model association is causal truth.