Classification and regression metrics turn raw predictions into evidence about whether a model is useful. In this course section on evaluation and generalization, the purpose is not to collect impressive numbers. The purpose is to choose measurements that reflect the cost of mistakes on examples the model did not learn from.
A classifier predicts categories, such as fraud or not fraud. A regressor predicts numeric values, such as delivery time or house price. Both can be wrong in different ways. A medical screening classifier that misses disease has a different failure cost from one that sends too many healthy patients for follow-up. A price model that is wrong by ten dollars on every item behaves differently from one that is usually close but occasionally off by thousands. Metrics name these differences so you can compare models, tune thresholds, and explain decisions.
Purpose and Outcome
By the end of this lesson, you should be able to compute common classification and regression metrics, interpret what they reward, recognize when they are misleading, and design an evaluation report that matches the decision being automated. You will also practice diagnosing metric failures caused by class imbalance, threshold mistakes, data leakage, and outliers.
Classification Internals
Most classification metrics are built from a confusion matrix. For binary classification, each example belongs to a positive or negative class. The model predicts positive or negative. A true positive is a positive example predicted positive. A false positive is a negative example predicted positive. A true negative is a negative example predicted negative. A false negative is a positive example predicted negative.
Accuracy is the fraction of all predictions that are correct: true positives plus true negatives divided by all examples. Accuracy is simple, but it can hide failure when one class is rare. If only one percent of transactions are fraud, a model that always predicts not fraud reaches ninety-nine percent accuracy while detecting no fraud.
Precision asks, among predicted positives, how many were truly positive. Recall asks, among actual positives, how many were found. F1 combines precision and recall using their harmonic mean, which punishes a model that is very low on either side. Specificity measures how many actual negatives were correctly rejected. Balanced accuracy averages recall for the positive class and recall for the negative class, making it more useful when classes are imbalanced.
Many classifiers produce a score or probability before producing a class label. A threshold converts that score into a label. Lowering the threshold usually increases recall because more examples become positive, but it often reduces precision because more negatives are swept in. Raising the threshold usually does the reverse. ROC curves plot true positive rate against false positive rate across thresholds. Precision-recall curves plot precision against recall. PR curves are often more revealing for rare positive classes because they focus on the quality and coverage of positive predictions.
Regression Internals
Regression metrics start with residuals. A residual is actual value minus predicted value. The mean error keeps the sign, so it detects bias: positive mean error means predictions are low on average, while negative mean error means predictions are high on average. Mean absolute error, or MAE, averages absolute residuals. It is easy to interpret in the target unit. Root mean squared error, or RMSE, squares residuals, averages them, and takes the square root. Because squaring makes large errors much larger, RMSE penalizes outliers more than MAE.
Mean absolute percentage error can be useful when relative error matters, but it breaks down when actual values are zero or near zero. R-squared compares the model to a baseline that predicts the mean target, but it can look acceptable while individual predictions are unusable. For most applied work, report at least one unit-based metric such as MAE or RMSE, a bias metric, and a domain-specific cost metric.
Metric API Anatomy
A reliable metric function has a small contract: aligned actual and predicted sequences, an explicit positive label or target unit, and clear behavior for zero denominators. The denominator matters. Precision divides by predicted positives. Recall divides by actual positives. If there are no predicted positives, precision is undefined; if there are no actual positives, recall is undefined. Many libraries let you choose whether an undefined metric becomes zero, one, or a warning. That choice should be documented because it affects dashboards and model comparisons.
Example 1: Confusion Metrics
This example computes the confusion matrix and derived binary metrics for a small classifier. The positive class is 1. The expected output shows two true positives, two true negatives, one false positive, and one false negative.
def binary_classification_report(actual, predicted, positive=1):
if len(actual) != len(predicted) or not actual:
raise ValueError("actual and predicted must be aligned and non-empty")
tp = fp = tn = fn = 0
for observed, guess in zip(actual, predicted):
if observed == positive and guess == positive:
tp += 1
elif observed != positive and guess == positive:
fp += 1
elif observed != positive and guess != positive:
tn += 1
else:
fn += 1
precision = tp / (tp + fp) if tp + fp else 0.0
recall = tp / (tp + fn) if tp + fn else 0.0
accuracy = (tp + tn) / len(actual)
f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
return {"tp": tp, "fp": fp, "tn": tn, "fn": fn,
"accuracy": accuracy, "precision": precision, "recall": recall, "f1": f1}
report = binary_classification_report([1, 0, 1, 1, 0, 0], [1, 0, 0, 1, 1, 0])
for key, value in report.items():
print(f"{key}: {value:.3f}" if isinstance(value, float) else f"{key}: {value}")The report makes the trade-off visible. Accuracy is four out of six, or 0.667. Precision is two out of three predicted positives, also 0.667. Recall is two out of three actual positives, also 0.667. In a real project, those equal values would not automatically mean the model is good. You would compare them with the cost of false alarms and missed positives.
Example 2: Threshold Choice
The next example starts from scores instead of fixed labels. A threshold of 0.30 catches more positives but creates more false positives. A threshold of 0.70 is stricter. This is the mechanism behind threshold tuning, ROC curves, and precision-recall curves.
def labels_from_scores(scores, threshold):
return [1 if score >= threshold else 0 for score in scores]
actual = [1, 0, 1, 0, 1, 0, 0, 1]
scores = [0.95, 0.80, 0.62, 0.55, 0.51, 0.40, 0.20, 0.10]
for threshold in [0.30, 0.50, 0.70]:
predicted = labels_from_scores(scores, threshold)
report = binary_classification_report(actual, predicted)
print(f"threshold={threshold:.2f} precision={report['precision']:.3f} recall={report['recall']:.3f} f1={report['f1']:.3f}")The deterministic output is threshold=0.30 precision=0.600 recall=0.750 f1=0.667, threshold=0.50 precision=0.600 recall=0.750 f1=0.667, and threshold=0.70 precision=0.500 recall=0.250 f1=0.333. The high threshold misses three of four positives. If missing positives is expensive, the high threshold is worse even though it may feel more cautious.
Example 3: Regression Error Shape
This regression example compares MAE, RMSE, bias, and a simple weighted business cost. The last prediction has a large error, so RMSE rises more sharply than MAE.
from math import sqrt
def regression_report(actual, predicted, under_cost=1.0, over_cost=1.0):
if len(actual) != len(predicted) or not actual:
raise ValueError("actual and predicted must be aligned and non-empty")
residuals = [a - p for a, p in zip(actual, predicted)]
absolute_errors = [abs(error) for error in residuals]
squared_errors = [error * error for error in residuals]
business_cost = []
for error in residuals:
business_cost.append(error * under_cost if error > 0 else abs(error) * over_cost)
return {
"mae": sum(absolute_errors) / len(actual),
"rmse": sqrt(sum(squared_errors) / len(actual)),
"bias": sum(residuals) / len(actual),
"cost": sum(business_cost) / len(actual),
}
report = regression_report([100, 120, 90, 500], [110, 115, 100, 420], under_cost=3.0, over_cost=1.0)
for key, value in report.items():
print(f"{key}: {value:.2f}")The expected output is mae: 26.25, rmse: 40.54, bias: 16.25, and cost: 63.75. The positive bias means actual values are higher than predictions on average, so the model underpredicts. The weighted cost triples underprediction errors, which might match a stock-planning scenario where underestimating demand loses more money than overestimating it.
Design Choices and Trade-offs
Choose classification metrics by the action attached to the prediction. Use recall when missing positives is the main danger. Use precision when acting on false positives is expensive. Use F1 when precision and recall both matter and you need one summary, but remember it hides the actual balance. Use ROC AUC to evaluate ranking quality across thresholds when both classes are sufficiently represented. Use PR AUC when positives are rare and positive prediction quality is the central concern.
Choose regression metrics by the error shape the business can tolerate. MAE is robust and interpretable. RMSE is better when large misses are disproportionately bad. Bias is essential when systematic overprediction or underprediction matters. Quantile metrics are useful when you need prediction intervals or asymmetric service guarantees, such as estimating the time by which ninety percent of deliveries will arrive.
A single leaderboard metric is rarely enough. During model selection, use a primary metric tied to the decision and secondary guardrail metrics that prevent unacceptable behavior. For example, a fraud model might optimize recall at a minimum precision, while also reporting review volume so the operations team is not overwhelmed.
Failure Modes and Troubleshooting
Symptom: accuracy is very high, but users report missed rare events. Cause: class imbalance makes the majority class dominate accuracy. Diagnostic steps: inspect class prevalence, print a confusion matrix, and compare against an always-negative baseline. Correction: use recall, precision, PR AUC, balanced accuracy, and threshold tuning.
Symptom: offline metrics are excellent, but live performance collapses. Cause: data leakage or a train-test split that allowed future information into training. Diagnostic steps: list every feature and confirm it exists at prediction time, split data by time or entity where appropriate, and rerun evaluation with leakage-prone columns removed. Correction: rebuild the pipeline with prediction-time features only and keep preprocessing fit only on training data.
Symptom: regression RMSE is much worse than MAE. Cause: a small number of large residuals dominates squared error. Diagnostic steps: sort residuals by absolute value, inspect outlier rows, and plot residuals against key features or time. Correction: fix data errors, cap impossible target values when justified, use robust models, or report both MAE and RMSE so the tail risk is visible.
Symptom: precision changes sharply after deployment even though code did not change. Cause: class prevalence or score distribution changed. Diagnostic steps: compare current score histograms and class rates with the evaluation set, then compute metrics by segment. Correction: recalibrate scores, adjust thresholds using recent labeled data, or retrain with representative samples.
Reliability and Performance Implications
Metrics are part of reliability because they decide which model ships. If the metric calculation silently drops rows, aligns predictions with the wrong labels, or averages over the wrong segment, the system can approve a harmful model. Keep evaluation data immutable, store the model version and dataset version with every report, and fail fast when counts do not match.
Metric computation is usually cheaper than training, but large evaluations still need care. Streaming confusion counts avoids storing every prediction. Regression sums can be accumulated in one pass. Curves require sorting scores, so they are more expensive than a single threshold report but still practical for many datasets. For privacy-sensitive targets, publish aggregated metrics by cohort only when each cohort has enough examples to avoid exposing individual outcomes.
Hands-on Lab
Prerequisites: Python available on your machine and a text editor. No external packages are required. Create a file named metrics_lab.py and paste the three example functions into it: binary_classification_report, labels_from_scores, and regression_report.
- Add a classification dataset with at least twenty examples and a separate list of probability-like scores.
- Compute reports for thresholds 0.20, 0.50, and 0.80.
- Choose the threshold you would deploy if false negatives cost five times more than false positives, and write a one-sentence justification.
- Add a regression dataset with one intentional outlier.
- Compute MAE, RMSE, bias, and weighted cost twice: once with equal costs and once with underprediction cost set to three.
Verification: run python metrics_lab.py. The script should print three classification reports and two regression reports. Check that lowering the threshold generally does not reduce recall, and check that RMSE reacts more strongly than MAE when the outlier is present.
Cleanup: delete metrics_lab.py if you do not need it. If you changed a shared notebook or repository, roll back exploratory threshold choices that were not selected for the final report.
Assessment Exercises
- A classifier for a rare disease has 98 percent accuracy and 20 percent recall. Explain why accuracy is misleading and name two metrics you would request next.
- You can review only 200 fraud alerts per day. Describe how precision, recall, and threshold selection interact with that capacity limit.
- A delivery-time regressor has MAE of 4 minutes and RMSE of 18 minutes. What does that suggest about the residual distribution, and what diagnostic would you run?
- Design a metric report for a loan default model where false positives deny good applicants and false negatives approve risky loans. Include at least one primary metric and one guardrail.
- Explain why fitting preprocessing on the full dataset before the train-test split can make metrics too optimistic.
Summary
Classification metrics are built from counts of true positives, false positives, true negatives, and false negatives. Regression metrics are built from residuals. The right metric depends on the decision, the class balance, the target distribution, and the cost of different errors. In machine learning evaluation, a metric is not just a score; it is a compact statement of what kind of generalization you are willing to reward.
