Metrics, Thresholds, and Calibration

Metrics, thresholds, and calibration turn model outputs into evidence for decisions. A PyTorch classifier usually emits logits: unconstrained scores before a sigmoid or softmax. Metrics summarize how the model behaves against labels, thresholds convert scores into actions, and calibration asks whether a reported probability means what it claims. Given validation outputs from a deep learning model, you should be able to compute task-relevant metrics, choose a cost-aware threshold, and detect overconfident or timid probability scores.

This belongs in the evaluation and improvement section because loss alone is not the same as usefulness. Cross-entropy can improve while the chosen operating point still misses rare positives, overwhelms reviewers with false alarms, or reports 0.9 confidence on examples that are correct only 70 percent of the time. In PyTorch work, evaluation is a separate phase from training: set the model to eval(), disable gradients with torch.no_grad(), collect logits and labels from validation or test data, and then make decisions outside the optimizer loop.

From Logits to Decisions

For binary classification, a model commonly returns one logit per example. Applying torch.sigmoid maps that logit to a number between 0 and 1. For multiclass classification, the model returns one logit per class and torch.softmax(logits, dim=1) creates a probability distribution across classes. These transformations are monotonic for the winning class: a larger binary logit produces a larger sigmoid probability, and a larger class logit relative to the others produces a larger softmax probability. They do not, by themselves, prove that the number is calibrated.

A threshold is the boundary between a score and an action. With the default binary threshold of 0.5, probabilities at least 0.5 are predicted positive. A fraud detector, tumor triage model, or content moderation queue may need a lower or higher threshold depending on the relative cost of false positives and false negatives. Multiclass models often use argmax, but many real systems still add thresholds: send to a human reviewer if the top probability is below 0.8, abstain when the margin between the top two classes is small, or trigger a high-severity workflow only above a stricter class-specific threshold.

Metric Anatomy

The confusion matrix is the base structure for many classification metrics. True positives are positive examples predicted positive. False positives are negative examples predicted positive. True negatives are negative examples predicted negative. False negatives are positive examples predicted negative. Accuracy is the fraction of all correct predictions, but it can hide poor behavior on imbalanced data. Precision answers: of the predicted positives, how many were truly positive? Recall answers: of the actual positives, how many did the model find? F1 is the harmonic mean of precision and recall, which punishes a model that is strong on one and weak on the other.

Ranking metrics evaluate score ordering before a single threshold is chosen. ROC AUC asks how often a random positive is scored above a random negative while sweeping false positive rate and true positive rate. Precision-recall AUC is usually more informative when positives are rare because it focuses on the positive class and the burden of false alarms. The final operating point should come from validation data and a stated objective, such as maximum expected utility, minimum recall at a bounded false positive rate, or highest F1 when false positives and false negatives are similarly costly.

Worked Example 1: Confusion Counts

The first example starts with probabilities already produced by a model. The threshold is 0.5. The deterministic output shows two true positives, one false positive, two true negatives, and one false negative. From these counts, precision is 2 / (2 + 1), recall is 2 / (2 + 1), and F1 is also 0.667 when rounded. These values describe one threshold, not every deployment setting.

import torch

def confusion_counts(probs: torch.Tensor, labels: torch.Tensor, threshold: float = 0.5) -> dict[str, int]:
    preds = probs >= threshold
    truth = labels.bool()
    return {
        "tp": int((preds & truth).sum()),
        "fp": int((preds & ~truth).sum()),
        "tn": int((~preds & ~truth).sum()),
        "fn": int((~preds & truth).sum()),
    }

probs = torch.tensor([0.90, 0.80, 0.55, 0.45, 0.40, 0.20])
labels = torch.tensor([1, 0, 1, 0, 1, 0])
counts = confusion_counts(probs, labels)
print(counts)

Expected output: {'tp': 2, 'fp': 1, 'tn': 2, 'fn': 1}

Worked Example 2: Choosing an Operating Threshold

The second example assigns utility to outcomes. A true positive is worth 5 points, a false positive costs 2 points, and a false negative costs 6 points. On this validation slice, threshold 0.5 wins. Threshold 0.2 catches more examples but creates too many false positives. Threshold 0.8 is stricter, but it misses too many positives.

import torch

def counts_at(probs: torch.Tensor, labels: torch.Tensor, threshold: float) -> tuple[int, int, int]:
    preds = probs >= threshold
    truth = labels.bool()
    tp = int((preds & truth).sum())
    fp = int((preds & ~truth).sum())
    fn = int((~preds & truth).sum())
    return tp, fp, fn

probs = torch.tensor([0.95, 0.81, 0.63, 0.51, 0.48, 0.35, 0.19, 0.08])
labels = torch.tensor([1, 1, 0, 1, 0, 0, 1, 0])
for threshold in [0.2, 0.5, 0.8]:
    tp, fp, fn = counts_at(probs, labels, threshold)
    utility = 5 * tp - 2 * fp - 6 * fn
    print(f"threshold={threshold:.1f} tp={tp} fp={fp} fn={fn} utility={utility}")

Expected output: threshold=0.2 tp=3 fp=3 fn=1 utility=3; threshold=0.5 tp=3 fp=1 fn=1 utility=7; threshold=0.8 tp=2 fp=0 fn=2 utility=-2

In a real project, sweep many thresholds on a validation set, not on the final test set. After choosing the threshold, evaluate once on the test set to estimate future behavior. Repeated test-set tuning makes the reported result optimistic.

Worked Example 3: Measuring Calibration

Calibration compares confidence with empirical correctness. If a model assigns 0.8 confidence to 100 predictions, roughly 80 should be correct for that bucket to be well calibrated. Expected calibration error, or ECE, bins predictions by confidence and computes a weighted average of the gap between average confidence and average accuracy in each bin. ECE depends on bin choices, so treat it as a diagnostic, not a universal truth.

import torch

def expected_calibration_error(confidence: torch.Tensor, correct: torch.Tensor, bin_edges: torch.Tensor) -> float:
    ece = torch.tensor(0.0)
    for left, right in zip(bin_edges[:-1], bin_edges[1:]):
        in_bin = (confidence > left) & (confidence <= right)
        if in_bin.any():
            accuracy = correct[in_bin].float().mean()
            avg_confidence = confidence[in_bin].mean()
            ece += in_bin.float().mean() * torch.abs(avg_confidence - accuracy)
    return float(ece)

confidence = torch.tensor([0.95, 0.85, 0.75, 0.65, 0.55, 0.45])
correct = torch.tensor([1, 1, 0, 1, 0, 0])
bins = torch.tensor([0.0, 0.5, 0.7, 0.9, 1.0])
print(round(expected_calibration_error(confidence, correct, bins), 4))

Expected output: 0.2167

This output means the binned confidence differs from observed accuracy by about 0.217 on average for this tiny example. On a validation set, use the same mechanism: compute confidence, correctness, confidence ranges, and the accuracy gap.

Worked Example 4: Temperature Scaling

Temperature scaling is a common post-training calibration method for classifiers. It learns one positive scalar on validation logits. Dividing logits by a temperature greater than 1 softens the probability distribution; dividing by a value below 1 sharpens it. The class ranking is preserved because every logit is scaled by the same positive value, so top-1 accuracy does not change, but confidence can become more honest. The example optimizes temperature against validation cross-entropy and verifies that the calibrated validation loss did not increase.

import torch
import torch.nn.functional as F

logits = torch.tensor([[3.0, 0.2], [2.2, 0.1], [1.8, 0.4], [0.6, 1.2]])
labels = torch.tensor([0, 0, 1, 1])
temperature = torch.nn.Parameter(torch.ones(()))
optimizer = torch.optim.LBFGS([temperature], lr=0.1, max_iter=25)

def closure() -> torch.Tensor:
    optimizer.zero_grad()
    scaled_logits = logits / temperature.clamp_min(0.05)
    loss = F.cross_entropy(scaled_logits, labels)
    loss.backward()
    return loss

optimizer.step(closure)
with torch.no_grad():
    before = F.cross_entropy(logits, labels).item()
    after = F.cross_entropy(logits / temperature.clamp_min(0.05), labels).item()
print(after <= before)
print(float(temperature.detach()) > 0.0)

Expected output: True; True

Design Choices and Trade-offs

Choose metrics from the failure cost, class distribution, and action that follows prediction. Accuracy is readable and useful for balanced tasks where all errors cost about the same. Precision matters when acting on a positive prediction is expensive or disruptive. Recall matters when missing a positive is dangerous. F1 is compact but assumes precision and recall should be balanced; it is a poor substitute for an explicit cost model when costs are known. ROC AUC is stable for broad ranking comparisons, while precision-recall curves usually expose behavior more clearly for rare positive classes.

Thresholds can be global, class-specific, or segment-specific. A global threshold is simple and easier to explain. Class-specific thresholds are often needed in multilabel systems where each label has a different prevalence and cost. Segment-specific thresholds can improve utility when populations differ, but they require careful fairness review and enough validation data per segment. Calibration methods also have trade-offs. Temperature scaling is simple and preserves accuracy, but it mainly corrects overconfidence shape. Platt scaling, isotonic regression, and histogram binning may fit more flexible mappings, but they need more held-out data and can overfit small validation sets.

Failure Modes and Troubleshooting

Symptom: validation accuracy is high, but users report many missed positives. Cause: the dataset is imbalanced and accuracy is dominated by true negatives. Diagnose by printing the confusion matrix, class prevalence, recall, and precision-recall curve. Correct by selecting a metric tied to the positive class, reviewing labels for rare positives, and choosing a threshold that satisfies a recall or utility target.

Symptom: metrics look excellent during development and much worse after launch. Cause: threshold tuning, calibration, or model selection leaked onto the test set, or the validation split does not represent deployed data. Diagnose by checking experiment history, split creation code, duplicate entities across splits, and time ordering. Correct by rebuilding leakage-safe train, validation, and test splits, then freezing the final test set until the end of model selection.

Symptom: predicted probabilities are consistently too high. Cause: modern neural networks trained with cross-entropy can be overconfident, especially after long training, heavy capacity, or distribution shift. Diagnose with reliability diagrams, ECE, negative log likelihood, and bucketed accuracy by confidence. Correct with temperature scaling on validation logits, stronger regularization, more representative data, or an abstention policy for low-margin predictions.

Symptom: threshold chosen offline produces too many alerts online. Cause: validation prevalence or score distribution differs from production traffic. Diagnose by comparing live score histograms, label delay effects, feature missingness, and segment mix against validation. Correct by recalibrating on recent labeled data, using prevalence-aware monitoring, and setting rollback criteria before changing thresholds.

Reliability and Performance Implications

Metric code must run under model.eval() so dropout and batch normalization use inference behavior. Evaluation should use torch.no_grad() or torch.inference_mode() to avoid unnecessary memory use. Accumulate logits and labels carefully: moving huge tensors to CPU can exhaust memory, while per-batch averages can hide class imbalance. For large datasets, accumulate sufficient statistics such as confusion counts or stream scores to disk for later curve computation.

Reliability also depends on label timing. Some outcomes arrive days later, so live calibration dashboards may lag. Monitor score distributions immediately and outcome-based metrics when labels mature. Store the model, threshold, calibration version, and data slice for each evaluation run. Without those identifiers, a regression cannot be traced to the network weights, threshold policy, calibration transform, or input data.

Hands-on Lab: Evaluate a Binary Classifier

Prerequisites: Python with PyTorch installed, a trained or toy binary classifier that emits logits, and a validation loader yielding input tensors and 0/1 labels. Stored validation logits and labels also work.

1. Put the model in evaluation mode with model.eval() and collect validation logits and labels inside torch.no_grad(). Store logits before sigmoid so calibration can reuse them.
2. Convert logits to probabilities with torch.sigmoid(logits). Confirm that probability tensor shape matches label shape after squeezing only the intended singleton dimension.
3. Compute confusion counts, precision, recall, and F1 for thresholds from 0.05 to 0.95. Record the best threshold according to your chosen utility or constraint.
4. Compute calibration diagnostics on the same validation split used for threshold selection, then fit temperature scaling on a separate calibration split when enough data is available. If data is limited, use cross-validation or keep calibration as a diagnostic rather than a fitted transform.
5. Run one final evaluation on the untouched test split using the selected threshold and calibration policy. Report confusion counts, the chosen metric, ECE or a reliability table, and class prevalence.

Verification: the selected threshold should be present in the saved evaluation report, the test metrics should be computed without changing the threshold, and the confusion counts should sum to the number of test examples. If using temperature scaling, verify that the learned temperature is positive and that class rankings are unchanged for binary probabilities crossing only through the monotonic sigmoid transform.

Cleanup or rollback: keep the previous threshold and calibration artifact available. If online score distributions or matured labels move outside the validation range, restore the previous threshold while collecting a fresh labeled calibration set. Delete temporary tensors or cached logits that contain sensitive examples when the evaluation report no longer needs them.

Assessment Exercises

1. A model has 98 percent accuracy on a dataset with 1 percent positives. What additional metrics would you request before approving it, and why?
2. Two thresholds produce the same F1. One has higher precision and lower recall. Describe a task where that threshold is preferable and a task where it is unsafe.
3. A classifier has unchanged ROC AUC after temperature scaling but lower ECE. Explain why both statements can be true.
4. You discover that the test set was used to pick the threshold. What result is no longer trustworthy, and how would you redesign the evaluation?
5. Design a reliability table for a three-class softmax model. What value is binned, and what observed quantity is compared against it?

Summary

Metrics describe behavior, thresholds define actions, and calibration checks whether confidence is meaningful. In PyTorch, collect logits and labels in evaluation mode, compute metrics on validation data, choose thresholds from explicit costs or constraints, and reserve the test set for the final estimate. Calibration is not a cosmetic layer: it changes how downstream systems interpret probability. A dependable evaluation report names the split, model version, threshold policy, calibration method, class prevalence, and failure slices.