Error Analysis and Dataset Slices
Error analysis turns a single validation score into a diagnosis. Dataset slicing is the companion technique: instead of asking whether a PyTorch model is good on average, you ask where it is wrong, which examples share a condition, and whether a proposed change improves the weak region without quietly damaging the rest of the distribution.
In this lesson, the outcome is practical: given model outputs, labels, and metadata, you will compute per-example errors, group them into meaningful slices, compare slice metrics, and decide what to do next. This belongs late in a deep learning workflow because it connects training, evaluation, data loading, loss functions, and model selection. A better optimizer or deeper network is not automatically useful if the main failure is a missing class, a mislabeled subset, or a threshold that fits one subgroup and harms another.
How Slice-Based Error Analysis Works
A classification model usually produces logits with shape [batch_size, num_classes]. The predicted class is often argmax over the class dimension, while the loss compares logits with integer class labels. Error analysis keeps the per-example view instead of reducing everything immediately to one scalar. You compute a vector such as wrong = prediction != label, a vector of per-example losses, or a vector of confidence values. Then you align those vectors with metadata: source, sequence length bucket, image brightness, language, device type, labeler, time period, or any feature that might explain model behavior.
A slice is a boolean mask over the evaluation set. In PyTorch that mask might be a torch.bool tensor, a list converted into a tensor, or indices returned by nonzero. The important invariant is alignment: the first prediction, first label, first loss, and first metadata row must describe the same example. Once aligned, a slice metric is just a reduction over a mask: accuracy for selected examples, mean loss for selected examples, false positive rate, recall, calibration error, or a task-specific cost.
Good slices are hypotheses, not decorations. A slice named long should represent a concrete condition, such as tokenized length above 256. A slice named low_light should come from a measurable image statistic or trusted annotation. Slices can overlap: an example may be both long and rare_class. That is useful, but it means slice metrics do not necessarily add up to the whole dataset.
API Anatomy in PyTorch
The mechanics use ordinary tensor operations. logits.argmax(dim=1) creates predictions for multiclass classification. preds.eq(labels) and preds.ne(labels) create correctness masks. CrossEntropyLoss(reduction="none") returns one loss per example instead of averaging the batch. Boolean indexing, such as losses[mask], selects the examples inside a slice. For larger evaluation jobs, you usually accumulate CPU tensors or Python records from each batch and concatenate them after the loop.
Metadata often starts outside PyTorch, for example in a CSV file, a dataset object, or filenames. A custom Dataset.__getitem__ can return (features, label, metadata), but the default collation rules must be respected. Numeric metadata can become tensors; strings are commonly collated into lists or tuples. For analysis code, clarity usually matters more than GPU speed, so it is acceptable to move predictions and losses back to CPU before grouping.
Example 1: Find Misclassified Examples
The first step is to preserve example-level predictions. The code below uses fixed logits so the output is deterministic. Class 1 is predicted when the second logit is larger than the first.
import torch
true_labels = torch.tensor([0, 1, 1, 0, 1, 0, 0, 1])
logits = torch.tensor([
[3.0, 0.2],
[0.4, 2.1],
[1.7, 0.6],
[2.4, 0.1],
[0.8, 1.9],
[0.2, 2.0],
[1.8, 0.5],
[0.3, 2.2],
])
predicted = logits.argmax(dim=1)
wrong = predicted.ne(true_labels)
print(predicted.tolist())
print(wrong.nonzero(as_tuple=True)[0].tolist())The expected output is [0, 1, 0, 0, 1, 1, 0, 1] followed by [2, 5]. Examples at positions 2 and 5 are wrong. This tiny report already tells you more than accuracy alone: two examples out of eight failed, and you know exactly which rows to inspect in the original dataset. In a real project, you would display those images, texts, waveforms, or tabular rows and look for shared causes.
Example 2: Compare Two Dataset Slices
Next, attach metadata. Here the metadata is a length bucket, but the same pattern works for camera model, speaker accent, document source, or clinical site when those fields are ethically and legally appropriate to use.
import torch
labels = torch.tensor([0, 1, 1, 0, 1, 0, 0, 1])
predictions = torch.tensor([0, 1, 0, 0, 1, 1, 0, 1])
slice_names = ["short", "short", "long", "short", "long", "long", "short", "long"]
for name in sorted(set(slice_names)):
mask = torch.tensor([item == name for item in slice_names])
accuracy = predictions[mask].eq(labels[mask]).float().mean().item()
errors = predictions[mask].ne(labels[mask]).sum().item()
print(f"{name}: n={int(mask.sum())}, accuracy={accuracy:.2f}, errors={errors}")The expected output is long: n=4, accuracy=0.50, errors=2 and short: n=4, accuracy=1.00, errors=0. Overall accuracy is 0.75, which sounds acceptable for a toy example, but the long slice is much weaker. The next action should be driven by inspection: maybe long examples are truncated, maybe the labels are noisier, or maybe the model architecture has too little context.
Example 3: Test a Slice-Specific Threshold
Some models produce a probability or score that is later thresholded. A global threshold can hide asymmetric costs across slices. The following example evaluates one threshold for short examples and another for long examples.
import torch
labels = torch.tensor([0, 1, 1, 0, 1, 0, 0, 1])
prob_positive = torch.tensor([0.12, 0.89, 0.35, 0.18, 0.74, 0.81, 0.28, 0.91])
slice_names = ["short", "short", "long", "short", "long", "long", "short", "long"]
thresholds = {"short": 0.50, "long": 0.70}
for name, threshold in thresholds.items():
mask = torch.tensor([item == name for item in slice_names])
preds = (prob_positive[mask] >= threshold).long()
accuracy = preds.eq(labels[mask]).float().mean().item()
false_positives = preds.eq(1).logical_and(labels[mask].eq(0)).sum().item()
false_negatives = preds.eq(0).logical_and(labels[mask].eq(1)).sum().item()
print(f"{name}: threshold={threshold:.2f}, accuracy={accuracy:.2f}, fp={false_positives}, fn={false_negatives}")The deterministic output is long: threshold=0.70, accuracy=1.00, fp=0, fn=0 and short: threshold=0.50, accuracy=1.00, fp=0, fn=0. This does not prove that slice-specific thresholds are always correct. It only proves that, on this validation set, the proposed thresholds fit the observed scores. In practice, you would choose thresholds on a validation split, then report final metrics once on a held-out test split to avoid tuning directly to test data.
Example 4: Per-Example Loss in an Evaluation Loop
When evaluating a real PyTorch model, use model.eval() and torch.no_grad(). The example below keeps the model untrained on purpose; the point is the evaluation pattern, not the quality of the predictions.
import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
features = torch.tensor([
[0.1, 0.0, 0.0],
[0.3, 0.1, 0.0],
[0.4, 0.0, 1.0],
[0.9, 0.2, 1.0],
[1.0, 0.4, 1.0],
[1.2, 0.3, 1.0],
], dtype=torch.float32)
labels = torch.tensor([0, 0, 0, 1, 1, 1], dtype=torch.long)
slices = ["low_signal", "low_signal", "flagged", "flagged", "flagged", "flagged"]
model = nn.Linear(3, 2)
loss_fn = nn.CrossEntropyLoss(reduction="none")
loader = DataLoader(TensorDataset(features, labels), batch_size=3, shuffle=False)
all_losses = []
all_predictions = []
with torch.no_grad():
for batch_features, batch_labels in loader:
batch_logits = model(batch_features)
all_losses.append(loss_fn(batch_logits, batch_labels))
all_predictions.append(batch_logits.argmax(dim=1))
losses = torch.cat(all_losses)
predictions = torch.cat(all_predictions)
for name in sorted(set(slices)):
mask = torch.tensor([item == name for item in slices])
print(name, round(losses[mask].mean().item(), 4), predictions[mask].tolist())The exact numeric losses and predictions depend on the randomly initialized linear layer, so there is no fixed expected output. The behavior is deterministic in shape: one line per slice, each with the slice name, mean loss rounded to four decimals, and the selected predictions. To make the numbers reproducible, call torch.manual_seed before constructing the model.
Design Choices and Trade-Offs
The first design choice is slice definition. Human-readable slices are easier to review, but hand-written labels can reflect bias or inconsistent judgment. Programmatic slices are reproducible, but they may encode a poor proxy. For example, short text length may correlate with a template, a language, or a user group; improving the slice metric without understanding that relationship can produce misleading conclusions.
The second choice is metric. Accuracy is simple, but it ignores confidence and class imbalance. Mean loss reveals overconfident wrong predictions. Precision and recall are better when false positives and false negatives have different costs. Calibration matters when downstream systems rely on probabilities. For rare slices, confidence intervals or bootstrap estimates are often more honest than a single decimal value.
The third choice is where to act. A weak slice can be addressed by collecting more data, fixing labels, changing preprocessing, altering augmentation, reweighting the loss, adjusting thresholds, or changing the architecture. Reweighting may improve a slice while lowering global accuracy. Extra augmentation may help images but damage natural texture statistics. Architecture changes may increase latency and memory. Error analysis should make these trade-offs visible before retraining becomes expensive.
Failure Modes and Troubleshooting
Symptom: every slice has the same metric. The likely cause is a mask bug, such as comparing every metadata value to the wrong constant or accidentally using the full dataset for each slice. Diagnose by printing mask.sum() for every slice and checking a few selected indices. Correct it by constructing masks from the same ordered metadata sequence used during evaluation.
Symptom: slice counts are correct, but examples shown for inspection do not match the reported errors. The common cause is shuffled evaluation without carrying stable example identifiers. The model outputs are in loader order, while the analyst looks up rows in original dataset order. Diagnose by returning an example_id from the dataset and storing it beside each prediction. Correct it by joining reports by identifier, not by implicit row number.
Symptom: validation slice improves, but test slice regresses. This often means you tuned repeatedly to the validation slice, the slice is too small, or the test distribution differs. Diagnose by reviewing experiment history and slice sample sizes. Correct it by freezing the test set, using cross-validation or bootstrap intervals for small slices, and requiring improvement on both the target slice and guardrail slices before accepting a change.
Symptom: per-example losses cannot be concatenated. The cause is usually mixed tensor shapes, forgotten device moves, or loss reduction set to the default mean. Diagnose by printing each batch loss shape and device. Correct it with CrossEntropyLoss(reduction="none"), ensure one scalar per example, and move accumulated tensors to CPU before concatenation if memory allows.
Reliability and Performance Implications
Slice analysis protects reliability by exposing concentrated failure. A model with 97 percent global accuracy can still be unusable for a minority class, a rare device, or long-tail input format. It also creates risk: if analysts mine hundreds of slices and report only the worst or best, the findings may be statistical noise. Keep a registry of planned slices, label exploratory slices as exploratory, and confirm major decisions on data that was not used to discover the pattern.
Performance is usually dominated by evaluation, not grouping. Run inference in batches, avoid storing large raw inputs in memory when identifiers are enough, and compute heavy visualizations after the tensor metrics are saved. Do not log sensitive text, images, or protected attributes casually. Store the minimum fields needed to reproduce the analysis and apply access controls appropriate for the data.
Hands-On Lab: Build a Slice Report
Prerequisites: Python with PyTorch installed, a terminal, and any small classification dataset or the toy tensors from this lesson. You do not need a GPU. Start by creating or loading logits, labels, and one metadata field with the same length. Step 1: compute preds = logits.argmax(dim=1). Step 2: compute wrong = preds.ne(labels). Step 3: define at least three slice masks, including one class-based slice such as labels == 1 and one metadata-based slice. Step 4: for each slice, print count, accuracy, mean loss if available, and wrong example identifiers. Step 5: inspect at least five wrong examples from the weakest slice and write a one-sentence hypothesis for each.
Verification: the sum of examples in non-overlapping slices should match the expected dataset size, each mask should have dtype torch.bool, and every listed wrong identifier should satisfy preds[id] != labels[id] or map to an external stable identifier with that same condition. Cleanup is simple for a local lab: delete temporary reports containing sensitive data, reset any notebook state before rerunning, and keep only the aggregate metrics or sanitized identifiers needed for comparison.
Assessment Exercises
- A model has 94 percent overall accuracy, 99 percent accuracy on short inputs, and 62 percent accuracy on long inputs. Propose two data-side fixes and two model-side fixes, and explain what guardrail metric would prevent harming short inputs.
- You discover a weak slice after checking 80 possible metadata fields. How would you distinguish a real pattern from a chance finding?
- Why is
CrossEntropyLoss(reduction="none")useful for error analysis, and what bug appears if you keep the default reduction? - Design a slice report for a medical image classifier without exposing raw patient images in logs. What identifiers and aggregate metrics are enough for debugging?
- Given a slice with 12 examples and 50 percent accuracy, what additional evidence would you require before changing training weights?
Summary
Error analysis and dataset slices make evaluation actionable. In PyTorch, the core mechanism is simple: keep per-example predictions, losses, labels, and metadata aligned, then reduce metrics over boolean masks. The difficult work is choosing meaningful slices, avoiding leakage from repeated tuning, interpreting small sample sizes, and selecting fixes that improve the weak region without damaging the rest of the task.
