Imbalanced Data, Sampling, and Label Quality

Imbalanced data is a data-pipeline problem with direct effects on optimization. If 98% of training examples are negative and 2% are positive, a model can minimize ordinary average loss while learning little about the rare class. In this lesson, the outcome is practical: inspect label distribution, choose between sampling and loss weighting, measure performance with class-aware metrics, and catch label-quality issues before they become model behavior.

In the Deep Learning with PyTorch course, this topic sits after dataset and dataloader construction because imbalance handling is usually implemented at the boundary between Dataset, Sampler, DataLoader, and the loss function. The model architecture may stay unchanged, but the stream of mini-batches and the gradient scale seen by the optimizer can change substantially.

What Imbalance Changes

Training minimizes an empirical objective. With ordinary mini-batch sampling, each example contributes roughly in proportion to how often it appears. Majority-class examples therefore dominate the average gradient. In binary classification, a highly imbalanced dataset can produce a low loss and high accuracy by predicting the majority class almost everywhere. That is not a PyTorch bug; it is the objective doing exactly what the pipeline asked it to do.

There are three common levers. First, change the examples drawn for each mini-batch with a sampler such as WeightedRandomSampler. Second, keep the data stream unchanged but change the loss contribution with class weights, pos_weight, or per-example weights. Third, improve label quality so the rare class is not polluted by duplicates, ambiguous labels, stale annotations, or inconsistent class definitions. These levers are related, but they are not interchangeable.

Mechanism Inside PyTorch

A PyTorch Dataset maps an integer index to one training item. A Sampler decides which indices the DataLoader asks for. A BatchSampler groups indices into batches. shuffle=True is a shorthand for random index order, while passing sampler=... gives direct control of index selection. For imbalanced data, the important point is that sampling weights are usually per-example weights, not per-class weights. You compute a weight for every row, often by looking up the inverse frequency of that row’s class.

WeightedRandomSampler(weights, num_samples, replacement=True) draws indices with probability proportional to their weights. If replacement=True, the same minority example can appear multiple times in one epoch. If replacement=False, the sampler can reorder examples but cannot create more minority observations than exist. That distinction matters: oversampling with replacement changes exposure frequency, while non-replacement sampling mainly changes ordering.

Loss weighting acts later. For single-label multi-class classification, torch.nn.CrossEntropyLoss(weight=class_weights) scales the loss for each target class. For binary or multi-label classification, torch.nn.BCEWithLogitsLoss(pos_weight=...) scales positive terms for each label. Loss weighting changes gradient magnitude without changing which examples are loaded. Sampling changes which examples are seen and can change batch normalization statistics, augmentation diversity, and epoch semantics.

API Anatomy

The usual pipeline is: compute labels from the training split only, count classes, convert class counts to class weights, map those class weights back to example weights, create a sampler, then pass it to DataLoader. Do not compute weights from validation or test labels when that changes a training decision. Validation and test sets should remain representative of the population you want to measure, unless you are deliberately building separate diagnostic slices.

import torch

labels = torch.tensor([0, 0, 0, 0, 1, 1])
class_counts = torch.bincount(labels)
class_weights = 1.0 / class_counts.float()
example_weights = class_weights[labels]

print(class_counts.tolist())
print([round(x, 3) for x in class_weights.tolist()])
print([round(x, 3) for x in example_weights.tolist()])

This first example separates class weights from example weights. The majority class 0 appears four times, so each class-0 row receives weight 0.25. The minority class 1 appears twice, so each class-1 row receives weight 0.5. The expected output is [4, 2], then [0.25, 0.5], then [0.25, 0.25, 0.25, 0.25, 0.5, 0.5].

Worked Example: Weighted Sampling

Now use those per-example weights to construct a mini epoch with a more balanced draw. The code uses a fixed generator seed so the demonstration is reproducible. In real training, you may still seed for experiment tracking, but you should not confuse one sampled epoch with the true dataset distribution.

import torch
from torch.utils.data import TensorDataset, DataLoader, WeightedRandomSampler

features = torch.arange(6).float().unsqueeze(1)
labels = torch.tensor([0, 0, 0, 0, 1, 1])
class_counts = torch.bincount(labels)
class_weights = 1.0 / class_counts.float()
example_weights = class_weights[labels]

generator = torch.Generator().manual_seed(7)
sampler = WeightedRandomSampler(
    weights=example_weights,
    num_samples=6,
    replacement=True,
    generator=generator,
)
loader = DataLoader(TensorDataset(features, labels), batch_size=3, sampler=sampler)

sampled_labels = []
for _, batch_labels in loader:
    sampled_labels.extend(batch_labels.tolist())

print(sampled_labels)
print(torch.bincount(torch.tensor(sampled_labels), minlength=2).tolist())

The exact sampled sequence is deterministic for the given seed and PyTorch generator behavior, but the important invariant is probabilistic: minority examples are drawn more often than ordinary uniform sampling would draw them. With replacement, one rare example may repeat and another may be skipped. That can help the optimizer receive more minority-class gradient signal, but it does not create new information.

Worked Example: Loss Weighting

Loss weighting is often simpler when the dataset is large enough and you do not want duplicated rows in batches. The next example compares unweighted and weighted cross-entropy for three examples. Two belong to class 0, one belongs to class 1. Class 1 receives a larger loss weight, so a mistake on class 1 matters more to the optimizer.

import torch
import torch.nn.functional as F

logits = torch.tensor([[2.0, -1.0], [1.5, -0.5], [0.3, 0.1]])
targets = torch.tensor([0, 0, 1])
class_weights = torch.tensor([0.5, 2.0])

plain_loss = F.cross_entropy(logits, targets)
weighted_loss = F.cross_entropy(logits, targets, weight=class_weights)

print(round(float(plain_loss), 4))
print(round(float(weighted_loss), 4))

The weighted loss is not just the plain loss multiplied by a constant. PyTorch applies the target class weight to each example and normalizes according to the loss reduction. The expected behavior is that the third example, the class-1 item, has more influence. Use this when you want the full dataset traversal to remain intact while changing the objective’s class priorities.

Worked Example: Label-Quality Audit

Imbalance handling can hide bad labels. If rare positives are mislabeled, oversampling repeats the mistakes more often. A basic audit should check impossible labels, duplicate identifiers with conflicting labels, and class counts per split.

from collections import Counter, defaultdict

rows = [
    {"id": "a1", "split": "train", "label": 0},
    {"id": "a2", "split": "train", "label": 0},
    {"id": "a3", "split": "train", "label": 1},
    {"id": "a3", "split": "train", "label": 0},
    {"id": "b1", "split": "val", "label": 1},
]
allowed_labels = {0, 1}
labels_by_id = defaultdict(set)
counts_by_split = defaultdict(Counter)

for row in rows:
    if row["label"] not in allowed_labels:
        raise ValueError(f"bad label: {row}")
    labels_by_id[row["id"]].add(row["label"])
    counts_by_split[row["split"]][row["label"]] += 1

conflicts = {item_id: labels for item_id, labels in labels_by_id.items() if len(labels) > 1}
print({split: dict(counts) for split, counts in counts_by_split.items()})
print(conflicts)

The expected output shows train counts for both classes and a conflict for a3. Correct that record before tuning sampler weights. If the duplicate is a true duplicate, keep one reviewed label. If it represents two different observations that share an identifier, fix the identifier before splitting, or leakage between train and validation can make metrics look better than they are.

Design Choices and Trade-Offs

Use weighted sampling when rare examples are so scarce that ordinary batches often contain none. It gives the optimizer frequent minority feedback early in training. The trade-off is higher overfitting risk because the same rare items may repeat. Strong augmentations can help for images, audio, and text, but augmentation must preserve the label.

Use loss weighting when you want every epoch to remain a pass over the original data distribution. It is usually easier to reason about with distributed training and large datasets. The trade-off is gradient scale: very large weights can make optimization noisy, especially with mislabeled minority examples.

Use threshold tuning and class-aware metrics at evaluation time. Accuracy is usually a weak metric for imbalance. Prefer precision, recall, F1, ROC-AUC, precision-recall AUC, confusion matrices, and per-class loss. Choose the metric that matches the cost of mistakes. For a medical triage screen, false negatives and false positives do not have symmetric cost. For moderation or fraud review, downstream human capacity may matter as much as raw recall.

Failure Modes and Troubleshooting

Symptom: validation accuracy is high but minority recall is near zero. Cause: the model predicts the majority class for most inputs. Diagnose: print a confusion matrix and per-class recall on the validation set. Correct: try loss weighting or weighted sampling, then choose a decision threshold on validation data rather than using a default threshold blindly.

Symptom: training loss drops quickly but validation minority performance gets worse. Cause: oversampling repeats rare examples until the model memorizes them. Diagnose: inspect duplicate sample frequency per epoch and compare train-versus-validation minority loss. Correct: reduce oversampling strength, add label-preserving augmentation, collect more minority data, or switch to moderate loss weighting.

Symptom: sampler construction fails or silently samples the wrong pattern. Cause: class weights were passed directly to WeightedRandomSampler instead of per-example weights. Diagnose: check that len(weights) == len(train_dataset). Correct: compute example_weights = class_weights[train_labels].

Symptom: validation metrics swing heavily between runs. Cause: the validation split contains too few minority examples, or labels are noisy. Diagnose: count labels per split and inspect a sample of high-loss minority examples. Correct: use stratified splitting before training, freeze the validation set, clean labels, and report confidence intervals or repeated-run summaries where appropriate.

Security, Performance, and Reliability

Imbalance handling is not usually a security feature, but it can affect fairness and reliability. If minority classes represent safety-critical events or protected groups, document how classes were defined, who labeled them, and which slices were evaluated. Do not log raw sensitive examples during audits; log bounded identifiers, aggregate counts, and review outcomes.

Performance costs come from duplicated loading, larger effective epochs, and expensive audits over metadata. Cache label arrays separately from heavy tensors so class counts can be computed without decoding every image or waveform. In multi-worker loading, keep sampling deterministic enough for reproducibility by recording seeds and sampler configuration. In distributed training, verify that each process sees the intended sampling scheme; naive weighted sampling per process can change effective class exposure.

Hands-On Lab

Prerequisites: Python with PyTorch installed, a small classification dataset or the toy tensors above, and permission to create a scratch script or notebook. Use only the training split to compute sampler or loss weights.

1. Create a label tensor for the training split and print torch.bincount(labels).
2. Compute inverse-frequency class weights and map them to per-example weights.
3. Build a WeightedRandomSampler with num_samples=len(train_dataset) and replacement=True.
4. Create a DataLoader using sampler=sampler. Do not also pass shuffle=True.
5. Run one epoch without updating a model and count the sampled labels that appear in batches.
6. Train a baseline model with ordinary sampling, then train the same model with weighted sampling or weighted loss.
7. Evaluate both runs on the unchanged validation set with a confusion matrix and minority recall.

Verification: the sampled training batches should contain more minority examples than uniform sampling, while the validation label distribution should remain unchanged. The revised model should improve the target class-aware metric without an unacceptable drop in the metric that represents majority-class cost.

Cleanup: remove scratch artifacts, reset temporary seeds if they were set globally in a notebook, and keep only the experiment record: split identifiers, label counts, sampler or loss configuration, metrics, and notes from label review.

Assessment Exercises

1. A dataset has 9,900 negative examples and 100 positive examples. Explain why 99% accuracy can be a weak result, and name two metrics that would reveal the issue.
2. You pass a two-element class-weight tensor to WeightedRandomSampler for a dataset with 10,000 rows. What is wrong, and what length should the sampler weight tensor have?
3. Compare weighted sampling and weighted cross-entropy for a dataset with noisy minority labels. Which one is more likely to repeat bad labels in batches, and how would you diagnose that?
4. Why should validation and test sets usually keep their natural distribution even when the training loader is balanced?
5. Design a label audit for a rare class in your own project. Include at least one duplicate check, one split check, and one manual review trigger.

Summary

Imbalanced learning in PyTorch is controlled through index selection, loss scaling, metrics, and label review. WeightedRandomSampler changes which examples appear in mini-batches; class-weighted losses change the objective contribution of loaded examples. Both can help, and both can amplify poor labels if the dataset is not audited. Keep validation honest, measure per-class behavior, and choose the mechanism that matches the failure mode you actually observed.