Detection, Segmentation, and Vision Evaluation

Detection, segmentation, and vision evaluation turn a PyTorch image model from a single-label classifier into a spatial reasoning system. Instead of answering “what is in this image?”, the model must answer “where is each object?”, “which pixels belong to each region?”, and “how close are those predictions to the annotation?” You will know the tensors a model emits, how those tensors become boxes or masks, and how IoU, AP, Dice, and pixel accuracy expose different kinds of correctness.

This computer vision chapter reuses convolutional and transfer-learning ideas, but changes the target representation. Detection and segmentation models are still optimized by losses over tensors, yet their labels are structured objects: variable-length box lists, per-pixel class maps, or instance masks. Evaluation therefore cannot stop at top-1 accuracy; it must compare geometry, class identity, confidence ranking, and object count.

Purpose and Outcome

Object detection predicts rectangular regions, class labels, and confidence scores. Semantic segmentation predicts one class per pixel. Instance segmentation separates individual objects as masks, so two adjacent cars are two instances rather than one “car” region. Panoptic segmentation combines semantic regions and instance identities. The common theme is alignment: the model output must align with image coordinates and annotation conventions. A good detector is not just confident; it places boxes tightly around the right objects. A good segmenter is not just smooth; it respects boundaries and rare classes.

Mechanism: From Feature Maps to Spatial Predictions

Most modern PyTorch vision models begin with a backbone that converts an input tensor shaped [N, C, H, W] into feature maps. A detector head then maps feature cells or query embeddings to candidate boxes. In anchor-based designs, each location starts from predefined anchor boxes of several sizes and aspect ratios; the model predicts offsets and class logits for each anchor. In anchor-free designs, the model predicts points, distances to edges, or object queries directly. Either way, training assigns predictions to ground-truth objects, computes classification loss, and computes a localization loss such as smooth L1, generalized IoU, or a related box-overlap loss.

Segmentation heads preserve or recover spatial resolution. A semantic segmentation model emits logits shaped [N, K, H, W], where K is the number of classes. Cross entropy compares each pixel’s class logits with an integer class map shaped [N, H, W]. Instance segmentation adds an object-level branch: after boxes or proposals are found, the model predicts a small mask for each instance and projects it back to image coordinates. Detection has a variable number of predictions per image, while semantic segmentation has a dense prediction for every pixel.

Post-processing is part of the mechanism. Detectors usually filter low-confidence boxes, apply non-maximum suppression, and map coordinates back to the original image size. Segmentation systems upsample logits, choose argmax classes, threshold binary masks, and sometimes remove tiny connected components. These steps change measured quality, so they should be configured and tested with the same care as the neural network.

API and Tensor Anatomy

PyTorch supplies tensors, autograd, losses, and modules; the core vision data contract is still explicit. For detection, one image target is often a dictionary containing boxes as floating point [num_objects, 4] coordinates in xyxy order, labels as integer class ids, and optionally masks, area, and iscrowd. For semantic segmentation, the target is usually an integer image where each pixel stores a class id; an ignore index marks unlabeled pixels that should not contribute to loss.

Coordinate conventions are a frequent source of bugs. xyxy means left, top, right, bottom. xywh means left, top, width, height. Normalized coordinates live in [0, 1]; pixel coordinates live in image units. If images are resized, padded, cropped, or flipped, every box and mask must receive the same transform. Assert shapes, dtypes, coordinate ranges, and image-target alignment before training.

Worked Example 1: IoU for Boxes

Intersection over Union compares a predicted box with a ground-truth box. It is the overlapping area divided by the area covered by either box. A perfect match has IoU 1.0, disjoint boxes have IoU 0.0, and partial overlaps fall between. Many detection metrics treat a prediction as a true positive only if its class is correct and its IoU is above a threshold such as 0.5.

import torch

def box_iou_xyxy(boxes_a: torch.Tensor, boxes_b: torch.Tensor) -> torch.Tensor:
    top_left = torch.maximum(boxes_a[:, None, :2], boxes_b[None, :, :2])
    bottom_right = torch.minimum(boxes_a[:, None, 2:], boxes_b[None, :, 2:])
    wh = (bottom_right - top_left).clamp(min=0)
    intersection = wh[..., 0] * wh[..., 1]
    area_a = (boxes_a[:, 2] - boxes_a[:, 0]) * (boxes_a[:, 3] - boxes_a[:, 1])
    area_b = (boxes_b[:, 2] - boxes_b[:, 0]) * (boxes_b[:, 3] - boxes_b[:, 1])
    union = area_a[:, None] + area_b[None, :] - intersection
    return intersection / union.clamp(min=1e-6)

pred = torch.tensor([[0., 0., 4., 4.], [6., 6., 10., 10.]])
target = torch.tensor([[1., 1., 5., 5.]])
print(box_iou_xyxy(pred, target).round(decimals=4))

The first predicted box overlaps the target in a 3 by 3 square. The union is 16 plus 16 minus 9, so the IoU is 9/23, about 0.3913. The second box is disjoint, so its IoU is 0. Expected output is a two-row tensor containing approximately [[0.3913], [0.0000]].

Worked Example 2: Confidence Ranking and NMS

A detector often emits several boxes for the same object. Non-maximum suppression sorts boxes by score, keeps the highest-scoring box, and removes lower-scoring boxes whose IoU with it exceeds a threshold. NMS improves precision by reducing duplicates, but aggressive thresholds can suppress nearby real objects.

import torch

def box_iou_xyxy(boxes_a: torch.Tensor, boxes_b: torch.Tensor) -> torch.Tensor:
    top_left = torch.maximum(boxes_a[:, None, :2], boxes_b[None, :, :2])
    bottom_right = torch.minimum(boxes_a[:, None, 2:], boxes_b[None, :, 2:])
    wh = (bottom_right - top_left).clamp(min=0)
    intersection = wh[..., 0] * wh[..., 1]
    area_a = (boxes_a[:, 2] - boxes_a[:, 0]) * (boxes_a[:, 3] - boxes_a[:, 1])
    area_b = (boxes_b[:, 2] - boxes_b[:, 0]) * (boxes_b[:, 3] - boxes_b[:, 1])
    union = area_a[:, None] + area_b[None, :] - intersection
    return intersection / union.clamp(min=1e-6)

def nms(boxes: torch.Tensor, scores: torch.Tensor, threshold: float) -> list[int]:
    order = torch.argsort(scores, descending=True)
    keep: list[int] = []
    while order.numel() > 0:
        current = int(order[0])
        keep.append(current)
        if order.numel() == 1:
            break
        overlaps = box_iou_xyxy(boxes[current].unsqueeze(0), boxes[order[1:]]).squeeze(0)
        order = order[1:][overlaps <= threshold]
    return keep

boxes = torch.tensor([[0., 0., 4., 4.], [0.5, 0.5, 4.5, 4.5], [7., 7., 9., 9.]])
scores = torch.tensor([0.90, 0.80, 0.70])
print(nms(boxes, scores, threshold=0.5))

The first two boxes describe nearly the same object. Because box 0 has the higher score, it is kept and box 1 is removed when their IoU exceeds 0.5. Box 2 is far away and survives. The deterministic output is [0, 2].

Worked Example 3: Mask IoU and Dice

Segmentation quality is measured over pixels or masks. Mask IoU uses the same intersection-over-union idea as boxes, but on binary pixels. Dice doubles the intersection and divides by the total predicted and target foreground pixels. Dice is common in medical and imbalanced segmentation because it responds strongly to missed foreground.

import torch

def mask_scores(pred: torch.Tensor, target: torch.Tensor) -> tuple[float, float]:
    pred_bool = pred.bool()
    target_bool = target.bool()
    intersection = torch.logical_and(pred_bool, target_bool).sum().float()
    union = torch.logical_or(pred_bool, target_bool).sum().float()
    dice_denominator = pred_bool.sum().float() + target_bool.sum().float()
    iou = intersection / union.clamp(min=1.0)
    dice = (2 * intersection) / dice_denominator.clamp(min=1.0)
    return float(iou), float(dice)

pred_mask = torch.tensor([[1, 1, 0], [0, 1, 0], [0, 0, 0]])
target_mask = torch.tensor([[1, 0, 0], [0, 1, 1], [0, 0, 0]])
print(tuple(round(value, 4) for value in mask_scores(pred_mask, target_mask)))

There are two overlapping foreground pixels. The union contains four foreground pixels, so mask IoU is 0.5. Dice is 2 * 2 / (3 + 3), or 0.6667. The expected output is (0.5, 0.6667).

Worked Example 4: Semantic Segmentation Loss

For semantic segmentation, the model emits one logit per class per pixel. Cross entropy expects raw logits, not probabilities, and labels shaped without a channel dimension. The predicted class map is the argmax over the class dimension.

import torch

logits = torch.tensor([[[[4.0, -2.0], [0.5, 1.5]], [[-1.0, 3.0], [0.2, 0.1]]]])
labels = torch.tensor([[[0, 1], [0, 0]]])
loss = torch.nn.functional.cross_entropy(logits, labels)
predicted_classes = logits.argmax(dim=1)
print(round(float(loss), 4))
print(predicted_classes.tolist())

The logits have shape [1, 2, 2, 2]: one image, two classes, and a 2 by 2 spatial grid. The label tensor has shape [1, 2, 2]. The predicted class map is [[[0, 1], [0, 0]]], matching the labels, while the loss remains positive because logits are confident but finite.

Design Choices and Trade-offs

Mean Average Precision rewards ranking, class correctness, and localization across confidence thresholds; it fits detectors that return candidate lists. A fixed IoU-at-threshold report is easier to explain but hides ranking quality. Pixel accuracy can look excellent when background dominates, so mean IoU or class-balanced Dice is often more honest for segmentation. Per-image averages highlight user-visible misses, while global averages can be dominated by large images or frequent classes.

Model design has similar trade-offs. Larger input sizes help small objects but increase memory and latency. Stronger backbones improve features but slow inference. Anchor settings can boost objects whose sizes match the prior boxes, yet fail badly when the dataset scale changes. Mask resolution controls boundary quality and memory. NMS thresholds balance duplicate removal against preserving crowded objects. Thresholds for confidence and mask binarization should be selected on a validation set, not on the test set.

Failure Modes and Troubleshooting

Symptom: loss becomes NaN or boxes explode. The usual cause is invalid geometry: right coordinates less than left coordinates, zero-area boxes, boxes outside resized images, or an overly large learning rate. Diagnose by checking torch.isfinite, printing min and max coordinates after augmentation, and asserting positive widths and heights. Correct by fixing the transform pipeline, clipping boxes after resize or crop, dropping boxes with no remaining area, and lowering the learning rate if geometry is valid.

Symptom: validation AP is near zero while training loss decreases. A common cause is mismatched label ids, coordinate formats, or image scaling during evaluation. Inspect a small batch by drawing predicted and target boxes on the same image. Confirm whether targets use zero-based or one-based class ids and whether predictions are in normalized or pixel coordinates. Correct the conversion before computing metrics; do not tune thresholds until annotations and predictions are visibly aligned.

Symptom: segmentation pixel accuracy is high but masks are useless. Background dominance or ignored labels may be hiding foreground failures. Diagnose by computing per-class IoU, foreground Dice, and a confusion matrix over pixels. View examples from rare classes. Correct with class-balanced sampling, loss weighting, focal loss, more representative crops, or a metric that excludes easy background from the headline score.

Symptom: inference returns hundreds of duplicate boxes. NMS may be disabled, applied per image incorrectly, or using a threshold that is too high. Log the number of boxes before confidence filtering, after confidence filtering, and after NMS. Correct by applying NMS per class or with class-aware offsets, selecting a validation-backed score threshold, and enforcing a maximum detections limit.

Security, Performance, and Reliability Implications

Vision inputs are large, so reliability starts with resource limits. Validate file type and decoded dimensions before passing images to a model. Bound batch size, maximum detections, mask resolution, and post-processing loops. Store evaluation artifacts without private image content when the dataset is sensitive; use image ids and aggregate metrics unless visual examples are approved. For performance, measure preprocessing, model forward time, and post-processing separately. A detector can be fast in the backbone and slow in NMS when many candidates survive confidence filtering.

For reproducible evaluation, freeze the validation split, class mapping, resizing policy, confidence thresholds, NMS threshold, and metric implementation. Small changes in any of these can move AP or Dice without improving the model. Report slices such as small objects, crowded scenes, rare classes, low-light images, and heavy occlusion; they often fail differently from the aggregate score.

Hands-on Lab

Prerequisites: a Python environment with PyTorch installed, basic tensor familiarity, and any local image dataset or the synthetic tensors from the examples above. No GPU is required for the verification steps.

1. Create a scratch script and paste the IoU, NMS, mask scoring, and segmentation-loss examples.
2. Run the script and confirm the deterministic outputs: IoU around 0.3913 and 0.0000, NMS indices [0, 2], mask scores (0.5, 0.6667), loss around 0.1971, and the predicted class map [[[0, 1], [0, 0]]].
3. Add assertions for box shape [num_boxes, 4], finite coordinates, positive width and height, integer segmentation labels, and matching image-mask height and width.
4. Modify the second box in the NMS example so it is far away from the first box. Verify that all three indices are kept, because there is no duplicate overlap to suppress.
5. Change one foreground pixel in the target mask and observe how IoU and Dice move. Explain which metric changed more and why.
6. Cleanup by deleting the scratch script or reverting it to the original examples. If you used a real dataset, remove any temporary rendered images that contain sensitive content.

Verification is complete when the code runs, all assertions pass, and you can explain each metric change from the geometry or pixel counts rather than from intuition.

Assessment Exercises

1. A detector predicts a correct class with IoU 0.48, and the evaluation threshold is 0.50. How does that prediction affect precision and recall, and what change would be required for it to count as a true positive?
2. You resize an image from 1000 by 500 to 500 by 250, but leave target boxes unchanged. Describe the visual symptom, the metric symptom, and the exact coordinate correction.
3. For a segmentation dataset with 96 percent background pixels, choose two metrics you would report instead of headline pixel accuracy and justify what each reveals.
4. NMS removes one of two people standing shoulder to shoulder. Which threshold direction might help, and what new error could that introduce?
5. Design a validation slice for small-object detection. What examples belong in it, and which metric would you compare against the full validation set?

Summary

Detection, segmentation, and vision evaluation are about structured spatial predictions. PyTorch models emit tensors, but the engineering meaning comes from coordinate conventions, target dictionaries, pixel maps, matching rules, thresholds, and metric aggregation. Use IoU to reason about overlap, AP to evaluate ranked detections, Dice or mean IoU to evaluate masks, and visual diagnostics to catch format bugs that scalar losses hide. In this course, these tools are the bridge from convolutional features to vision systems that can be inspected and compared honestly.