Convolutional Neural Networks

A convolutional neural network, or CNN, learns visual features by sliding small trainable filters across an image or feature map. In this PyTorch chapter the practical outcome is concrete: you should be able to predict tensor shapes, build a small classifier, choose kernel, stride, padding, pooling, and channel counts deliberately, and debug the common mistakes that appear when image tensors meet convolution layers.

CNNs matter in a deep learning course because they encode an assumption that ordinary fully connected networks ignore. Nearby pixels tend to be related, and the same visual pattern can appear in many locations. A convolution layer reuses the same weights at every spatial position, so it can detect an edge, texture, or part without learning a separate parameter for every pixel location.

What A CNN Computes

A PyTorch image batch is usually shaped N, C, H, W: batch size, channels, height, and width. A grayscale batch might be 16, 1, 28, 28; an RGB batch might be 32, 3, 224, 224. The layer nn.Conv2d expects that four-dimensional layout. Its weights have shape out_channels, in_channels, kernel_height, kernel_width. During the forward pass, each output channel is produced by one learned filter bank that spans all input channels.

At each spatial location, convolution multiplies a small input patch by the kernel weights, sums the result, and adds a bias when bias is enabled. The same kernel is used for every location. That weight sharing is why CNNs have far fewer parameters than a dense layer applied directly to pixels. A dense layer mapping a 3 x 224 x 224 image to 64 features would need millions of weights. A 3 x 3 convolution from 3 channels to 64 channels needs only 64 * 3 * 3 * 3 weights, plus optional biases.

The spatial output size is governed by kernel size, stride, padding, and dilation. With dilation left at its default, the height formula is floor((H + 2P - K) / S) + 1, and width follows the same pattern. Padding adds border values, usually zeros, before the kernel is applied. Stride moves the kernel by more than one pixel and reduces resolution. Larger kernels see more local context but add parameters and computation. Stacking small kernels often gives richer nonlinear features than one large kernel because each convolution can be followed by an activation such as nn.ReLU.

Layer Anatomy In PyTorch

The most common constructor is nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0). in_channels must equal the channel dimension of its input. out_channels chooses how many feature maps the layer learns. kernel_size may be an integer or a height-width tuple. stride controls movement across the image. padding controls whether borders shrink. After convolution, CNN blocks typically apply an activation, optional normalization, and sometimes pooling.

Pooling layers summarize local regions. nn.MaxPool2d(2) keeps the largest value in each non-overlapping 2 x 2 window when stride defaults to the kernel size, halving height and width. nn.AdaptiveAvgPool2d((1, 1)) is useful near the classifier because it converts each channel to a single average regardless of the input’s earlier spatial size. Finally, nn.Flatten() changes N, C, H, W into N, C*H*W so a linear classifier can consume it.

Example 1: One Convolution And Its Shape

This first example keeps spatial size unchanged. The input has four RGB images of size 32 x 32. The convolution has eight output channels, a 3 x 3 kernel, stride one, and padding one. The padding compensates for the border pixels consumed by the kernel, so height and width stay at 32 while the channel dimension changes from 3 to 8.

import torch
from torch import nn

conv = nn.Conv2d(in_channels=3, out_channels=8, kernel_size=3, stride=1, padding=1)
images = torch.randn(4, 3, 32, 32)
features = conv(images)
print(tuple(features.shape))

The deterministic output is the shape (4, 8, 32, 32). The numeric feature values are not deterministic because the input and initial weights are random, but the dimensions are fixed by the layer configuration. This is the first habit to build with CNNs: reason about the shape before training.

Example 2: A Minimal Image Classifier

A classifier usually alternates feature extraction and resolution reduction. The next model accepts grayscale 28 x 28 images, creates four feature maps, halves the spatial size with max pooling, creates eight higher-level feature maps, averages each channel down to one number, then emits ten class logits. Logits are raw scores; during training they can be passed directly to nn.CrossEntropyLoss, which applies the appropriate log-softmax internally.

import torch
from torch import nn

model = nn.Sequential(
    nn.Conv2d(1, 4, kernel_size=3, padding=1),
    nn.ReLU(),
    nn.MaxPool2d(kernel_size=2),
    nn.Conv2d(4, 8, kernel_size=3, padding=1),
    nn.ReLU(),
    nn.AdaptiveAvgPool2d((1, 1)),
    nn.Flatten(),
    nn.Linear(8, 10),
)

batch = torch.randn(16, 1, 28, 28)
logits = model(batch)
print(tuple(logits.shape))

The expected printed shape is (16, 10): one row per image and one score per class. Notice that AdaptiveAvgPool2d((1, 1)) removes the need to hard-code 8 * 14 * 14 in this specific architecture. That makes the classifier head less brittle if earlier spatial sizes change, although it also discards spatial arrangement at the final stage.

Example 3: Training Step With Backpropagation

A CNN trains like other PyTorch modules: compute logits, compute loss, call backward(), and ask the optimizer to update parameters. This example uses a named module because real projects usually outgrow a single nn.Sequential. The class separates feature extraction from classification and explicitly flattens from the second dimension onward, preserving the batch dimension.

import torch
from torch import nn

torch.manual_seed(7)

class TinyCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 6, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2),
        )
        self.classifier = nn.Linear(6 * 14 * 14, 2)

    def forward(self, x):
        x = self.features(x)
        x = torch.flatten(x, 1)
        return self.classifier(x)

model = TinyCNN()
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.05)
images = torch.randn(8, 1, 28, 28)
labels = torch.tensor([0, 1, 0, 1, 0, 1, 0, 1])

logits = model(images)
loss = criterion(logits, labels)
loss.backward()
optimizer.step()
print(logits.shape, labels.shape)

The printed result is torch.Size([8, 2]) torch.Size([8]). The logits have two columns because this toy task has two classes. The labels are a one-dimensional integer tensor because CrossEntropyLoss expects class indices, not one-hot encoded rows, for ordinary multiclass classification.

Design Choices And Trade-Offs

Kernel size controls local receptive field. A 3 x 3 kernel is cheap and common; two stacked 3 x 3 layers see a broader region than one layer and insert an extra nonlinearity. A 1 x 1 convolution mixes channel information without looking across neighboring pixels, which is useful for changing channel count cheaply. Larger kernels may help when low-level context is genuinely broad, but they increase compute and can blur the advantage of locality.

Stride and pooling both reduce spatial resolution. Reduction saves memory and expands the effective receptive field of later units, but aggressive downsampling can erase small objects or fine boundaries. Padding preserves border information and convenient shapes, but too much padding lets the model spend capacity on artificial borders. Channel count sets representational width. More channels can learn more feature types but raise memory use, latency, and overfitting risk.

For small datasets, a CNN from scratch may memorize. Data augmentation, weight decay, dropout in the classifier head, and transfer learning from a pretrained backbone are common responses. For large images or deployment on limited hardware, use smaller input sizes, depthwise separable convolutions, quantization-aware workflows, or a lighter architecture. The right choice depends on the error pattern, latency budget, and cost of false positives versus false negatives.

Failure Modes And Troubleshooting

Symptom: PyTorch reports that it expected a different number of channels. Cause: in_channels in the convolution does not match the input’s C dimension, or the tensor is laid out as N, H, W, C. Diagnose: print batch.shape immediately before the layer. Correct: construct the layer with the right channel count or convert image batches with batch = batch.permute(0, 3, 1, 2) when your loader produced channel-last tensors.

Symptom: the linear layer fails with a matrix multiplication shape error. Cause: the flattened feature size is not what the classifier expects. Diagnose: pass one batch through model.features and print its shape before flattening. Correct: recompute in_features, use nn.LazyLinear carefully during prototyping, or add adaptive pooling to make the head independent of spatial size.

Symptom: training accuracy rises while validation accuracy stays poor. Cause: overfitting, data leakage, mismatched preprocessing, or an evaluation split that differs from training distribution. Diagnose: compare train and validation transforms, inspect misclassified images, and evaluate per class rather than only overall accuracy. Correct: simplify the model, add augmentation, collect more representative data, rebalance classes, or use a pretrained backbone.

Symptom: loss becomes nan or accuracy never improves. Cause: learning rate may be too high, labels may have the wrong dtype or range, inputs may be unnormalized, or the model may be in the wrong mode. Diagnose: check images.dtype, labels.dtype, min and max label values, input value ranges, and gradient norms. Correct: normalize inputs consistently, use torch.long labels for CrossEntropyLoss, reduce the learning rate, and call model.train() during training and model.eval() during evaluation.

Reliability, Performance, And Data Concerns

CNN reliability starts with consistent preprocessing. The same resizing, channel order, scaling, and normalization used during training must be applied during inference. A model trained on RGB images scaled to 0..1 can fail silently if production sends BGR images scaled to 0..255. Record the transform pipeline with the model artifact, and test inference on known images with expected classes.

Performance is dominated by image size, channel counts, batch size, and convolution choices. Measure throughput and latency on the hardware that will run inference. Larger batches improve accelerator utilization but increase memory and may harm interactive latency. Reliability also includes class imbalance and domain shift. A CNN can look accurate while failing on rare classes, different cameras, lighting changes, compression artifacts, or rotated objects. Slice metrics by meaningful visual conditions when the task carries real consequences.

Hands-On Lab: Build And Verify A CNN Block

Prerequisites: Python with PyTorch installed, a terminal or notebook, and basic familiarity with tensors. No external dataset is required because the lab uses synthetic inputs for shape verification.

  1. Create a new scratch script named cnn_lab.py outside production code.
  2. Define a module with two Conv2d layers, ReLU after each, one MaxPool2d(2), AdaptiveAvgPool2d((1, 1)), Flatten, and a Linear layer for three classes.
  3. Feed a tensor shaped 5, 3, 64, 64 through the model and assert that the output shape is 5, 3.
  4. Intentionally change the input to 5, 64, 64, 3. Verify that the model fails because channel position is wrong.
  5. Fix the tensor with permute(0, 3, 1, 2) and verify the output shape again.
  6. Run one training step using CrossEntropyLoss and labels shaped 5 with values from 0 to 2.

Verification: the script should print or assert the final logits shape (5, 3), and the intentional channel-last input should produce a clear channel mismatch before you fix it. Cleanup: delete the scratch script or keep it in a learning directory; no model files or external resources are created unless you add saving yourself.

Assessment Exercises

  1. An input has shape 10, 3, 64, 64. A convolution uses out_channels=12, kernel_size=5, stride=2, and padding=2. Compute the output shape and explain each dimension.
  2. Replace a hard-coded flatten size with adaptive pooling in a small CNN. Explain what flexibility you gained and what spatial information you discarded.
  3. A model performs well on centered objects but poorly when objects shift toward image edges. Which architecture choices and augmentations would you inspect first?
  4. You receive labels shaped N, num_classes for a model trained with CrossEntropyLoss. Explain why that may be wrong and how you would verify the expected target format.
  5. Design a validation report for a CNN used on camera images. Include at least three slices beyond aggregate accuracy.

Summary

CNNs work by applying learned local filters across spatial tensors, sharing weights across positions, expanding channel representations, and reducing resolution when useful. In PyTorch, most CNN bugs become understandable when you track N, C, H, W, match in_channels, compute spatial sizes, and keep preprocessing identical between training and inference. Start with shape reasoning, then use metrics and inspected examples to decide whether the architecture is learning the visual evidence your task actually requires.