Deep Learning and the PyTorch Workflow

Deep learning uses layered mathematical functions to turn examples into predictions. In PyTorch, the workflow is deliberately direct: represent data as tensors, define a differentiable computation, measure error with a loss function, let automatic differentiation compute gradients, and let an optimizer update learnable parameters. This lesson establishes the working mental model for the rest of the course: every later model, from image classifiers to sequence models, is a larger version of this same loop.

The outcome is practical. You should be able to read a PyTorch training script and identify where data enters, where predictions are made, where gradients are created, where parameters change, and where evaluation must avoid training behavior. You should also be able to debug the common early failures: wrong tensor shapes, missing gradients, exploding loss, device mismatches, and misleading validation results.

The Core Mechanism

A PyTorch program is built around tensors. A tensor is a multidimensional array with a dtype, shape, and device. A batch of 32 grayscale images might have shape [32, 1, 28, 28]; a batch of tabular rows with four features might have shape [32, 4]. The shape is not decoration. It determines which operations are valid and what the model sees as one example.

When a tensor participates in operations that involve parameters with requires_grad=True, PyTorch records a dynamic computation graph. Dynamic means the graph is built as Python runs, not compiled from a separate static model file. Each differentiable operation stores enough information to compute local derivatives later. Calling loss.backward() walks backward through that graph and accumulates gradients into each parameter’s .grad field.

The optimizer does not compute gradients. It reads them. A typical step is therefore ordered: run the model, compute the loss, clear old gradients, call backward, then call optimizer.step(). Clearing gradients matters because PyTorch accumulates by default. Accumulation is useful for advanced training patterns, but in a beginner loop accidental accumulation makes updates too large and hard to reason about.

Workflow Anatomy

The usual API shape is compact. Subclass nn.Module, create layers in __init__, and implement forward. PyTorch calls forward when you call the module like a function: pred = model(x). Parameters are discovered automatically when layers are assigned as module attributes, which is why layers should normally be created in __init__ rather than inside forward.

Example 1: Tensor Shapes And A Forward Pass

This first example shows the smallest useful forward computation. A linear layer with three inputs and two outputs contains a weight matrix of shape [2, 3] and a bias vector of shape [2]. Feeding a batch with shape [4, 3] produces a prediction tensor with shape [4, 2]: four examples, two outputs per example.

import torch
from torch import nn

torch.manual_seed(7)
layer = nn.Linear(3, 2)
x = torch.randn(4, 3)
y = layer(x)

print(x.shape)
print(y.shape)
print(layer.weight.shape)
print(layer.bias.shape)

The deterministic behavior here is the set of shapes, not the learned values. The output is torch.Size([4, 3]), torch.Size([4, 2]), torch.Size([2, 3]), and torch.Size([2]). If x had shape [4, 4], the layer would fail because the last input dimension would not match the layer’s expected three features.

Example 2: Autograd On A Single Parameter

The next example exposes the training mechanism without hiding it inside a module. We create one scalar parameter w, predict w * x, compute mean squared error, and inspect the gradient. Because the target relationship is y = 2x and w starts at 1, the gradient is negative: increasing w will reduce the loss.

import torch

x = torch.tensor([1.0, 2.0, 3.0])
y_true = torch.tensor([2.0, 4.0, 6.0])
w = torch.tensor(1.0, requires_grad=True)

y_pred = w * x
loss = torch.mean((y_pred - y_true) ** 2)
loss.backward()

print(round(loss.item(), 4))
print(round(w.grad.item(), 4))

The expected output is 4.6667 and -9.3333. The loss is a scalar because backward() needs a single objective unless you explicitly provide gradient arguments. The gradient lives on w.grad; it is not returned from backward(). This distinction is important when you later inspect model parameters during debugging.

Example 3: A Complete Tiny Training Loop

A real PyTorch workflow wraps parameters inside a module and delegates updates to an optimizer. This example fits a small linear model to synthetic data. The training loop has the canonical order: set training mode, predict, compute loss, clear gradients, backpropagate, update, and periodically report loss.

import torch
from torch import nn

torch.manual_seed(3)
x = torch.linspace(-1, 1, 20).unsqueeze(1)
y = 3 * x - 0.5

model = nn.Linear(1, 1)
loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

for epoch in range(60):
    model.train()
    pred = model(x)
    loss = loss_fn(pred, y)

    optimizer.zero_grad(set_to_none=True)
    loss.backward()
    optimizer.step()

print(round(loss.item(), 4))
print(round(model.weight.item(), 2))
print(round(model.bias.item(), 2))

The final loss should be small, and the printed weight and bias should be close to 3.0 and -0.5. Exact last decimals can vary with environment details, but the trend is deterministic: the loss falls because gradients repeatedly adjust the linear layer toward the generating relationship. If the learning rate were much larger, the same loop could overshoot and the loss could grow instead.

Example 4: Evaluation Without Training Effects

Evaluation must not update parameters or build unnecessary gradient graphs. The pattern is model.eval() plus torch.no_grad(). The first changes behavior of training-sensitive layers such as dropout and batch normalization. The second tells autograd not to track operations, reducing memory use and preventing accidental gradient inspection from stale training state.

import torch
from torch import nn

model = nn.Sequential(nn.Linear(2, 4), nn.ReLU(), nn.Linear(4, 1))
x_val = torch.tensor([[0.25, 0.75], [1.0, -1.0]])

model.eval()
with torch.no_grad():
    predictions = model(x_val)

print(predictions.shape)
print(predictions.requires_grad)

The expected output is torch.Size([2, 1]) and False. This does not mean the model has no trainable parameters. It means this specific prediction tensor is not attached to a graph. That is exactly what you want for validation, test evaluation, and inference.

Design Choices And Trade-Offs

PyTorch favors imperative Python control flow. You can use ordinary loops, conditionals, debugging prints, and breakpoints while the graph is created dynamically. This makes research and teaching approachable, and it makes unusual model structures easier to express. The trade-off is that performance and deployment often require more discipline: avoid Python work inside tight inner loops, batch operations whenever possible, and separate training-time flexibility from inference-time efficiency.

Another design choice is explicit training control. PyTorch will not infer that you meant to evaluate just because your variable is named x_val. You call train() and eval(); you decide when gradients are tracked; you decide which parameters the optimizer owns. This explicitness prevents hidden magic, but it also means silent mistakes are possible. A validation loop without no_grad() may still produce numerically correct predictions while wasting memory.

Batch size, learning rate, optimizer, and loss function are not interchangeable knobs. Batch size affects memory use and gradient noise. Learning rate controls update size. Optimizers such as SGD and Adam use gradients differently. Loss functions encode what errors matter: mean squared error is natural for many regression tasks, while cross entropy is the standard starting point for single-label classification logits.

Failure Modes And Troubleshooting

• Shape mismatch: the symptom is an error mentioning matrix multiplication shapes, often during a linear layer. Check the printed shape immediately before the failing layer. Correct the feature dimension, flattening logic, or layer input size.
• Loss does not decrease: symptoms include a flat or increasing training loss. Confirm that optimizer.zero_grad(), loss.backward(), and optimizer.step() all run in the correct order. Then inspect learning rate, target scaling, and whether model parameters are actually passed to the optimizer.
• Gradients are missing: the symptom is param.grad is None after backward. Check whether computation used non-differentiable conversions such as .item() in the loss path, whether tensors were detached, or whether the parameter was excluded from the forward computation.
• Device mismatch: the symptom says tensors are on different devices. Print tensor.device for inputs, targets, and model parameters. Move the model and every batch tensor to the same device before the forward pass.
• Validation looks too good: the cause is often data leakage or evaluating on training data. Verify split creation, transforms, shuffling, and whether fitted preprocessing statistics were computed only from training data.

Security, Performance, And Reliability

For early PyTorch work, the main reliability risk is drawing conclusions from an experiment that cannot be reproduced. Set random seeds for demonstrations, record data splits, keep preprocessing code with the model code, and save enough metadata to explain a result later. For performance, prefer vectorized tensor operations over Python loops across examples. Use no_grad() for inference, choose batch sizes that fit memory with headroom, and measure throughput on representative input sizes.

Security matters when models and data come from outside your environment. Do not load arbitrary serialized model files from untrusted sources. Treat datasets as potentially sensitive, especially when examples contain personal or proprietary information. Keep training data paths, credentials, and exported artifacts under the same access controls you would apply to any other valuable application asset.

Hands-On Lab

Prerequisites: Python with PyTorch installed, a terminal or notebook, and enough CPU memory for small tensors. No GPU is required.

1. Create a fresh script named tiny_regression.py.
2. Paste the complete tiny training loop from Example 3.
3. Run it once and record the final loss, weight, and bias.
4. Change the learning rate from 0.1 to 1.0. Run again and compare the loss behavior.
5. Restore the learning rate to 0.1, then comment out optimizer.zero_grad(set_to_none=True). Run again and observe whether training becomes less stable.
6. Add an evaluation block using model.eval() and torch.no_grad() that predicts for x = [[2.0]].

Verification: with the original settings, the learned weight should be near 3, the bias near -0.5, and prediction for 2.0 near 5.5. If those checks fail, print tensor shapes, confirm the target formula, and confirm the optimizer receives model.parameters().

Cleanup: remove the temporary script if you do not need it, or keep it as a reference for later lessons. If you changed an environment just for the lab, deactivate it after finishing.

Assessment Exercises

1. A model receives input with shape [64, 10] and uses nn.Linear(8, 3). Explain the failure and give two valid corrections.
2. In a training loop, what changes if optimizer.zero_grad() is removed? Describe when accumulation is useful and when it is a bug.
3. Write a validation loop for a trained model and explain why both eval() and no_grad() appear in it.
4. A loss becomes nan after a few batches. List three diagnostic checks in the order you would run them.
5. Given a binary classifier that performs well on training data but poorly on validation data, identify whether the first suspect should be autograd, optimization, or generalization, and justify your answer.

Summary

The PyTorch workflow is a disciplined cycle: create tensors, run an nn.Module forward pass, compute a scalar loss, clear old gradients, call backward(), update parameters with an optimizer, and evaluate without training effects. Once this cycle is clear, larger deep learning systems become easier to understand because their complexity is mostly in data handling, architecture design, loss selection, and experiment control rather than in a different training mechanism.