Recurrent Networks, LSTMs, and GRUs

Recurrent networks let a neural network read an ordered sequence one step at a time while carrying information forward in a hidden state. In this lesson, the outcome is practical: you should be able to choose between a vanilla RNN, an LSTM, and a GRU in PyTorch; predict the tensor shapes they return; and diagnose the training problems that appear when sequence length, padding, or hidden state handling is wrong.

This chapter sits in the sequence part of the Deep Learning with PyTorch course because recurrence is the classic bridge between feed-forward models and transformers. Transformers usually dominate large language tasks, but recurrent layers remain useful for smaller time-series models, streaming signals, embedded inference, and cases where a compact stateful model is easier to run than full attention over a long context.

Purpose and Outcome

A recurrent layer maps an input sequence x_1, x_2, ..., x_T to output vectors while updating a hidden vector at each time step. The same learned weights are reused at every step, so the model can process sequences of different lengths. The hidden state is the model’s running summary: after reading earlier tokens, samples, or timesteps, it influences how the current item is interpreted.

The basic PyTorch choices are nn.RNN, nn.LSTM, and nn.GRU. They share a similar interface, but they do not have the same internal dynamics. A vanilla RNN is simple and cheap, yet it struggles to preserve information over long spans because gradients repeatedly pass through the same transformation. LSTMs add gates and a separate cell state to protect long-term information. GRUs use fewer gates and no separate cell state, often giving a strong middle ground when speed and parameter count matter.

How Recurrence Works Internally

At time step t, a vanilla RNN combines the current input vector and the previous hidden vector, applies learned affine transformations, and passes the result through a nonlinearity such as tanh. Conceptually, the update is h_t = tanh(W_x x_t + W_h h_prev + b). The key idea is weight sharing across time: the same matrices are used for every position, so the layer learns transition rules rather than separate weights for position one, position two, and so on.

Training uses backpropagation through time. PyTorch records the chain of operations over the sequence and sends gradients backward from later outputs to earlier recurrent states. Long chains can shrink gradients until early steps barely learn, or amplify them until training becomes unstable. Gradient clipping is common for recurrent models because it bounds a large update before the optimizer applies it.

An LSTM changes the internal state update. It maintains both hidden state h_t and cell state c_t. The cell state acts like a controlled memory path. The forget gate decides which previous cell values to keep, the input gate decides what new candidate information to write, and the output gate decides how much cell information becomes the exposed hidden state. The gates are learned sigmoid outputs, so each dimension can be kept, overwritten, or exposed independently.

A GRU merges some of that machinery. It has an update gate that controls how much previous state survives and a reset gate that controls how strongly the previous state participates in the candidate update. Because a GRU has no separate cell state, it returns only a hidden state. This makes it lighter than an LSTM, though sometimes less expressive for tasks where a long-lived memory lane is valuable.

PyTorch API Anatomy

All three recurrent modules accept an input tensor, optional initial state, and return sequence outputs plus final state. With batch_first=True, the input shape is (batch, time, features). Without it, PyTorch expects (time, batch, features). The output tensor contains the hidden representation at every time step. The final state summarizes the last step for every layer and direction.

The important constructor arguments are input_size, hidden_size, num_layers, batch_first, bidirectional, and dropout. input_size must equal the number of features per time step, such as embedding width for text or sensor channels for time series. hidden_size controls memory capacity and output width. num_layers stacks recurrent layers vertically. Dropout inside these modules applies between stacked layers, so it has no effect when num_layers is one.

State shapes are the source of many bugs. For a unidirectional single-layer RNN or GRU, h_n has shape (1, batch, hidden_size). For an LSTM, the state is a pair (h_n, c_n) with the same leading shape. If the layer is bidirectional, the first dimension becomes num_layers * 2. When feeding a classifier, h_n[-1] is commonly used for a single-direction final representation, while bidirectional models usually concatenate the final forward and backward states.

Worked Example 1: Inspect a Vanilla RNN

The first example verifies the minimum shape contract. Two sequences, each with five time steps and three features per step, are passed through a four-unit RNN. The output keeps one four-dimensional vector for every time step. The final hidden state keeps one vector per layer for each batch item.

import torch
from torch import nn

torch.manual_seed(7)
# batch_first=True makes input shape (batch, time, features).
rnn = nn.RNN(input_size=3, hidden_size=4, batch_first=True)
x = torch.randn(2, 5, 3)
y, h_n = rnn(x)
print(tuple(y.shape))
print(tuple(h_n.shape))

The deterministic output is the two printed shapes: (2, 5, 4) and (1, 2, 4). The actual floating-point values are not printed because they depend on initialized weights, but the shapes are fixed by the module configuration. If you remove batch_first=True while keeping the same input tensor, PyTorch will interpret the leading dimension as time instead of batch, silently changing the meaning of your data.

Worked Example 2: Read LSTM State

This example uses a two-layer LSTM. The input has four sequences, six time steps, and two features. The sequence output reports the top layer’s hidden vector at each time step. h_n and c_n each contain one final state per layer, so their first dimension is two.

import torch
from torch import nn

torch.manual_seed(3)
lstm = nn.LSTM(input_size=2, hidden_size=5, num_layers=2, batch_first=True)
x = torch.randn(4, 6, 2)
output, (h_n, c_n) = lstm(x)
last_layer_hidden = h_n[-1]
print(tuple(output.shape))
print(tuple(last_layer_hidden.shape))
print(tuple(c_n.shape))

The expected output is (4, 6, 5), (4, 5), and (2, 4, 5). Notice that output[:, -1, :] and h_n[-1] match for a simple unidirectional LSTM without packing complications. With padded variable-length batches, the last column of output may point at padding for shorter sequences, so using lengths or packed sequences becomes important.

Worked Example 3: Sequence Classification with a GRU

A common pattern is embedding tokens or events, running a recurrent layer, and classifying from the final hidden state. This small GRU reads a batch of embedded token sequences and maps each sequence to two logits. In a real text model, embedded_tokens would usually come from nn.Embedding.

import torch
from torch import nn

torch.manual_seed(11)
gru = nn.GRU(input_size=8, hidden_size=6, batch_first=True)
classifier = nn.Linear(6, 2)
embedded_tokens = torch.randn(3, 4, 8)
_, h_n = gru(embedded_tokens)
logits = classifier(h_n[-1])
print(tuple(logits.shape))
print(torch.isfinite(logits).all().item())

The expected output is (3, 2) followed by True. The logits shape confirms that the classifier receives one six-dimensional representation per sequence and returns one two-class score vector per batch item. The finite-value check is a small sanity test that the recurrent and linear layers produced usable numeric outputs.

Design Choices and Trade-Offs

Use a vanilla RNN when sequences are short, the task is simple, and interpretability or minimal compute matters. Use an LSTM when the task needs information from much earlier steps and you can afford extra parameters. Use a GRU when you want gated recurrence with lower cost and a simpler state interface. These are empirical choices: compare them against a feed-forward baseline using the same train, validation, and test split.

Sequence length changes both accuracy and cost. Truncating a sequence may remove decisive context, but very long sequences make training slower and increase gradient path length. Truncated backpropagation through time processes long streams in windows, carrying hidden state between windows while detaching the computational graph so memory does not grow forever. This is useful for streaming, but the detach point limits which earlier steps can receive gradient credit.

Bidirectional recurrence reads a sequence forward and backward, often improving offline classification because each position can use past and future context. It is a poor fit for live forecasting where future inputs are not available at prediction time. Packed sequences reduce wasted compute on padding, but they add complexity: lengths must match the unpadded examples, and outputs may need to be unpacked before later layers.

Failure Modes and Troubleshooting

• Symptom: loss becomes nan or jumps sharply. Likely cause: exploding gradients through repeated recurrent transitions. Diagnose: log gradient norms before the optimizer step and check whether they spike on long batches. Correction: use torch.nn.utils.clip_grad_norm_, lower the learning rate, and normalize input scales.
• Symptom: validation accuracy is high in notebooks but poor in deployment. Likely cause: sequence leakage or future context, especially from bidirectional layers or window construction. Diagnose: audit how windows are built and confirm that validation examples occur after training examples for forecasting tasks. Correction: split by time or entity before windowing and avoid bidirectional recurrence for causal predictions.
• Symptom: the model learns padding tokens or short sequences behave strangely. Likely cause: using output[:, -1, :] when the final column is padding for many examples. Diagnose: print true lengths and compare the selected timestep with each example’s last real timestep. Correction: use packed sequences or gather the output at length - 1 for each sample.
• Symptom: runtime shape errors mention hidden state size. Likely cause: initial hidden state was created for the wrong batch size, number of layers, or direction count. Diagnose: print expected state shape from num_layers * num_directions, current batch size, and hidden_size. Correction: initialize state per batch or let PyTorch default to zeros when no custom state is required.

Reliability and Performance Implications

Recurrent models are sequential along the time dimension, so they parallelize less naturally than convolution or attention. This can make training slow on long sequences even when the parameter count is small. Batch work across examples, keep tensors on one device, and consider packing padded batches when length variation is large. For inference, a recurrent model can be efficient in streaming mode because it only needs the new input and previous hidden state rather than recomputing a full context window.

Reliability depends on making state ownership explicit. In stateless batch inference, every request starts with zero state. In streaming inference, the caller or service must decide when to reset state, how to key it to a user or device, and how to prevent one stream’s hidden state from contaminating another. Save model weights and preprocessing parameters together, because a changed tokenizer, feature order, or normalization scale can break a recurrent model while leaving tensor shapes valid.

Hands-On Lab: Train a Tiny Next-Step Classifier

Prerequisites: Python, PyTorch, and a CPU environment are enough. Create a scratch script or notebook cell; no dataset download is required. The synthetic task is to read a short binary sequence and predict whether the final value is greater than the first value. It is intentionally small so you can focus on recurrent shapes and state.

1. Create training tensors with shape (batch, time, 1). Use random zeros and ones converted to floats.
2. Build nn.GRU(input_size=1, hidden_size=8, batch_first=True) and nn.Linear(8, 2).
3. Run the GRU, take h_n[-1], pass it through the linear layer, and optimize with cross-entropy loss.
4. Verify that logits have shape (batch, 2) and that training loss falls below the initial loss after several hundred mini-batches.
5. Test edge cases: all-zero sequences, all-one sequences, and sequences of length one. Define the expected label before evaluating.
6. Cleanup is simple: delete the scratch script or restart the notebook kernel. If you saved a checkpoint, remove it or label it as synthetic-only so it is not reused for real data.

Verification should include both tensor checks and behavior checks. A model that prints the right shape can still solve the wrong target if labels are off by one timestep. For this lab, inspect a few generated sequences, print their labels, and confirm that the first and last values match the rule before trusting the training curve.

Assessment Exercises

1. You receive text batches padded to the same length and a tensor of true lengths. Explain why output[:, -1, :] can be wrong, then write the indexing logic you would use instead.
2. A forecasting model uses a bidirectional LSTM and performs well in validation. What experiment would reveal whether it is using future information unavailable at inference time?
3. Compare an LSTM with hidden_size=64 and a GRU with hidden_size=64. Which has more parameters, and what practical trade-off follows from that difference?
4. Your GRU training loss is unstable only on long sequences. Describe a diagnostic metric and two corrections you would try before changing the dataset.
5. Design a state reset policy for streaming sensor inference. What event should reset hidden state, and what bug appears if resets are missed?

Summary

RNNs, LSTMs, and GRUs are sequence layers that reuse weights across time and carry information in hidden state. Vanilla RNNs are light but fragile over long spans; LSTMs preserve memory with gates and cell state; GRUs keep gated behavior with a smaller interface. In PyTorch, most mistakes come from tensor layout, state shape, padding, and causal assumptions. Treat recurrent models as state machines over tensors, verify shapes at every boundary, and choose the simplest recurrent cell that beats a defensible baseline for your sequence task.