Batch and Online Inference Services

Batch and online inference services are two ways to turn a trained PyTorch model into predictions for users, jobs, or downstream systems. Online inference answers one request, or a small group of live requests, while a caller waits. Batch inference scores a prepared dataset and writes results for later consumption. The model can be identical in both cases, but the service shape, latency target, retry behavior, and data validation are different.

In this PyTorch deployment lesson, the outcome is concrete: you should be able to package an nn.Module or exported artifact, put it in evaluation mode, feed tensors with the expected shape and dtype, choose between online and batch serving, and explain what breaks when the serving path silently diverges from training preprocessing.

How PyTorch Inference Runs

A PyTorch training loop records operations for autograd, updates parameters, and often uses training-only behavior such as dropout and batch normalization statistics. An inference service should do the opposite. It loads frozen parameters, calls model.eval(), wraps prediction in torch.inference_mode() or torch.no_grad(), converts request data into tensors, runs the model forward pass, then converts tensors back into a response format.

The core execution unit is still a tensor program. A model might receive a tensor shaped [batch, features], [batch, channels, height, width], or token tensors such as input_ids and attention_mask. The first dimension is usually the batch dimension. Online services often use batch size one, but many production servers collect several requests briefly and run one larger tensor through the model. Batch jobs use larger batches because throughput matters more than per-request wait time.

Device placement is part of the mechanism. Parameters and input tensors must be on the same device, commonly CPU or CUDA GPU. A mismatch produces a runtime error before the model can compute. A service also needs a deterministic preprocessing contract: the same normalization, vocabulary, category ordering, image resize policy, and missing-value handling used during training must be available during inference. Saving only state_dict is not enough if the feature pipeline lives in a notebook cell.

API Anatomy

A minimal online service has four functions: load the artifact once at process start, validate and transform incoming data, call the model, and format the result. A batch service uses the same model call but wraps it in a dataset reader, a DataLoader, checkpointed output writes, and usually a manifest that records model version, input path, output path, schema version, and row counts.

The safest API boundary names every feature and validates both type and size before tensor creation. A classifier response should identify the model version and class mapping, because a score of 1 is meaningless if the caller does not know which label index it represents.

Example 1: Single Online Prediction

This first example simulates a live request with three numeric features. The model uses fixed weights so the output is deterministic. The important serving details are eval(), inference_mode(), a two-dimensional request tensor, and conversion of the one-element tensor into a plain number for the response.

import torch
from torch import nn

class PurchaseModel(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.linear = nn.Linear(3, 1)
        with torch.no_grad():
            self.linear.weight.copy_(torch.tensor([[0.8, -0.4, 0.6]]))
            self.linear.bias.copy_(torch.tensor([-0.2]))

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return torch.sigmoid(self.linear(x))

model = PurchaseModel().eval()
request = torch.tensor([[1.0, 0.0, 2.0]])
with torch.inference_mode():
    probability = model(request).item()
print(round(probability, 4))

The expected output is 0.8581. The calculation is sigmoid of 0.8*1.0 - 0.4*0.0 + 0.6*2.0 - 0.2. In a real HTTP handler, the tensor would be built from validated request fields and the printed number would become a JSON response such as {"purchase_probability": 0.8581}.

Example 2: Offline Batch Scoring

Batch inference keeps the same forward pass but changes the outer loop. Instead of answering immediately, it preserves row ids, scores records in chunks, and writes results. The example uses shuffle=False because output rows must line up with input identities.

import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset

class PurchaseModel(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.linear = nn.Linear(3, 1)
        with torch.no_grad():
            self.linear.weight.copy_(torch.tensor([[0.8, -0.4, 0.6]]))
            self.linear.bias.copy_(torch.tensor([-0.2]))

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return torch.sigmoid(self.linear(x))

features = torch.tensor([
    [1.0, 0.0, 2.0],
    [0.0, 3.0, 1.0],
    [2.0, 1.0, 0.0],
    [1.0, 1.0, 1.0],
])
ids = torch.arange(features.shape[0])
loader = DataLoader(TensorDataset(ids, features), batch_size=2, shuffle=False)
model = PurchaseModel().eval()
rows: list[tuple[int, float]] = []
with torch.inference_mode():
    for batch_ids, batch_features in loader:
        scores = model(batch_features).squeeze(1)
        rows.extend((int(i), round(float(s), 4)) for i, s in zip(batch_ids, scores))
print(rows)

The expected output is [(0, 0.8581), (1, 0.31), (2, 0.7311), (3, 0.69)]. Notice that the model sees two rows at a time, producing a tensor shaped [2, 1] for each loader step. Batch jobs should tolerate restarts by writing partitioned output or temporary files that are atomically promoted after verification.

Example 3: Micro-Batching Live Requests

Micro-batching sits between the two modes. A service waits for a tiny window or until enough requests arrive, stacks compatible requests into one tensor, runs one forward pass, then splits the scores back by request id. This can improve GPU utilization, but it adds queueing delay and requires careful per-item error handling.

import torch
from torch import nn

class PurchaseModel(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.linear = nn.Linear(3, 1)
        with torch.no_grad():
            self.linear.weight.copy_(torch.tensor([[0.8, -0.4, 0.6]]))
            self.linear.bias.copy_(torch.tensor([-0.2]))

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return torch.sigmoid(self.linear(x))

def collate_requests(requests: list[dict[str, object]]) -> tuple[list[str], torch.Tensor]:
    ids: list[str] = []
    vectors: list[list[float]] = []
    for item in requests:
        features = item.get("features")
        if not isinstance(features, list) or len(features) != 3:
            raise ValueError(f"bad feature vector for {item.get('id')}")
        ids.append(str(item["id"]))
        vectors.append([float(value) for value in features])
    return ids, torch.tensor(vectors, dtype=torch.float32)

requests = [
    {"id": "a", "features": [1, 0, 2]},
    {"id": "b", "features": [0, 3, 1]},
    {"id": "c", "features": [2, 1, 0]},
]
model = PurchaseModel().eval()
ids, tensor = collate_requests(requests)
with torch.inference_mode():
    scores = model(tensor).squeeze(1)
response = {id_: round(float(score), 4) for id_, score in zip(ids, scores)}
print(response)

The expected output is {'a': 0.8581, 'b': 0.31, 'c': 0.7311}. If one request contains only two features, this implementation rejects the whole micro-batch. A more robust service would validate each item before batching and return an individual client error for the bad item while still scoring the valid items.

Design Choices and Trade-Offs

Choose online inference when the prediction changes the immediate user experience: fraud decisions, recommendations, moderation, autocomplete, or interactive copilots. The trade-off is operational pressure. The service must load quickly, control tail latency, shed excess traffic, and return useful errors when dependencies fail.

Choose batch inference when the result can be prepared ahead of time: nightly churn scores, product embeddings, risk reports, or backfills. Batch mode usually gives better hardware efficiency because it feeds larger tensors and avoids per-request overhead. The trade-off is staleness. A score generated at 02:00 may be wrong by noon if the user or inventory state changed.

Model format also matters. Keeping a Python nn.Module is flexible and easy to debug. Exporting with TorchScript, ONNX, or a serving runtime can reduce Python coupling and improve portability, but exported graphs may not support every dynamic Python branch. For this course, the key is to separate the stable tensor contract from the packaging choice: the service must know input names, shapes, dtypes, preprocessing, output interpretation, and version.

Failure Modes and Troubleshooting

• Symptom: live predictions differ from notebook evaluation. Likely cause: missing model.eval(), different normalization, or a changed category order. Diagnose: save one training validation row, run it through the service path, and compare intermediate tensors. Correct: package preprocessing with the model and add a golden-row test.
• Symptom: runtime error says tensors are on different devices. Likely cause: the model was moved to CUDA but request tensors stayed on CPU, or the reverse. Diagnose: log next(model.parameters()).device and input_tensor.device without logging raw user data. Correct: centralize device selection and move every tensor before the forward pass.
• Symptom: batch output has the right number of rows but wrong ids. Likely cause: shuffling, dropped invalid rows, or unordered joins after scoring. Diagnose: carry an immutable id through the Dataset, inspect first and last records per partition, and count rejected rows separately. Correct: disable shuffle for scoring and write id, score, model version, and error status together.
• Symptom: online latency spikes under load although average latency is acceptable. Likely cause: oversized micro-batches, GPU contention, cold starts, or synchronous preprocessing. Diagnose: measure queue time, preprocessing time, forward-pass time, and serialization time separately. Correct: cap batch wait time, warm the model, reduce payload size, or split CPU preprocessing from GPU inference.

Security, Performance, and Reliability

Inference endpoints should reject oversized inputs before allocating large tensors. Image, text, and tabular payloads can all be used to exhaust memory if the service trusts dimensions from the caller. Batch jobs need similar limits on partition size and retry count. Do not log full prompts, images, customer rows, or feature vectors unless policy allows it; log bounded ids, schema version, model version, latency, and failure category.

Performance tuning starts with measurement. For PyTorch, the most common wins are disabling gradient tracking, using consistent tensor shapes, choosing a batch size that fits memory, avoiding repeated model loads, and keeping data transfer between CPU and GPU under control. Reliability comes from making outputs reproducible: pin the artifact, record the code revision and preprocessing version, and make repeated batch runs either overwrite a known destination safely or create versioned outputs.

Hands-On Lab: Build Both Paths

Prerequisites: Python with PyTorch installed, a shell, and permission to run a local script. No web server is required; the lab exercises the serving core that an HTTP route or scheduled job would call.

1. Create a fresh directory and a file named inference_lab.py.
2. Add the model class from the examples and one function named predict_one(features) that validates exactly three numeric values, builds a tensor shaped [1, 3], and returns a rounded probability.
3. Add a function named score_batch(rows) where each row has an id and features. Preserve ids, use a DataLoader with shuffle=False, and return a list of dictionaries.
4. Run the online path with [1, 0, 2] and verify the result is 0.8581.
5. Run the batch path with the four rows from Example 2 and verify the ids remain 0, 1, 2, 3 with the expected scores.
6. Add a bad row with two features. Verify that the script reports a validation error before scoring rather than producing a shorter output silently.

Cleanup: remove the lab directory or keep only the script and a short README that records the expected outputs. If you wrote temporary result files, delete them after confirming the verification checks.

Assessment Exercises

1. A recommendation model is refreshed nightly, but users expect page loads under 100 ms. Which predictions should be precomputed by batch inference, and which should remain online? Justify the boundary.
2. Modify Example 3 so one malformed request returns an error while valid requests in the same micro-batch still receive scores. Describe the response shape.
3. Given a model trained with standardized numeric features, list the artifact files or metadata the inference service must load besides weights. Explain the failure if one is missing.
4. A batch job is interrupted after writing half of its output. Design a restart strategy that avoids duplicate or mixed-version scores.
5. Profile an online handler and find that tensor creation takes longer than the forward pass. Name two changes that could reduce latency without changing the model.

Summary

Batch and online inference are serving patterns around the same PyTorch forward pass. Online services optimize for immediate answers and predictable latency; batch services optimize for complete, restartable, high-throughput scoring. In both cases, correctness depends on evaluation mode, disabled gradients, stable preprocessing, explicit tensor shapes and dtypes, preserved identities, bounded inputs, and versioned outputs. Treat the model artifact and the inference code as one deployable contract, then choose the serving pattern that matches when the prediction is needed.