Batch, Online, Streaming, and Edge Inference

Batch, online, streaming, and edge inference are four ways to turn a trained model artifact into predictions. The outcome is to choose the serving shape that matches the work: input freshness, response latency, record volume, data location, and failure behavior when the model, feature store, or network is unavailable.

In MLOps, serving mode is a deployment contract. It determines how features are computed, how model versions are selected, how results are stored or returned, how drift is detected, and where rollback happens. The same fraud model, recommender, or forecast can be correct in one mode and unusable in another because timing and reliability assumptions differ.

What Each Mode Does

Batch inference scores a bounded set of records at scheduled or manually triggered times. Inputs usually come from files, tables, or partitions; outputs are written back to a warehouse, object store, queue, or operational table. It fits nightly churn scores, weekly forecasts, and catalog re-ranking before a campaign. Its internal unit of work is a job: select records, load model, compute features, score, validate, write results, and mark the partition complete.

Online inference scores one request or a small group of requests while a user or service waits. The model is loaded into a long-running process behind an API. The internal unit is a request: validate payload, fetch or compute features, run the model, post-process, return a response, and emit request telemetry. It fits credit decisions, search ranking, personalization, and real-time risk checks.

Streaming inference scores events as they flow through a stream processor. The unit is an event or micro-batch with an offset. The system consumes events, joins state or windowed aggregates, scores them, writes predictions, and commits offsets. It fits anomaly detection, sensor monitoring, and transaction scoring where the answer should follow the event quickly without requiring a synchronous API response.

Edge inference runs the model close to the source of data, such as a mobile device, browser, camera, robot, vehicle, or factory gateway. The unit is local input under constrained CPU, memory, battery, and storage. It fits low-latency vision, privacy-sensitive audio, offline recommendations, and unreliable connectivity. Updates are packaged, signed, staged, and sometimes rolled back across many devices.

Mechanism and Internals

The modes differ most in where they place the model, feature computation, state, and acknowledgement. Batch puts state in durable stores and treats completion as a partition-level fact. Online keeps model and feature clients warm in memory and treats each response as externally visible immediately. Streaming places state in the stream processor and uses offsets or checkpoints so events are not silently skipped. Edge packages model, preprocessing, and sometimes calibration thresholds into a bundle that can run without a datacenter round trip.

Feature freshness is often decisive. A batch churn score might use yesterday’s account aggregates and remain useful for days. An online fraud score might need the last transaction count in the past ten minutes. A streaming sensor model might use a rolling average over the last 100 readings. An edge vision model might use only pixels and local device metadata because sending frames to a server would be too slow or too sensitive.

Model loading also changes. Batch jobs can load a model once per worker and amortize startup over millions of rows. Online services need controlled warmup, health checks, and enough replicas to avoid cold paths. Streaming jobs must load the same version across tasks or record which version scored each event. Edge deployments must balance model size, quantization, hardware acceleration, and device runtime compatibility.

API and Configuration Anatomy

A serving definition should name the mode, model version, input schema, output schema, feature source, timeout or schedule, retry behavior, idempotency key, and destination. For batch, the idempotency key is often a dataset partition plus model version. For online, it is commonly a request id. For streaming, it is the topic, partition, offset, and model version. For edge, it is the device id, model package version, and local input timestamp.

from dataclasses import dataclass
from typing import Literal

Mode = Literal["batch", "online", "streaming", "edge"]

@dataclass(frozen=True)
class ServingPlan:
    mode: Mode
    model_version: str
    input_schema: str
    max_latency_ms: int | None
    output_destination: str

def describe_plan(plan: ServingPlan) -> str:
    latency = "scheduled" if plan.max_latency_ms is None else f"{plan.max_latency_ms} ms"
    return f"{plan.mode}:{plan.model_version}:{latency}:{plan.output_destination}"

plan = ServingPlan("online", "risk-model-42", "risk-request-v3", 80, "https-response")
print(describe_plan(plan))

This configuration shows the anatomy: mode, version, schema, latency budget, and destination. Its deterministic output is online:risk-model-42:80 ms:https-response, which ties a prediction path to a model and serving contract.

Worked Example 1: Batch Churn Scores

A retention team wants a daily table of customers ranked by churn risk. The system does not need a response while an agent waits; it needs complete, repeatable output by 7 a.m. Batch inference is the natural shape. The job reads yesterday’s customer feature partition, scores every active customer, writes a new partition, then records row count and model version.

customers = [
    {"customer_id": "c1", "days_since_login": 3, "tickets": 0},
    {"customer_id": "c2", "days_since_login": 40, "tickets": 3},
]

def churn_score(row: dict[str, int | str]) -> float:
    score = 0.10
    score += min(int(row["days_since_login"]) / 100, 0.50)
    score += min(int(row["tickets"]) * 0.08, 0.40)
    return round(score, 2)

output = [(row["customer_id"], churn_score(row)) for row in customers]
print(output)

The expected output is [('c1', 0.13), ('c2', 0.74)]. The important behavior is repeatability: re-running the same model against the same input partition should produce the same customer ids and scores. If a retry appends instead of overwriting or merging by key, the downstream tool may see duplicates.

Worked Example 2: Online Risk API

A checkout service needs a risk decision before authorizing a payment. The user is waiting, so a nightly table is too stale. Online inference keeps the model ready and imposes a strict timeout. The service should return a bounded result, not an unhandled stack trace, when the request is malformed.

def online_risk(amount: float, account_age_days: int) -> dict[str, object]:
    if amount <= 0:
        raise ValueError("amount must be positive")
    risk = 0.02 + min(amount / 1000, 0.60)
    if account_age_days < 30:
        risk += 0.20
    decision = "review" if risk >= 0.50 else "approve"
    return {"risk": round(risk, 2), "decision": decision}

print(online_risk(75.0, 400))
print(online_risk(900.0, 10))

The expected responses are {'risk': 0.1, 'decision': 'approve'} and {'risk': 0.82, 'decision': 'review'}. The design pressure is latency and graceful refusal. Every feature fetch, model call, and post-processing rule consumes part of the request budget.

Worked Example 3: Streaming and Edge

Streaming and edge are both used when waiting for a scheduled job is unacceptable, but they solve different constraints. A streaming pipeline can score equipment readings and maintain a moving average in managed state. An edge model can make a camera decision locally when the network is slow or policy forbids sending raw frames away from the site.

readings = [18.0, 19.5, 31.0, 33.0]
window: list[float] = []
alerts: list[str] = []

for offset, value in enumerate(readings):
    window.append(value)
    window = window[-3:]
    average = sum(window) / len(window)
    if average >= 28.0:
        alerts.append(f"offset={offset}:avg={average:.1f}")

print(alerts)

The expected output is [] because (19.5 + 31.0 + 33.0) / 3 is 27.8, below the 28.0 threshold. This example is intentionally close to the boundary: threshold definitions and rounding rules must be explicit or operators will chase disagreements between dashboards and alerts.

Design Choices and Trade-Offs

Choose batch when throughput, cost efficiency, and reproducibility matter more than immediate response. It is easier to backfill, audit, and compare model versions, but it can make stale predictions. Choose online when a caller needs an answer now. It improves freshness and interaction quality but introduces replica sizing, tail latency, dependency timeouts, and partial outages. Choose streaming when events must be scored continuously with ordered progress. It handles high event volume and near-real-time state, but offset management, late events, and replay semantics become core design work. Choose edge when latency, privacy, autonomy, or connectivity dominate. It reduces server round trips and can protect raw data, but model updates, device heterogeneity, and local observability are harder.

Hybrid designs are common. A recommender may use batch to precompute candidates, online inference to re-rank them for the current session, streaming inference to update recent behavior features, and edge inference to personalize notifications on a phone. The MLOps task is to make versioning and evaluation coherent across those paths.

Failure Modes and Troubleshooting

Batch symptom: the daily score table has half the normal rows. The likely cause is an upstream partition arriving late or a filter using the wrong date. Diagnose by comparing input row count, partition timestamp, model version, and output row count before publishing. Correct by failing before publish, backfilling the missing partition, and making the completion marker depend on validation.

Online symptom: p95 latency jumps while CPU remains low. The likely cause is a slow feature store, cold model load, DNS issue, or exhausted connection pool. Diagnose with per-stage timing around validation, feature retrieval, inference, and response serialization. Correct by setting timeouts, keeping model warm, caching safe features, right-sizing pools, and returning a controlled fallback when policy allows it.

Streaming symptom: alerts repeat after a restart. The likely cause is committing offsets before output is durably written, or writing without an idempotency key. Diagnose by checking offsets, checkpoint age, and duplicate prediction keys. Correct by writing predictions with topic-partition-offset-model identifiers and committing only after durable output succeeds.

Edge symptom: older devices produce a spike in false rejects. The likely cause is runtime incompatibility, quantization loss, camera preprocessing differences, or a failed staged rollout. Diagnose by grouping metrics by device class, model package, preprocessing version, and hardware accelerator. Correct by pausing rollout, reverting the affected package, and adding device-specific validation before promotion.

Security, Performance, and Reliability

Serving mode changes risk. Online services must authenticate callers and prevent sensitive features from leaking in logs. Batch jobs need scoped warehouse permissions because they often touch large historical tables. Streaming jobs need controlled replay access because a replay can regenerate decisions at scale. Edge packages need signing so devices reject tampered models.

Performance tests should match the mode. Batch tests measure records per worker, shuffle volume, memory pressure, and publish time. Online tests measure p50, p95, p99 latency, saturation, and timeout behavior under concurrent calls. Streaming tests measure lag, checkpoint recovery, and late-event handling. Edge tests measure startup time, battery impact, memory use, thermal throttling, and accuracy after quantization.

Hands-On Lab: Compare Serving Modes

Prerequisites: Python 3.10 or newer, a terminal, and permission to create a temporary directory. No external packages are required.

  1. Create a scratch directory and save the four Python snippets from this lesson as separate files.
  2. Run the serving plan snippet and verify it prints online:risk-model-42:80 ms:https-response.
  3. Run the batch snippet and verify the two customer scores are stable across repeated runs.
  4. Run the online snippet, then change the amount to -1 and verify it raises ValueError: amount must be positive before returning a decision.
  5. Run the streaming snippet, then lower the threshold to 27.0 and verify the final offset emits an alert.
  6. Choose modes for nightly lead scoring, checkout fraud checks, factory temperature alerts, and offline image classification on a phone.

Cleanup: delete the scratch directory. If you adapted the snippets inside a shared repository, remove generated files and do not commit temporary output. Verification is complete when each deterministic output matches the lesson and each changed threshold or invalid input produces the expected behavior.

Assessment Exercises

  1. A model must score 200 million accounts once per week, and analysts need exact backfills for audits. Which serving mode is the default choice, and what completion checks would you require before publishing?
  2. An API calls a feature store and a model server. Latency increased, but model inference time is unchanged. List the per-stage measurements needed to isolate the cause.
  3. A streaming fraud job replays yesterday’s events after a bug fix. How should prediction keys and offset commits be designed to avoid duplicate customer actions?
  4. An edge model is smaller and faster after quantization, but accuracy falls for one device class. What validation split and rollout control would catch this before full deployment?
  5. Design a hybrid serving approach for recommendations that uses at least two modes. Explain what data or model version must be shared between them.

Summary

Batch inference optimizes repeatable scoring of bounded datasets. Online inference optimizes synchronous answers under a latency budget. Streaming inference optimizes continuous scoring with offset-aware recovery. Edge inference optimizes local autonomy under device constraints. The MLOps work is to bind each mode to explicit schemas, model versions, feature freshness rules, telemetry, and rollback paths so predictions can be operated as deliberately as they are trained.