Streaming responses let an AI application show useful output before the model has finished the whole answer. Instead of waiting for one large response body, the server forwards small events as they arrive: text deltas, tool-call fragments, refusal or moderation signals, completion metadata, and final usage. The user sees lower perceived latency, and the application can expose stop buttons, progress indicators, and partial transcripts.
In this course section on AI application patterns, streaming is the place where model behavior meets ordinary web backpressure, retries, cancellation, and interface state. The outcome is not simply a typing animation. A good streaming design preserves message order, separates provisional text from final records, detects incomplete structured data, and records token usage only after the stream has closed cleanly.
What Actually Streams
A non-streaming request is simple: send messages, receive one completed object. A streaming request is a sequence. The provider sends an ordered series of events over a long-lived HTTP connection. Common transports include server-sent events from the model provider to your backend, and server-sent events or WebSocket messages from your backend to the browser. Each event has a type and a payload. A text delta appends characters to the visible answer. A tool-call delta appends a fragment to an argument buffer. A final event says generation is complete and may include finish reason, usage, or response identifiers.
The important internal concept is that a stream is stateful. The client cannot interpret every event independently. It must assemble deltas into a stable message, track whether it is currently inside a tool call or JSON object, and know when the final event has committed the answer. Until that point, the visible text is provisional. If the connection fails halfway through, the application should not pretend the partial answer is a complete assistant message.
API Anatomy
A streaming request usually differs from a normal model request by a single option such as stream=true, but the application code changes more than that option suggests. The request still contains model choice, instructions, user content, tools, response format, and limits such as maximum output tokens. The response becomes an iterator of events. The event loop reads each event, updates local state, forwards safe pieces to the user interface, and watches for terminal events.
Useful event categories are content.delta, tool_call.delta, message.completed, usage.final, error, and done. Provider names vary, but the shape is stable: small payloads arrive in order, and the application must join them. For structured output, a delta may contain only {"cit or ations":, which is invalid JSON until later chunks arrive. Parse only when the relevant stream segment is complete, or use an incremental parser that is designed for partial input.
Example 1: Text Delta Assembly
The first example shows the core mechanism without any network dependency. Three content deltas become one visible answer. The deterministic output is the completed sentence, printed once after assembly.
events = [
{"type": "content.delta", "text": "Streaming "},
{"type": "content.delta", "text": "reduces perceived "},
{"type": "content.delta", "text": "latency."},
{"type": "done"},
]
buffer = []
for event in events:
if event["type"] == "content.delta":
buffer.append(event["text"])
elif event["type"] == "done":
break
print("".join(buffer))Expected output: Streaming reduces perceived latency. The buffer matters because real chunks rarely align with words or sentences. A browser can render each delta immediately, but the backend should still retain an assembled copy if it needs to persist the final answer.
Example 2: Event Types and Finish Reasons
The next example distinguishes visible text from terminal metadata. The finish reason says why generation ended. A normal stop can be stored as a complete assistant message. A length stop means the model hit the output budget, so the user interface should mark the answer incomplete or offer continuation.
events = [
{"type": "content.delta", "text": "The report has three sections"},
{"type": "message.completed", "finish_reason": "length"},
]
text = ""
complete = False
for event in events:
if event["type"] == "content.delta":
text += event["text"]
elif event["type"] == "message.completed":
complete = event["finish_reason"] == "stop"
print({"text": text, "complete": complete})Expected output: {'text': 'The report has three sections', 'complete': False}. Treating that answer as complete would be a product bug, not a model bug. The stream told the application exactly what happened.
Example 3: Structured Data Arrives in Pieces
Streaming structured output is useful for progressive interfaces, but only if you separate display from validation. This example receives a JSON object in fragments, waits for the completion event, then parses it. Parsing earlier would raise an error even though the model is behaving correctly.
import json
events = [
{"type": "content.delta", "text": "{\"title\": "},
{"type": "content.delta", "text": "\"Quarterly summary\", "},
{"type": "content.delta", "text": "\"score\": 0.91}"},
{"type": "message.completed", "finish_reason": "stop"},
]
raw = ""
finished = False
for event in events:
if event["type"] == "content.delta":
raw += event["text"]
elif event["type"] == "message.completed":
finished = event["finish_reason"] == "stop"
parsed = json.loads(raw) if finished else None
print(parsed["title"])
print(parsed["score"])Expected output: Quarterly summary and 0.91 on separate lines. In production, validate the parsed object against the schema before storing it or using it to call another system.
Design Choices and Trade-Offs
The first choice is transport. Server-sent events are simple for one-way model output: they work over HTTP, preserve event order, and fit text streams well. WebSockets are better when the browser must send mid-stream control messages, collaborative edits, or bidirectional tool progress. Plain chunked HTTP can work, but explicit event names make clients easier to debug.
The second choice is where to stream from. Streaming directly from provider to browser lowers backend work but exposes credentials and removes policy control, so most applications stream provider to backend, then backend to browser. That middle layer can redact sensitive fragments, enforce tenant limits, translate provider events into stable internal event names, and attach request identifiers for diagnostics.
The third choice is how quickly to render. Rendering every tiny token maximizes immediacy but can cause layout churn and high CPU use in the browser. Batching deltas every few dozen milliseconds often feels just as fast while reducing work. For code blocks and markdown, keep a raw buffer and render through a parser that tolerates incomplete syntax, or render plain text until completion.
Failure Modes and Troubleshooting
Symptom: the answer stops mid-sentence with no error. The cause is often a dropped network connection or a finish reason such as length. Diagnose by checking whether the stream emitted a terminal event and by logging finish reason separately from transport status. Correct it by marking partial answers as incomplete, offering continuation, and increasing the output budget only when the prompt genuinely requires it.
Symptom: JSON parsing fails during streaming. The usual cause is parsing a partial buffer. Inspect the last received event and the accumulated raw string. If the stream has not completed, the invalid JSON may be expected. Correct it by delaying parse until a terminal event, using an incremental parser, or streaming a companion plain-text preview while validating the final object later.
Symptom: the user clicks Stop but tokens keep appearing. The browser may have closed its local reader while the backend request remains active. Diagnose by correlating browser cancellation, backend abort signal, and provider request id. Correct it by propagating cancellation through every layer and by discarding late events for the canceled response id.
Symptom: usage accounting is missing or wrong. Many APIs emit token usage only at the end. If the stream fails, final usage may never arrive. Diagnose by checking whether usage.final was received. Correct it by recording a pending usage state, reconciling from provider logs when available, and avoiding billing decisions based on estimated partial counts.
Security, Performance, and Reliability
Streaming changes when users see text, so moderation and policy decisions need clear placement. If the model may produce sensitive content, do not forward every raw delta blindly. Options include pre-moderating user input, streaming through a backend filter, buffering until sentence boundaries, or using a delayed reveal for high-risk surfaces. The right choice depends on the harm of showing a bad fragment versus the value of immediacy.
Performance depends on time to first token, tokens per second, network buffering, and browser rendering cost. Reverse proxies sometimes buffer responses unless configured for streaming, which makes a correct backend appear non-streaming. Reliability depends on heartbeats, timeouts, idempotent retries before generation begins, and careful no-retry behavior after visible text has been emitted. Retrying a half-finished stream can duplicate or contradict content unless the user interface clearly starts a new attempt.
Hands-On Lab: Build a Local Stream Assembler
Prerequisites: Python 3, a terminal, and no external API key. This lab simulates provider events so you can test stream handling deterministically.
- Create a small script named
stream_lab.py. - Paste the following code.
- Run
python stream_lab.py. - Verify that text appears progressively, the final record is complete, and usage is printed only after the usage event.
- Change
finish_reasonfromstoptolengthand verify that the final record becomes incomplete.
from dataclasses import dataclass
@dataclass
class StreamState:
text: str = ""
complete: bool = False
usage: dict[str, int] | None = None
def apply_event(state: StreamState, event: dict[str, object]) -> None:
event_type = event["type"]
if event_type == "content.delta":
state.text += str(event["text"])
print("preview:", state.text)
elif event_type == "message.completed":
state.complete = event.get("finish_reason") == "stop"
elif event_type == "usage.final":
state.usage = dict(event["usage"])
state = StreamState()
for event in [
{"type": "content.delta", "text": "Fast "},
{"type": "content.delta", "text": "feedback"},
{"type": "message.completed", "finish_reason": "stop"},
{"type": "usage.final", "usage": {"input_tokens": 12, "output_tokens": 2}},
]:
apply_event(state, event)
print("final:", state.text)
print("complete:", state.complete)
print("usage:", state.usage)Expected verification output includes preview: Fast , then preview: Fast feedback, followed by complete: True and usage: {'input_tokens': 12, 'output_tokens': 2}. Cleanup is simply deleting stream_lab.py. If you adapted the pattern inside an application, rollback means disabling streaming at the feature flag and falling back to the existing non-streaming path.
Assessment Exercises
- A stream returns five text deltas and then the connection drops before any terminal event. What should the UI display, and what should be stored in the conversation history?
- Design an event schema for your backend that hides provider-specific event names but still preserves text deltas, tool-call fragments, finish reason, errors, and final usage.
- Explain why parsing JSON on every received chunk can be incorrect even when the model eventually emits valid JSON.
- Choose between server-sent events and WebSockets for a chat screen with a Stop button and background tool progress. Defend the choice and name one operational cost.
- Write a test that proves cancellation prevents late provider events from modifying the current visible answer.
Summary
Streaming responses are ordered event streams, not completed messages delivered slowly. The engineering task is to assemble deltas, preserve terminal metadata, propagate cancellation, handle partial structured data, and account for usage after the stream closes. Done well, streaming makes generative AI interfaces feel faster while keeping the final transcript, billing record, and error state honest.
