Redis Streams Explained

A Redis Stream is an append-only log data type: each write adds a new, uniquely identified entry to the end of the stream, and existing entries are never modified in place. Streams are Redis’s answer to “model a sequence of events” — sensor readings, order events, activity feeds, or a lightweight message queue — with built-in support for many independent readers or a coordinated, load-balanced group of workers. Unlike a List, a Stream keeps a permanent, strictly ordered record of everything appended to it (until you explicitly trim it), and unlike Pub/Sub, that record persists so late-joining or reconnecting readers don’t miss anything.

Overview / How it works

Every entry appended to a stream is a small ordered map of field-value pairs, tagged with an ID of the form <milliseconds-time>-<sequence>, for example 1699999999999-0. When you call XADD key * field value ..., Redis generates this ID for you: the millisecond part comes from the server clock, and the sequence part increments whenever more than one entry lands in the same millisecond, so IDs are always strictly increasing. You can also assign an explicit ID yourself, but Redis rejects any ID that is not strictly greater than the stream’s current last ID — this monotonic ordering is what makes range queries and “give me everything since X” reads possible at all.

Internal storage

Internally, a stream is stored as a radix tree (Redis calls it a “rax”) that indexes compact, contiguous chunks of entries called listpacks. This structure is what makes ID-range lookups fast even on streams with millions of entries, and it’s far more memory-efficient than storing the same events as individual list items or separate hash keys. Because Redis is single-threaded, every command that touches a stream — appending, trimming, or delivering to a consumer group — runs to completion without interleaving with any other client command, so you never observe a half-written entry or a consumer group whose bookkeeping is out of sync with the data.

Two read patterns

Streams support two very different ways of reading. Plain reads with XRANGE or XREAD let any number of independent clients read the same entries at their own pace — nothing is consumed or removed, and each client tracks its own position. Consumer groups add a second layer: a group has a cursor (the last-delivered ID) and, per consumer, a Pending Entries List (PEL) recording which IDs were handed out but not yet acknowledged. This turns a stream into a real work queue, where each entry goes to exactly one consumer within the group, and entries a consumer never acknowledges can be detected and reclaimed if that consumer crashes.

Syntax

The core write command:

XADD key [NOMKSTREAM] [MAXLEN | MINID [= | ~] threshold] <id | *> field value [field value ...]

• key — the stream’s key name, conventionally namespaced like orders:stream.
• NOMKSTREAM — optional; if given, XADD fails silently (returns nil) instead of auto-creating the stream when the key doesn’t exist.
• MAXLEN / MINID — optional trimming applied on every write, capping the stream by entry count or by a minimum ID (effectively an age cutoff). The ~ modifier means “approximately”, letting Redis trim in efficient whole-node chunks instead of exactly.
• id | * — either the literal * to auto-generate an ID, or an explicit ID greater than the stream’s current last ID.
• field value ... — one or more field/value pairs making up the entry; at least one pair is required.

Command reference

Command	Purpose	Time Complexity
XADD	Append an entry	O(1) normally; O(N) if MAXLEN/MINID trims N entries
XLEN	Number of entries in a stream	O(1)
XRANGE / XREVRANGE	Read entries by ID range	O(N) for N entries returned, with fast radix-tree seeking
XREAD	Read entries from one or more streams	O(N) per stream, for N entries returned
XDEL	Delete specific entries by ID	O(1) per ID
XTRIM	Cap a stream’s size or age	O(N) for N entries removed
XGROUP CREATE	Create a consumer group	O(1)
XREADGROUP	Read as part of a consumer group	O(N) per stream, for N entries returned
XACK	Acknowledge processed entries	O(1) per ID
XPENDING	Inspect unacknowledged entries	O(1) summary form; O(M) extended form for M entries returned
XCLAIM / XAUTOCLAIM	Reassign stuck pending entries	O(log N) per entry claimed

Examples

Example 1: Appending and reading a range

XADD orders:stream * customer "alice" amount "49.99"
XADD orders:stream * customer "bob" amount "12.50"
XLEN orders:stream
XRANGE orders:stream - +

Output:

"1699999999999-0"
"1699999999999-1"
(integer) 2
1) 1) "1699999999999-0"
   2) 1) "customer"
      2) "alice"
      3) "amount"
      4) "49.99"
2) 1) "1699999999999-1"
   2) 1) "customer"
      2) "bob"
      3) "amount"
      4) "12.50"

Each XADD returns the ID it assigned — your actual IDs will differ since they’re based on the current server time, but the sequence portion (the number after the dash) increments when two adds land in the same millisecond, exactly as shown here. XLEN is an O(1) counter Redis maintains alongside the stream, and XRANGE key - + reads every entry from the lowest possible ID (-) to the highest (+), returning each entry as its ID plus its flattened field/value list.

Example 2: Reading with XREAD

XADD sensors:stream * device "temp-1" reading "21.5"
XADD sensors:stream * device "temp-1" reading "21.7"
XREAD COUNT 10 STREAMS sensors:stream 0

Output:

1) 1) "sensors:stream"
   2) 1) 1) "1699999999999-0"
         2) 1) "device"
            2) "temp-1"
            3) "reading"
            4) "21.5"
      2) 1) "1699999999999-1"
         2) 1) "device"
            2) "temp-1"
            3) "reading"
            4) "21.7"

XREAD ... STREAMS sensors:stream 0 asks for every entry with an ID greater than 0, i.e. everything. In real applications you’d normally pass the last ID you’ve already processed instead of 0, so you only get new entries — XREAD itself does not remember your position between calls, which is exactly the gap consumer groups fill.

Example 3: Consumer groups

XADD notifications:stream 1-1 type "email" to "user:42"
XADD notifications:stream 2-1 type "sms" to "user:77"
XGROUP CREATE notifications:stream workers 0
XREADGROUP GROUP workers worker-1 COUNT 10 STREAMS notifications:stream >
XPENDING notifications:stream workers
XACK notifications:stream workers 1-1
XPENDING notifications:stream workers

Output:

"1-1"
"2-1"
OK
1) 1) "notifications:stream"
   2) 1) 1) "1-1"
         2) 1) "type"
            2) "email"
            3) "to"
            4) "user:42"
      2) 1) "2-1"
         2) 1) "type"
            2) "sms"
            3) "to"
            4) "user:77"
1) (integer) 2
2) "1-1"
3) "2-1"
4) 1) 1) "worker-1"
      2) "2"
(integer) 1
1) (integer) 1
2) "2-1"
3) "2-1"
4) 1) 1) "worker-1"
      2) "1"

Here explicit IDs (1-1, 2-1) are used instead of * so the example is deterministic. XGROUP CREATE notifications:stream workers 0 creates a group named workers starting from ID 0, meaning it hasn’t delivered anything yet. XREADGROUP ... STREAMS notifications:stream > — the special ID > means “only entries never delivered to any consumer in this group” — delivers both entries to worker-1 and records them in its PEL. The first XPENDING summary shows 2 pending entries owned by worker-1. After XACK ... 1-1 removes that ID from the PEL, the second XPENDING shows only 1 pending entry left.

How it works step by step

When you run XADD, Redis: (1) validates that the target ID (or the auto-generated one) is strictly greater than the stream’s last ID; (2) appends the entry into the tail listpack node of the underlying radix tree, starting a new node if the current one is full; (3) updates the stream’s length and last-ID metadata, both O(1) operations; and (4) if MAXLEN or MINID was given, evicts entries from the front of the stream until the constraint is satisfied. Consumer groups are not notified automatically — an entry just sits in the stream until some consumer calls XREADGROUP with >, at which point Redis advances that group’s last-delivered-id and adds the entry’s ID to the requesting consumer’s PEL. XACK simply removes an ID from the PEL; nothing about the stream entry itself changes.

Trimming in practice:

XADD metrics:stream * cpu "42"
XADD metrics:stream * cpu "55"
XADD metrics:stream * cpu "61"
XTRIM metrics:stream MAXLEN 2
XLEN metrics:stream

Output:

"1699999999999-0"
"1699999999999-1"
"1699999999999-2"
(integer) 1
(integer) 2

XTRIM metrics:stream MAXLEN 2 removes the single oldest entry to bring the stream down to exactly 2 entries, and returns the count of entries it removed. Because streams never shrink on their own, some form of trimming (or explicit XDEL) is the only thing standing between an active stream and unbounded memory growth.

Common Mistakes

Mistake 1: relying on plain XREAD to track your position. Calling XREAD with a fixed starting ID like 0 every time re-delivers the same entries over and over, because XREAD has no memory of what a particular client already consumed:

XADD tasks:stream * job "resize-image"
XREAD COUNT 10 STREAMS tasks:stream 0
XREAD COUNT 10 STREAMS tasks:stream 0

Output:

"1699999999999-0"
1) 1) "tasks:stream"
   2) 1) 1) "1699999999999-0"
         2) 1) "job"
            2) "resize-image"
1) 1) "tasks:stream"
   2) 1) 1) "1699999999999-0"
         2) 1) "job"
            2) "resize-image"

Both reads return the identical entry — nothing was consumed. The fix is either to remember the last ID you saw and pass that instead of 0 on the next call, or, far more robustly, to use a consumer group (XGROUP CREATE + XREADGROUP ... >) so Redis tracks delivery for you and guarantees each entry is only handed out once per group.

Mistake 2: writing to a stream key that already holds a different type. Every Redis key has exactly one type, and mixing them produces a WRONGTYPE error rather than silently working:

SET session:abc123 "active"
XADD session:abc123 * field "value"

Output:

OK
(error) WRONGTYPE Operation against a key holding the wrong kind of value

The fix is simply to use a distinct key name for the stream (e.g. session:abc123:events) rather than reusing a key that already stores a plain string.

Mistake 3: never acknowledging entries. In Example 3, if worker-1 had crashed before calling XACK, both entries would stay in its PEL forever, invisible to new XREADGROUP > calls (since they were already delivered) yet never processed. Always call XACK once an entry is durably handled, and use XPENDING plus XCLAIM/XAUTOCLAIM to recover entries stuck on a dead consumer instead of letting them accumulate silently.

Mistake 4: scanning for stream keys with KEYS. KEYS pattern is O(N) over the entire keyspace and blocks Redis’s single thread for the whole scan — fine on a toy dataset, dangerous on a production instance with millions of keys. Use SCAN with a MATCH pattern instead; it’s cursor-based and returns results incrementally without blocking other clients.

Best Practices

• Cap growth with MAXLEN ~ threshold on every XADD, or run periodic XTRIM — streams never shrink automatically, so an unbounded stream is an unbounded memory leak.
• Prefer consumer groups over manual ID tracking whenever more than one worker needs to share the load or you need at-least-once delivery guarantees.
• Give consumers stable, meaningful names (e.g. host:pid) so pending entries can be correctly identified and reclaimed after a restart.
• Always XACK after successfully processing an entry; check XPENDING periodically to catch entries that were delivered but never acknowledged.
• Use XCLAIM or XAUTOCLAIM with a sensible minimum idle time to recover work from crashed or stalled consumers rather than letting it sit forever.
• Use SCAN MATCH, never KEYS, to discover stream key names in a production environment.
• Inspect stream and group health with XINFO STREAM key and XINFO GROUPS key instead of guessing at internal state.

Practice Exercises

• Create a stream user:1001:activity and append three entries representing login, purchase, and logout events with relevant fields. Use XRANGE with explicit start/end IDs to fetch only the middle (purchase) entry.
• Create a stream jobs:stream, create a consumer group called processors, and have two differently named consumers each call XREADGROUP to split up a batch of entries. Use XPENDING to confirm how many entries each consumer currently holds unacknowledged.
• Using the group from the previous exercise, acknowledge only some of the delivered entries with XACK, then use XPENDING to identify the remaining ones and XCLAIM them onto a different consumer name, verifying the entry’s new owner.

Summary

• A stream is an append-only log of ID-ordered entries, each ID formatted as milliseconds-sequence and guaranteed strictly increasing.
• XADD appends and auto-generates IDs with *; XRANGE/XREVRANGE read by ID range; XREAD reads new entries without any built-in position tracking.
• Consumer groups (XGROUP CREATE, XREADGROUP ... >) turn a stream into a work queue, delivering each entry to exactly one consumer and tracking pending entries in a per-consumer PEL.
• XACK clears a delivered entry from the PEL; unacknowledged entries can be inspected with XPENDING and reassigned with XCLAIM/XAUTOCLAIM.
• Streams never shrink on their own — use MAXLEN/MINID on XADD or periodic XTRIM to bound memory use.
• Redis’s single-threaded execution model makes every stream command atomic, so append, trim, and delivery bookkeeping never interleave unsafely.