Pub/Sub Limitations and When to Use Streams Instead

Redis Pub/Sub lets one client run PUBLISH on a named channel and have it instantly delivered to every client currently subscribed to that channel – there is no queue, no storage, and no history. It is extremely fast and simple, which makes it a natural fit for ephemeral broadcasts like live chat notifications or cache-invalidation pings. But that same simplicity means Pub/Sub silently drops a message the instant nobody is listening, gives you no acknowledgment that a subscriber actually processed it, and cannot replay anything after the fact. This lesson covers exactly where Pub/Sub breaks down in production, and shows how Redis Streams – a separate, persistent data type – solves the same publish/consume problem with durability, replay, and consumer groups.

How Redis Pub/Sub Works

Pub/Sub is built directly into the Redis server as an in-memory routing table, not as a data type stored in the keyspace. When a client runs SUBSCRIBE notifications:user:1001, Redis adds that client’s connection to a list associated with the channel name notifications:user:1001 in an internal dictionary. When another client runs PUBLISH notifications:user:1001 "Your order has shipped", Redis looks up that dictionary entry and, for every currently subscribed connection, writes the message directly onto that client’s output buffer. PUBLISH returns the number of clients the message was delivered to.

Because Redis processes one command at a time on a single thread, this fan-out is atomic with respect to every other command: no other command can run in the middle of a PUBLISH, so every subscriber observes the message at a consistent point relative to the rest of the command stream. But atomic delivery is not the same as durable delivery. The message is never written to a key, never touches the RDB snapshot or the AOF log, and is never buffered anywhere beyond the receiving client’s own output socket. If zero clients are subscribed at the moment PUBLISH runs, the message is gone forever – PUBLISH still returns (integer) 0 and reports no error, because from Redis’s point of view nothing went wrong: it successfully delivered the message to zero recipients.

PSUBSCRIBE extends this with glob-style pattern matching (news.* matches news.sports and news.weather), so a client can listen to a family of channels without knowing their exact names in advance. Every PUBLISH now has to check both the exact-channel subscriber list and every registered pattern, which is why its time complexity is O(N+M): N direct subscribers plus M registered patterns.

The Limitations of Pub/Sub

These properties are fine for some workloads and disqualifying for others. The concrete failure modes are:

  • No persistence or replay. A message exists only for the instant it takes to fan out to currently-connected subscribers. A client that connects one second late has permanently missed it, and there is no command that can retrieve it afterward.
  • At-most-once delivery, no acknowledgment. A subscriber cannot tell Redis “I received message X, don’t consider it lost” because Redis never considered it “in flight” to begin with. If the subscriber crashes mid-processing, Redis has no idea and does nothing differently.
  • No real consumer groups. Subscribing to the same channel from three worker processes does not spread the work across them – all three receive every message. Pub/Sub can broadcast to many listeners, but it cannot load-balance a stream of work across a pool of workers the way a task queue needs to.
  • Slow subscribers get disconnected, silently losing messages. Redis enforces a client-output-buffer-limit pubsub cap; if a subscriber can’t keep up and its output buffer exceeds that limit, Redis forcibly closes the connection to protect server memory. Every message that was queued for that client, and every message published afterward before it reconnects, is lost.
  • Invisible to normal keyspace tooling. Channels are not keys. They don’t show up in SCAN, don’t have a TYPE, and can’t be inspected with TTL or OBJECT ENCODING. The only introspection is the dedicated PUBSUB subcommands.
  • No backpressure to the publisher. PUBLISH “succeeds” (returns a receiver count) regardless of whether those receivers are keeping up, so a fast publisher can overwhelm slow subscribers with no signal until they’re disconnected.

Syntax

Pub/Sub commands:

PUBLISH channel message
SUBSCRIBE channel [channel ...]
PSUBSCRIBE pattern [pattern ...]
UNSUBSCRIBE [channel ...]
PUNSUBSCRIBE [pattern ...]
PUBSUB CHANNELS [pattern]
PUBSUB NUMSUB [channel ...]
PUBSUB NUMPAT
  • channel / pattern – the destination name; a pattern uses glob syntax (*, ?, character classes).
  • message – an arbitrary string payload; Redis does not interpret it.
  • PUBSUB CHANNELS lists channels that currently have at least one subscriber, optionally filtered by a glob pattern.
  • PUBSUB NUMSUB returns the subscriber count for each named channel.
  • PUBSUB NUMPAT returns the number of patterns currently registered via PSUBSCRIBE.

The Streams commands that address these limitations:

XADD key [NOMKSTREAM] [MAXLEN|MINID [=|~] threshold] (* or explicit-id) field value [field value ...]
XLEN key
XRANGE key start end [COUNT count]
XREAD [COUNT count] STREAMS key [key ...] id [id ...]
XGROUP CREATE key groupname (id or $) [MKSTREAM]
XREADGROUP GROUP groupname consumer [COUNT count] STREAMS key [key ...] id [id ...]
XACK key groupname id [id ...]
XPENDING key groupname
  • key – the stream’s key name.
  • * in XADD asks Redis to auto-generate a strictly increasing ID (a millisecond timestamp plus a sequence number, e.g. 1723300000000-0); you can also supply your own ID explicitly.
  • MAXLEN/MINID caps the stream’s size at insert time, trimming old entries.
  • start/end in XRANGE accept - and + to mean “the lowest possible ID” and “the highest possible ID”.
  • XGROUP CREATE ... $ starts the group’s cursor at the end of the stream (only new entries); 0 starts it at the beginning (replay everything).
  • > as the ID in XREADGROUP means “give me entries never delivered to any consumer in this group before”.
  • XACK removes an entry from the group’s pending entries list (PEL) once a consumer has finished processing it.

Examples

Example 1: A published message with no subscribers is lost

PUBLISH notifications:user:1001 "Your order has shipped"
PUBSUB CHANNELS
PUBSUB NUMSUB notifications:user:1001

Output:

(integer) 0
(empty array)
1) "notifications:user:1001"
2) (integer) 0

PUBLISH returns 0 because no client was subscribed at that instant – the message was not queued anywhere, it simply ceased to exist. PUBSUB CHANNELS returns an empty array because a channel only appears in that list while at least one client is actively subscribed to it; publishing to a channel does not “create” it in any lasting sense. PUBSUB NUMSUB confirms zero current subscribers for that channel name.

Example 2: A live subscriber receiving a published message

This requires two separate redis-cli connections open at the same time, so it’s shown here for illustration rather than as a single runnable block:

# Terminal A - subscriber (this call blocks, waiting for messages)
SUBSCRIBE notifications:user:1001

# Terminal B - publisher, run in a second redis-cli session
PUBLISH notifications:user:1001 "Your order has shipped"

Output:

# Terminal A prints immediately on subscribing:
1) "subscribe"
2) "notifications:user:1001"
3) (integer) 1

# ...then, the moment Terminal B publishes, Terminal A prints:
1) "message"
2) "notifications:user:1001"
3) "Your order has shipped"

# Terminal B prints:
(integer) 1

Notice the receiver count is now 1 instead of 0, because Terminal A was subscribed when the publish happened. This is the entire Pub/Sub contract: deliver to whoever happens to be listening right now, and nothing more.

Example 3: Streams remember history a late reader can still see

XADD notifications:user:1001 * message "Your order has shipped" type "order_update"
XADD notifications:user:1001 * message "Your order was delivered" type "order_update"
XLEN notifications:user:1001
XRANGE notifications:user:1001 - +

Output:

"1723300000000-0"
"1723300000001-0"
(integer) 2
1) 1) "1723300000000-0"
   2) 1) "message"
      2) "Your order has shipped"
      3) "type"
      4) "order_update"
2) 1) "1723300000001-0"
   2) 1) "message"
      2) "Your order was delivered"
      3) "type"
      4) "order_update"

Both entries are written to the stream and stay there. A consumer that connects long after these writes happened can still run XRANGE notifications:user:1001 - + and see every event from the beginning – the exact capability Pub/Sub is missing. The auto-generated IDs are strictly increasing, so they double as both a unique identifier and a timestamp.

Example 4: Consumer groups track who has acknowledged what

XADD orders:events * order_id "5001" status "created"
XGROUP CREATE orders:events order_processors 0
XREADGROUP GROUP order_processors worker-1 COUNT 1 STREAMS orders:events >
XPENDING orders:events order_processors

Output:

"1723300500000-0"
OK
1) 1) "1723300500000-0"
   2) 1) "order_id"
      2) "5001"
      3) "status"
      4) "created"
(integer) 1
"1723300500000-0"
"1723300500000-0"
1) 1) "worker-1"
   2) "1"

XGROUP CREATE ... 0 creates a consumer group starting from the beginning of the stream. XREADGROUP then delivers the one pending entry to consumer worker-1 and simultaneously adds it to that group’s pending entries list (PEL) – Redis is now tracking that this specific entry was handed out but not yet confirmed. XPENDING confirms: one pending entry, its ID range, and that worker-1 holds it. In a real worker, the next step would be running XACK with the real ID printed above once processing succeeds – that removes it from the PEL. If worker-1 crashes before acknowledging, the entry stays in the PEL and another consumer can claim it with XCLAIM or XAUTOCLAIM. Pub/Sub has no equivalent of a PEL: a crashed subscriber’s in-flight message is simply gone.

How It Works Step by Step

What happens inside Redis for PUBLISH channel message:

  1. Redis looks up channel in its in-memory pubsub dictionary (channel name to list of subscribed client connections).
  2. It also checks the separate list of registered PSUBSCRIBE patterns and tests each one against channel.
  3. For every matching connection, Redis writes a three-element reply ("message", the channel name, the payload) directly onto that client’s output buffer.
  4. It returns the total count of connections the message was written to. Nothing is stored; the dictionary lookup and the writes are the entire operation.

What happens for a Streams write and a group read:

  1. XADD appends an entry to the stream’s underlying structure – a compact radix tree (internally called a “rax”) keyed by ID, with recent entries packed for efficiency. The entry becomes part of the key’s value, subject to the same RDB/AOF persistence as any other key.
  2. If MAXLEN/MINID was given, Redis trims the oldest entries from the other end of the tree until the stream fits the threshold.
  3. XGROUP CREATE attaches a named cursor (the group’s “last delivered ID”) to the stream, stored alongside it, plus an empty PEL for that group.
  4. XREADGROUP ... > advances the group’s cursor past any entries never handed to this group before, copies each into the calling consumer’s slice of the PEL (entry ID, consumer name, delivery time, delivery count), and returns them to the client.
  5. XACK deletes the specified entry IDs from the PEL. The stream entries themselves are untouched – acknowledging only affects delivery bookkeeping, not the data.

Time Complexity Reference

Command Time Complexity Notes
PUBLISH O(N+M) N = subscribers on the channel, M = registered PSUBSCRIBE patterns
SUBSCRIBE / UNSUBSCRIBE O(N) N = number of channels named in the call
PUBSUB CHANNELS O(N) N = number of currently active channels
PUBSUB NUMSUB O(N) N = number of channels requested
XADD O(1) Becomes O(N) when a MAXLEN/MINID trim removes N entries
XLEN O(1) –
XRANGE / XREVRANGE O(log N + M) N = stream length, M = entries returned
XREAD / XREADGROUP O(log N + M) per stream N = stream length, M = entries returned
XACK O(M) M = number of IDs acknowledged
XGROUP CREATE O(1) –
XPENDING (summary form) O(1) Extended form is O(M) for M entries returned

Common Mistakes

Mistake 1: Assuming a published message is buffered for the next subscriber. As Example 1 showed, PUBLISH to a channel with zero current subscribers just discards the message – there’s no queue behind it waiting for someone to connect. If your use case needs “whoever shows up eventually will see this,” you need a persistent structure, not Pub/Sub:

XADD notifications:user:1001 * message "Your order has shipped"

Output:

"1723300800000-0"

A late-arriving reader can XRANGE from the beginning and still see it.

Mistake 2: Calling XREADGROUP against a group that was never created. Unlike Pub/Sub, where any client can just SUBSCRIBE to any channel name, Streams consumer groups must be explicitly created with XGROUP CREATE before anyone can read through them:

XADD logs:app * event "startup"
XREADGROUP GROUP missing_group worker-1 COUNT 1 STREAMS logs:app >

Output:

"1723300900000-0"
(error) NOGROUP No such key 'logs:app' or consumer group 'missing_group' in XREADGROUP with GROUP option

The fix is to create the group first (optionally with MKSTREAM so it also creates the stream if it doesn’t exist yet):

XGROUP CREATE logs:app missing_group 0 MKSTREAM

Output:

OK

Mistake 3: Treating Streams as free persistence and never trimming them. Because Streams solve Pub/Sub’s “no history” problem by keeping every entry, an unbounded stream (a sensor writing every second, for example) grows forever and can quietly consume all available memory. Cap it at write time with MAXLEN:

XADD sensor:temp:001 MAXLEN ~ 1000 * reading "22.5"
XLEN sensor:temp:001

Output:

"1723301000000-0"
(integer) 1

The ~ tells Redis it can trim approximately (rather than exactly) to 1000 entries, which is far cheaper because it avoids repacking the underlying structure on every single write.

Mistake 4: Not knowing a slow subscriber can be silently dropped. Redis limits how much unread data it will buffer for a Pub/Sub client:

CONFIG GET client-output-buffer-limit

Output:

1) "client-output-buffer-limit"
2) "normal 0 0 0 slave 268435456 67108864 60 pubsub 33554432 8388608 60"

The pubsub class here allows a 32 MB hard limit (or 8 MB sustained for 60 seconds) before Redis force-disconnects the client. If your subscriber can’t keep up with publish volume, it gets dropped and misses everything published after that point – with no error delivered to the publisher and no way for the subscriber to know what it missed.

Best Practices

  • Use Pub/Sub only for messages where losing one occasionally is acceptable – live cache-invalidation pings, presence/typing indicators, broadcasting a config-reload signal to all app instances.
  • Use Streams (with consumer groups) whenever you need at-least-once delivery, work distribution across multiple workers, or the ability to replay history.
  • Always cap long-lived streams with MAXLEN ~ (or MINID ~ if you trim by ID/time) at write time so they don’t grow unbounded.
  • Always XACK an entry once it’s genuinely processed, and monitor XPENDING for entries stuck too long – that signals a stuck or crashed consumer.
  • If you must use Pub/Sub for something business-critical, pair it with a durable fallback (write the event to a Stream first, then also PUBLISH as a low-latency notification) rather than relying on Pub/Sub alone.
  • Watch the pubsub class of client-output-buffer-limit in production and alert on subscriber disconnects – a disconnect there is a silent data-loss event, not just a reconnect.
  • Don’t try to build a work queue out of multiple subscribers on one channel – every subscriber gets every message, so you get duplicated work, not load balancing. Reach for Streams consumer groups instead.

Practice Exercises

  • Exercise 1: Using two separate redis-cli sessions, subscribe to a channel named chat:room:42 in one, then publish three different messages to it from the other. Note which messages the subscriber receives, then close the subscriber, publish a fourth message, and reopen the subscription – confirm the fourth message is unrecoverable.
  • Exercise 2: Model the same chat room as a stream called chat:room:42:events instead. Add the same three messages with XADD, then create a consumer group called readers starting from ID 0. Use XREADGROUP to read all three, confirm with XPENDING that they’re all still unacknowledged, then acknowledge them one at a time with XACK and watch the pending count drop.
  • Exercise 3: Create a stream tasks:signup and a consumer group workers. Read one entry as consumer worker-a without acknowledging it. Assume worker-a then crashed – look up how XCLAIM or XAUTOCLAIM would let worker-b take over that pending entry, and write out the command you’d use.

Summary

  • Pub/Sub is an in-memory, fire-and-forget broadcast: PUBLISH delivers only to clients subscribed at that exact instant, and the message is never stored anywhere.
  • A PUBLISH to a channel with no subscribers returns 0 and silently discards the message – this is normal behavior, not an error.
  • Pub/Sub has no replay, no acknowledgment, no consumer groups, and can silently drop a slow subscriber via the client-output-buffer-limit pubsub cap.
  • Redis Streams (XADD, XRANGE, XREAD) persist every entry in the keyspace, so late readers can replay full history.
  • Stream consumer groups (XGROUP CREATE, XREADGROUP, XACK, XPENDING) track exactly which entries were delivered to which consumer and haven’t yet been acknowledged, enabling true work distribution and crash recovery.
  • Choose Pub/Sub for cheap, latency-sensitive, loss-tolerant broadcasts; choose Streams whenever the message itself matters and must not be lost.