Message Queues Delivery Semantics and Dead-Letter Handling

Published by Programming Line in System Design and Distributed Systems. Last updated .

Purpose and Outcome

Message queues let one part of a system hand work to another part without requiring both to be available at the same instant. An order API can accept a checkout, enqueue a payment task, and return quickly while payment workers process the task. The system-design question is not simply whether to use a queue. It is what delivery promise the queue and consumers actually provide, how duplicates and lost work are prevented, and where poison messages go when they cannot be processed.

By the end of this lesson you should be able to describe at-most-once, at-least-once, and effectively-once processing; explain acknowledgements, offsets, visibility timeouts, retry policies, and dead-letter queues; choose a retry and DLQ strategy for a workload; and troubleshoot duplicate processing, stuck messages, and retry storms.

How Queue Delivery Works Internally

A queue stores messages durably or in memory, presents them to consumers, and tracks whether each message is still pending, leased to a consumer, acknowledged, or failed. Different queue products use different names, but the internal state machine is similar. A producer publishes a message with a body and metadata. The broker persists it, assigns an identifier or offset, and eventually delivers it to a consumer. The consumer performs side effects, then tells the broker whether the message can be removed or marked complete.

The critical detail is that message processing spans two systems: the broker and the consumer’s database or downstream API. If the consumer writes to its database and crashes before acknowledging the message, the broker will eventually deliver the same message again. If the consumer acknowledges first and then crashes before writing, the work is lost. This gap is why most queues provide at-least-once delivery by default and require consumers to be idempotent.

At-most-once delivery means a message is delivered zero or one time. It is fast and simple because the broker can consider a message handled as soon as it is sent or fetched. The trade-off is loss during crashes or network failures. At-least-once delivery means a message should not be lost if the broker and storage are healthy, but duplicates are possible. Effectively-once processing is an application-level result: duplicates may be delivered, but deduplication keys, idempotent writes, transactions, or offset coordination make repeated delivery produce the same final state.

Pull-based queues often use a visibility timeout. When a consumer receives a message, the broker hides it from other consumers for a lease period. If the consumer acknowledges before the lease expires, the broker deletes or completes it. If the lease expires first, the broker makes the message visible again. Log-based systems such as Kafka expose offsets instead: consumers read records in order from partitions and commit the offset that marks progress. A retry is often just rereading an uncommitted offset, although applications may also publish failed records to retry topics.

API and Configuration Anatomy

Queue behavior is shaped by a small set of controls. The producer decides the message body, message key, correlation ID, deduplication ID, delay, and sometimes partition or routing key. The broker decides retention, partitioning, ordering scope, maximum receive count, visibility timeout, and dead-letter routing. The consumer decides concurrency, batch size, ack timing, retry classification, idempotency checks, and whether to extend a lease for long-running work.

A useful consumer contract contains a stable business identifier. For example, order_id is better than a random message ID when charging a customer because it lets the payment worker detect that order O-1024 has already been charged. Broker-assigned IDs are still useful for tracing, but business idempotency keys protect the side effect itself.

from dataclasses import dataclass
from typing import Literal

@dataclass(frozen=True)
class QueueMessage:
    message_id: str
    order_id: str
    attempt: int
    body: dict

Delivery = Literal["ack_before_work", "ack_after_work"]


def process_payment(message: QueueMessage, ack_mode: Delivery) -> str:
    if ack_mode == "ack_before_work":
        return f"{message.order_id}: loss possible if worker crashes after ack"
    return f"{message.order_id}: duplicate possible if worker crashes before ack"

print(process_payment(QueueMessage("m1", "O-1024", 1, {"amount": 25}), "ack_after_work"))

The deterministic output is O-1024: duplicate possible if worker crashes before ack. That sentence is the central design trade-off in miniature. Most payment, inventory, and notification systems choose ack-after-work plus idempotency because losing committed work is worse than safely ignoring a duplicate.

Worked Example 1: At-Most-Once Telemetry

Consider a fleet of mobile clients sending click events. Losing a small fraction of events is acceptable because analytics aggregates millions of samples, but slowing the app to retry every event would hurt users. The producer can publish events without a synchronous confirmation, and a consumer can acknowledge batches before inserting them into analytical storage. If a worker crashes after acknowledging a batch, that batch disappears. The expected behavior is lower latency and possible gaps.

events = ["click:home", "click:buy", "click:help"]
acked = list(events)
inserted = events[:1]
missing = [event for event in acked if event not in inserted]
print(missing)

The output is ['click:buy', 'click:help']. This example is not a bug if the requirement explicitly tolerates sampling loss. It would be unacceptable for order creation because acknowledged-but-unwritten messages cannot be recovered from the queue.

Worked Example 2: At-Least-Once Orders

An order service publishes OrderPlaced. The fulfillment consumer reserves inventory and acknowledges only after the reservation commit succeeds. If it commits and then crashes before ack, the message returns after the visibility timeout. The expected behavior is a second delivery of the same order. Correctness depends on the reservation table having a uniqueness constraint on order_id.

reservations = set()

def reserve(order_id: str) -> str:
    if order_id in reservations:
        return "duplicate ignored"
    reservations.add(order_id)
    return "reserved"

print(reserve("O-2001"))
print(reserve("O-2001"))
print(sorted(reservations))

The output is reserved, then duplicate ignored, then ['O-2001']. The queue still delivered twice, but the application produced one inventory reservation. In a design review, call this effectively-once processing for the reservation effect, not exactly-once delivery.

Worked Example 3: Retries and a Dead-Letter Queue

A message may fail because of a transient dependency outage, a temporary rate limit, or a permanent schema problem. Retrying all failures forever blocks capacity and hides bad data. A common policy is: retry transient failures with backoff; stop after a maximum receive count; move the message plus error metadata to a dead-letter queue; alert or create an operational ticket; and replay only after the underlying cause is fixed.

def route_message(error: str, attempt: int, max_attempts: int = 3) -> str:
    permanent = {"missing_required_field", "invalid_currency"}
    if error in permanent:
        return "dead-letter"
    if attempt >= max_attempts:
        return "dead-letter"
    return f"retry-after-{2 ** attempt}s"

for error, attempt in [("timeout", 1), ("timeout", 3), ("invalid_currency", 1)]:
    print(route_message(error, attempt))

The output is retry-after-2s, dead-letter, and dead-letter. The first timeout gets another chance. The third timeout has exhausted the budget. The invalid currency message is not retried because another attempt will not repair the payload.

Design Choices and Trade-Offs

Choose the delivery semantic from the business consequence. Fire-and-forget telemetry can use at-most-once. Financial movement, entitlement grants, and inventory changes should use at-least-once plus idempotent effects. Workflows that need a single ordered history, such as account ledger events, usually need partitioning by account or aggregate ID so related messages are consumed in order.

Set visibility timeouts from real processing time, not from hope. If normal processing takes 45 seconds and the visibility timeout is 30 seconds, concurrent consumers will work the same message. If the timeout is 30 minutes, crashed work takes too long to retry. Long tasks should heartbeat or extend the lease while they make progress. Batch consumers must consider partial success: acknowledging a whole batch after one item fails can lose the failed item, while rejecting the whole batch can duplicate the successful items.

Dead-letter queues are not trash cans. They are quarantine areas for messages that need inspection, repair, replay, or deletion. A useful DLQ record keeps the original payload, original topic or queue, first failure time, last failure time, attempt count, consumer version, exception class, and correlation ID. Without that context, operators cannot distinguish a bad deploy from a malformed customer record.

Failure Modes and Troubleshooting

Duplicate side effects. Symptoms include customers charged twice, duplicate emails, or inventory decremented twice. The common cause is at-least-once redelivery after the consumer committed a side effect but did not acknowledge in time. Diagnose by comparing broker delivery count, consumer crash logs, and records with the same business key. Correct it with idempotency keys, uniqueness constraints, conditional writes, and ack-after-commit ordering.

Messages stuck in flight. Symptoms include queue depth staying high while visible message count is low, or work reappearing only after a long delay. The cause is often an excessive visibility timeout, a worker that hangs without failing, or a missing ack path. Check in-flight counts, worker thread dumps, lease-extension logs, and timeout configuration. Correct it by bounding handler runtime, adding cancellation, shortening the lease, and extending leases only for actively progressing work.

Retry storm. Symptoms include dependency saturation, rapidly rising attempts, and old messages starving new ones. The cause is usually immediate retry on transient failures. Diagnose by grouping failures by exception and dependency, then compare retry rate to downstream capacity. Correct it with exponential backoff, jitter, circuit breakers, separate delayed retry queues, and a maximum receive count that routes exhausted work to a DLQ.

DLQ grows silently. Symptoms include successful main-queue metrics while customer actions never complete. The cause is missing alerts on dead-letter depth or an overly broad permanent-failure classifier. Diagnose by sampling DLQ payloads and error metadata. Correct it by alerting on DLQ age and count, preserving replay tooling, and reviewing classifier logic during schema changes.

Security, Performance, and Reliability Implications

Messages often contain sensitive business data because queues sit between services. Encrypt queues at rest where available, restrict producers and consumers to the narrow queues they need, and avoid putting secrets in payloads. If a DLQ has broader operator access than the primary service database, it may become the easiest place to leak personal data. Apply retention limits and redaction consistently.

Performance tuning is mostly about controlled parallelism. More consumers increase throughput until they hit partition limits, database locks, API rate limits, or ordering constraints. Prefetch and large batches reduce broker round trips but increase duplicate work after a crash. Small batches improve fairness and reduce replay size but cost more per message. Reliability comes from making every side effect repeatable or compensatable, then proving that retry and replay paths use the same protections as the normal path.

Hands-On Lab: Build a Local Retry and DLQ Simulator

Prerequisites: Python 3, a terminal, and permission to create a temporary file. The lab simulates a broker by keeping three lists: ready messages, delayed retries, and the dead-letter queue. It demonstrates at-least-once delivery, idempotent success, transient retry, and permanent failure routing.

  1. Create a temporary file named queue_lab.py.
  2. Paste the simulator code below.
  3. Run python queue_lab.py.
  4. Verify that order O-1 is processed once even though it appears twice, order O-2 retries before succeeding, and order O-3 goes to the DLQ.
  5. Cleanup by deleting queue_lab.py.
processed = set()
dlq = []

messages = [
    {"order_id": "O-1", "error": None, "attempt": 1},
    {"order_id": "O-1", "error": None, "attempt": 2},
    {"order_id": "O-2", "error": "timeout", "attempt": 1},
    {"order_id": "O-3", "error": "bad_schema", "attempt": 1},
]

def handle(message):
    order_id = message["order_id"]
    if message["error"] == "bad_schema":
        dlq.append({**message, "reason": "permanent schema failure"})
        return f"{order_id}: dead-lettered"
    if message["error"] == "timeout" and message["attempt"] == 1:
        message["attempt"] += 1
        message["error"] = None
        messages.append(message)
        return f"{order_id}: retry scheduled"
    if order_id in processed:
        return f"{order_id}: duplicate ignored"
    processed.add(order_id)
    return f"{order_id}: processed"

while messages:
    print(handle(messages.pop(0)))

print("processed", sorted(processed))
print("dlq", [item["order_id"] for item in dlq])

Expected output is deterministic: O-1: processed, O-1: duplicate ignored, O-2: retry scheduled, O-3: dead-lettered, O-2: processed, followed by processed ['O-1', 'O-2'] and dlq ['O-3']. If O-1 appears twice in the processed set, the idempotency check is missing or placed after a non-repeatable side effect. If O-3 keeps retrying, the permanent-failure branch is wrong. If O-2 never succeeds, the retry path is not clearing the transient error.

Assessment Exercises

  1. An email worker sends a message and crashes before acknowledging. What duplicate behavior should you expect, and where would you put the idempotency key?
  2. A payment consumer has a 20-second visibility timeout, but p95 processing time is 35 seconds. Explain the symptom and two corrections.
  3. Design a DLQ record format for a failed InvoiceGenerated event. Which fields are required to replay it safely?
  4. A team claims their queue gives exactly-once delivery. What questions would you ask to distinguish broker delivery from application-level exactly-once effects?
  5. Given a permanent validation error and a transient HTTP 503, describe different retry paths and why treating them the same is harmful.

Summary

Delivery semantics are the contract between broker state, consumer acknowledgements, and application side effects. At-most-once favors low latency and accepts loss. At-least-once protects work but requires idempotent consumers because duplicates are normal under crashes and timeouts. Effectively-once behavior is built with business keys, conditional writes, offset or ack discipline, and careful replay design. Dead-letter queues complete the reliability loop by isolating messages that should not consume retry capacity until someone inspects, fixes, and deliberately replays or discards them.