Delivery Semantics Idempotence and Exactly-Once Claims

Published by Programming Line in Data Engineering in Depth. Last updated .

Purpose and outcome

Delivery semantics describe what a streaming system promises when machines crash, networks pause, clients retry, and consumers restart. In Kafka-based data engineering, the important question is not whether a record is processed once in perfect conditions. The question is what can happen after a partial failure: can a record be lost, can it be duplicated, and can output and offsets move out of sync?

By the end of this lesson, you should be able to separate at-most-once, at-least-once, idempotent production, and exactly-once processing claims. You will also know where Kafka’s guarantees stop. Kafka can prevent duplicate appends from a retrying producer and can atomically commit consumed offsets with produced records in a transaction. It cannot make an arbitrary email API, payment endpoint, search index, or data warehouse magically exactly-once unless that external sink participates through idempotent writes, transactions, or deterministic replacement.

How the mechanism works

Kafka stores records in ordered partitions. A producer sends records to a partition leader. A consumer reads offsets from partitions and later records progress by committing offsets. Most delivery surprises come from the gap between side effects and offset commits. If a consumer writes to a database and crashes before committing the offset, it will read the same Kafka record again after restart. That is at-least-once behavior: no loss if the write succeeded, but duplicates are possible. If the consumer commits the offset before writing and then crashes, the record may never affect the sink. That is at-most-once behavior: duplicates are avoided by allowing loss.

Kafka producer idempotence addresses a narrower problem. With idempotence enabled, each producer session gets a producer id, and records sent to each partition carry increasing sequence numbers. The broker remembers the last accepted sequence for that producer and partition. If an acknowledgement is lost and the producer retries the same batch, the broker can recognize the repeated sequence and avoid appending a duplicate. This protects Kafka logs from retry duplicates caused by uncertain acknowledgements. It does not deduplicate records that your application intentionally sends twice with different sequence numbers.

Kafka transactions extend this idea across multiple partitions and consumed offsets. A transactional producer uses a stable transactional id. It begins a transaction, produces output records, sends consumer offsets to the transaction, and commits or aborts. Consumers configured to read committed data do not see records from aborted transactions. This lets a consume-transform-produce topology make output topics and input offsets advance atomically: either both are visible, or neither is.

The phrase exactly once is therefore precise only inside a defined boundary. Kafka can provide exactly-once stream processing between Kafka topics when producers are transactional, consumers read committed records, and offsets are committed as part of the transaction. Once the pipeline writes to an external sink, the sink must provide its own protection, such as primary-key upserts, compare-and-swap writes, transactional batches, or a write-ahead table keyed by event id.

API and configuration anatomy

The core terms appear in client configuration and processing code. Producer idempotence is commonly controlled with enable.idempotence. Transactions require a stable transactional.id, initialization of transactional state, and explicit begin, commit, or abort calls. Consumers that must ignore aborted writes use isolation.level=read_committed. Consumer groups track progress through committed offsets. In transactional processing, offsets are sent to the transaction rather than committed independently.

The design anatomy is: input record identity, transformation determinism, output write identity, and offset movement. If any one of these is vague, the claim becomes vague. A robust pipeline can name the input topic, partition, offset, source event id, output key, transaction boundary, and retry behavior. That vocabulary is what lets engineers debug real duplicate or loss incidents.

Worked example 1: at-least-once duplicates at the sink

This example simulates a consumer that writes output and crashes before committing its offset. On restart, it reads the same event again. An append-only sink records the duplicate. An idempotent upsert sink stores one final value under a deterministic key.

events = [{"offset": 10, "order_id": "A-7", "amount": 42}]

append_only_sink = []
upsert_sink = {}
committed_offset = 9

for attempt in ["before crash", "after restart"]:
    event = events[0]
    append_only_sink.append((event["order_id"], event["amount"]))
    upsert_sink[event["order_id"]] = event["amount"]
    if attempt == "after restart":
        committed_offset = event["offset"]

print(append_only_sink)
print(upsert_sink)
print(committed_offset)

The deterministic output is:

[('A-7', 42), ('A-7', 42)]
{'A-7': 42}
10

The consumer did not lose the event, but the append-only table contains two rows. The upsert table is idempotent because repeating the same write with the same key leaves the same final state.

Worked example 2: producer retry deduplication

Now the failure is between producer and broker. The broker appends a batch, but the acknowledgement is lost. The producer retries the same sequence. A broker that tracks sequence numbers for a producer-partition pair can reject the duplicate retry.

accepted = []
last_sequence = {}
producer_id = "producer-3"
partition = 0

batches = [
    {"sequence": 0, "value": "user-created"},
    {"sequence": 0, "value": "user-created"},
    {"sequence": 1, "value": "user-updated"},
]

for batch in batches:
    key = (producer_id, partition)
    if batch["sequence"] > last_sequence.get(key, -1):
        accepted.append(batch["value"])
        last_sequence[key] = batch["sequence"]

print(accepted)

The expected output is:

['user-created', 'user-updated']

This is not general business deduplication. If the application creates two different records for the same user action, Kafka sees two valid sequence numbers and stores both. Producer idempotence solves retry ambiguity, not poor event identity.

Worked example 3: atomic output and offsets

A transactional consume-transform-produce step needs output records and input offsets to commit together. The next simulation shows two attempts. The first produces output but aborts before the offset becomes visible. The second commits both output and offset.

visible_output = []
committed_offsets = {}

attempts = ["abort", "commit"]
for outcome in attempts:
    transaction_output = []
    transaction_offsets = {}
    event = {"topic": "orders", "partition": 0, "offset": 15, "order_id": "A-8"}

    transaction_output.append(("shipments", event["order_id"], "ready"))
    transaction_offsets[(event["topic"], event["partition"])] = event["offset"] + 1

    if outcome == "commit":
        visible_output.extend(transaction_output)
        committed_offsets.update(transaction_offsets)

print(visible_output)
print(committed_offsets)

The expected output is:

[('shipments', 'A-8', 'ready')]
{('orders', 0): 16}

Only the committed attempt becomes visible. The offset is 16 because committed consumer offsets point to the next record to read, not the last processed record.

Design choices and trade-offs

At-most-once is simple and can be acceptable for expendable telemetry, sampled metrics, or low-value UI activity. Its trade-off is permanent loss when a consumer commits progress before completing the side effect. At-least-once is the usual default for important events because retries preserve data, but every sink must tolerate duplicates. That pushes design effort into keys, constraints, merge statements, and deduplication windows.

Kafka transactions are powerful when the pipeline reads Kafka and writes Kafka. They add coordination, state, and operational constraints. A transactional id must be stable for the logical producer instance, but not accidentally shared by two active instances. Transactions that remain open too long delay visibility for read-committed consumers. Very large transactions increase recovery work and can make lag harder to interpret.

For external sinks, idempotent design is often more practical than distributed transactions. A warehouse load can merge by event id. A serving database can use a primary key derived from the source event. A search index can replace a document by deterministic id. A payment API should use an idempotency key supplied by the caller. These approaches change the target from exactly one execution to exactly one final effect for a named operation.

Failure modes and troubleshooting

Symptom: row counts are higher than Kafka input counts after consumer restarts. Cause: the sink uses append-only inserts while offsets are committed after the insert. Diagnostic steps: compare duplicate business keys or source topic-partition-offset columns in the sink; inspect restart timestamps; check whether retries happened before offset commits. Correction: add a deterministic event id, enforce a uniqueness constraint, or write through a staging table and merge.

Symptom: downstream consumers see records from failed processing attempts. Cause: consumers are reading uncommitted transactional data or the producer is not actually using transactions for output and offsets. Diagnostic steps: verify consumer isolation level, check whether aborted transaction markers exist, and inspect whether offsets are committed inside the transaction. Correction: configure read-committed consumption and use the transactional offset commit API in the same transaction as output writes.

Symptom: duplicate Kafka records appear after producer retries. Cause: idempotence is disabled, retries occur from a new producer identity, or the application generated multiple logical events. Diagnostic steps: inspect producer configuration, compare record keys and payload ids, and distinguish broker-level retry duplicates from business duplicates. Correction: enable idempotent production, preserve stable event ids, and make the application create one event per logical operation.

Reliability and performance implications

Reliability improves when every record carries enough identity to make retries harmless: source topic, partition, offset, event id, and business key where appropriate. Performance can decline if every event performs a heavyweight deduplication lookup, so many systems batch merges or keep a bounded deduplication store. Bounded windows are fast but only protect against duplicates that arrive before the window expires. Permanent uniqueness constraints cost storage and write coordination but give stronger protection.

Security matters when idempotency keys cross service boundaries. Keys should not leak sensitive business data if they appear in logs or HTTP headers. Use opaque request ids or hashes when needed, and treat replay protection as part of the authorization story: an attacker should not be able to reuse a valid idempotency key to overwrite someone else’s operation.

Hands-on lab: prove the sink determines the final guarantee

Prerequisites: Python 3 and a shell. No Kafka cluster is required because the lab isolates the offset and sink behavior that causes most delivery-semantic mistakes.

  1. Create a file named delivery_semantics_lab.py.
  2. Paste the following program into it.
  3. Run python delivery_semantics_lab.py.
  4. Verify that the append-only sink has duplicates while the idempotent sink has one row per order.
  5. Change the crash flag to False and rerun to observe the no-retry path.
events = [
    {"offset": 0, "order_id": "A-1", "amount": 30},
    {"offset": 1, "order_id": "A-2", "amount": 19},
]

append_only = []
upserts = {}
committed_offset = 0
crash_after_first_write = True

for event in events:
    append_only.append((event["order_id"], event["amount"]))
    upserts[event["order_id"]] = event["amount"]
    if crash_after_first_write and event["offset"] == 0:
        break
    committed_offset = event["offset"] + 1

for event in events[committed_offset:]:
    append_only.append((event["order_id"], event["amount"]))
    upserts[event["order_id"]] = event["amount"]
    committed_offset = event["offset"] + 1

print("append_only", append_only)
print("upserts", sorted(upserts.items()))
print("committed_offset", committed_offset)

Expected verification output with the crash flag left enabled:

append_only [('A-1', 30), ('A-1', 30), ('A-2', 19)]
upserts [('A-1', 30), ('A-2', 19)]
committed_offset 2

Cleanup is simply deleting the lab file. In a real Kafka environment, the equivalent rollback would also remove temporary topics, consumer groups, and test sink tables so committed offsets from the experiment cannot affect later runs.

Assessment exercises

  1. A consumer writes to a warehouse and then commits its offset. It crashes after the warehouse write but before the offset commit. Describe the next run’s behavior under at-least-once delivery and name two sink designs that prevent duplicate final rows.
  2. A producer sends one batch, the broker appends it, and the acknowledgement is lost. Explain how producer idempotence lets the broker handle the retry, and explain one duplicate scenario it does not solve.
  3. You are reviewing an exactly-once claim for a pipeline that reads Kafka, calls an HTTP billing API, and writes a Kafka audit topic transactionally. What part of the claim is valid, and what additional billing API feature would you require?
  4. Why do read-committed consumers sometimes appear to lag behind read-uncommitted consumers during long transactions? What operational change would reduce that delay?

Summary

Delivery semantics are about failure boundaries. At-most-once can lose data. At-least-once preserves data by allowing retries, so sinks must be idempotent. Kafka producer idempotence removes duplicate appends from producer retry ambiguity. Kafka transactions can atomically publish output and consumed offsets for Kafka-to-Kafka processing. Exactly-once claims are credible only when they state the boundary and explain how every side effect inside that boundary is made atomic or idempotent.