Publish-Subscribe Event-Driven Architecture and Event Sourcing

Published by Programming Line in System Design and Distributed Systems. Last updated 7th September 2026.

Purpose and outcome

Publish-subscribe architecture separates the component that announces something happened from the components that react to it. Instead of an order service calling inventory, billing, email, analytics, and fulfillment directly, it publishes an OrderPlaced event. Each interested subscriber receives the event through a broker, stream, or log and performs its own work. The immediate outcome is lower coupling: the publisher does not need to know how many consumers exist, where they run, or how quickly they process.

Event sourcing is a related but different pattern. It stores state changes as an append-only sequence of events, then derives current state by replaying those events. A bank account balance is not stored as the primary fact; AccountOpened, MoneyDeposited, and MoneyWithdrawn are the primary facts. The balance is a projection. In system design, these patterns often meet: an event-sourced service appends domain events to its event store and publishes them so other services can build their own projections.

How the mechanism works

A pub-sub system has publishers, topics or channels, events, subscriptions, and consumers. A publisher writes an event to a named topic such as orders. The broker accepts the event, assigns metadata such as offset or sequence number, and makes it available to subscriptions. A subscription may represent one consumer instance, a group of competing consumers, or a durable cursor that tracks how far a downstream system has read.

Delivery semantics matter. At-most-once delivery may lose events but avoids duplicate work. At-least-once delivery retries until acknowledged, which means consumers must tolerate duplicates. Exactly-once is usually a constrained broker and processing guarantee, not a universal promise across arbitrary databases, APIs, and side effects. In most designs, the practical target is at-least-once delivery with idempotent consumers.

Event sourcing adds aggregate streams and optimistic concurrency. An aggregate is a consistency boundary such as an order, account, or shopping cart. Its event stream contains only the events for that aggregate. To update it, the service loads prior events, rebuilds aggregate state, validates the command, appends new events with an expected version, and publishes or exposes those events. If another writer already appended to the same stream, the expected version check fails and the command must be retried or rejected.

Read models are built by projections. A projection subscribes to events and writes query-optimized state, such as an order summary table or customer balance view. Rebuilding a projection means deleting or ignoring the old projection state and replaying events from the beginning or from a snapshot. This is powerful, but it creates operational duties: events must be immutable, schemas must evolve carefully, and replay must not resend external side effects such as emails or payments.

API anatomy

A useful event has a stable name, event id, aggregate id, aggregate version, occurrence time, schema version, and payload. The event id supports deduplication. The aggregate version supports ordering within one entity. The schema version helps consumers parse old and new shapes. The payload should describe what happened, not what a consumer should do. Prefer PaymentAuthorized over SendFulfillmentEmail when the event represents a business fact.

Common broker operations are publish(topic, event), subscribe(topic, group), ack(message), and seek(offset). Event store operations are read_stream(id), append_to_stream(id, expected_version, events), and sometimes snapshot(id, version, state). The important difference is that a broker is optimized for distribution and cursors, while an event store is the durable source of business history.

Example 1: one event, two subscribers

This first example shows fan-out. One publisher emits an order event. Two subscribers independently receive the same event and update separate local results.

from collections import defaultdict

subscribers = defaultdict(list)

def subscribe(topic, handler):
    subscribers[topic].append(handler)

def publish(topic, event):
    for handler in subscribers[topic]:
        handler(event)

emails = []
analytics = []

subscribe("orders", lambda event: emails.append(f"email:{event['order_id']}"))
subscribe("orders", lambda event: analytics.append(f"metric:{event['total']}"))

publish("orders", {"type": "OrderPlaced", "order_id": "O-100", "total": 42})

print(emails)
print(analytics)

The deterministic output is one email action and one analytics action. The publisher did not call either subscriber directly; it only published to the orders topic.

Example 2: event-sourced account state

This example stores the history as events and computes the balance by replaying them. The current state is derived, so a later projection can be rebuilt from the same facts.

events = [
    {"type": "AccountOpened", "account_id": "A-1"},
    {"type": "MoneyDeposited", "account_id": "A-1", "amount": 100},
    {"type": "MoneyWithdrawn", "account_id": "A-1", "amount": 35},
]

def apply(balance, event):
    if event["type"] == "AccountOpened":
        return 0
    if event["type"] == "MoneyDeposited":
        return balance + event["amount"]
    if event["type"] == "MoneyWithdrawn":
        return balance - event["amount"]
    raise ValueError(event["type"])

balance = None
for event in events:
    balance = apply(balance, event)

print(balance)

The output is 65. If a balance projection is lost, the service can replay the three events and recover the same value. If an event is wrong, do not edit it in place; append a compensating event such as WithdrawalReversed.

Example 3: idempotent consumer

At-least-once delivery can deliver the same event more than once. A consumer protects itself by recording event ids it has already applied.

processed_event_ids = set()
shipments = []

def handle_order_paid(event):
    if event["event_id"] in processed_event_ids:
        return "duplicate ignored"
    processed_event_ids.add(event["event_id"])
    shipments.append({"order_id": event["order_id"], "status": "ready"})
    return "shipment created"

event = {"event_id": "evt-9", "type": "OrderPaid", "order_id": "O-200"}
print(handle_order_paid(event))
print(handle_order_paid(event))
print(shipments)

The expected behavior is that only one shipment is created. The second delivery is acknowledged without repeating the side effect. In production, the processed-event record must be stored atomically with the consumer’s state change.

Design choices and trade-offs

Choose pub-sub when multiple independent reactions follow the same fact, when consumers evolve at different speeds, or when a system needs asynchronous buffering. Avoid it for operations that require an immediate strongly consistent answer from another service; a synchronous request may be clearer.

Choose event sourcing when the history is valuable, state transitions are central to the domain, audits matter, or projections need to be rebuilt in different forms. It adds complexity: developers must reason about replay, schema evolution, duplicate delivery, and temporal queries. For simple CRUD records with little audit value, a regular database table plus a transactional outbox may be easier.

Ordering is another trade-off. Global ordering limits throughput and availability. Ordering per aggregate, partition, or key is usually enough. For example, all events for one order should keep their order, but events for unrelated orders can process in parallel. Partition keys therefore become design decisions, not just broker settings.

Failure modes and troubleshooting

Duplicate side effects show up as repeated emails, repeated shipments, or double accounting entries. The cause is usually retry after a lost acknowledgement or consumer restart. Diagnose by searching for the same event id in broker logs and consumer storage. Correct it by adding idempotency keys and committing the deduplication record in the same transaction as the state update.

Projection lag appears when a write succeeds but the read model still shows old data. Causes include a slow consumer, a hot partition, broker backlog, or a failing projection loop. Diagnose by comparing the subscription cursor with the latest topic offset and checking consumer error logs. Correct it by fixing poison events, scaling consumers within partition limits, or separating slow projections from latency-sensitive ones.

Concurrency conflicts occur when two commands load the same aggregate version and both try to append. One append succeeds; the other fails its expected-version check. Diagnose by logging aggregate id, expected version, and actual version. Correct it by retrying commands that are safe to re-evaluate or returning a conflict to the caller when business intent must be reconsidered.

Replay causes external actions when a projection rebuild accidentally sends emails or calls payment APIs. The cause is mixing event interpretation with irreversible side effects. Diagnose by inspecting consumers that run during replay. Correct it by separating pure projections from process managers that perform side effects, and by marking replay mode so side-effect handlers are disabled.

Reliability, performance, and security implications

Reliability depends on durable append, acknowledgement discipline, and replayability. Use a transactional outbox when a service writes database state and publishes an event: write both the state change and an outbox row in one database transaction, then let a relay publish the outbox row. This avoids the common split-brain failure where the database commit succeeds but the publish call fails.

Performance is shaped by event size, partitioning, retention, and projection cost. Keep events compact but complete enough to be useful. Large payloads slow brokers and replay; store bulky documents elsewhere and put a content reference in the event. Snapshots can speed aggregate loading, but snapshots are caches of event history, not replacements for it.

Security is not just topic permissions. Events often contain business facts that many consumers can see. Minimize sensitive fields, encrypt where appropriate, and give publishers and subscribers least-privilege access to topics. Treat event schemas as public contracts inside the organization because accidental leakage into analytics, logs, or dead-letter queues is common.

Hands-on lab

Prerequisites: Python 3 and a terminal. No external broker is required.

Create a scratch Python file and paste the three code examples in order.
Run the file and confirm that fan-out produces one email action and one analytics action.
Add a second order event and verify both subscribers receive it without changing the publisher.
Change the account event list by adding another deposit of 10 and verify the computed balance becomes 75.
Call the idempotent shipment handler three times with the same event id and confirm the shipments list still has one item.
Cleanup by deleting the scratch file. If you used a shared repository, do not commit the lab file unless it is part of your course notes.

Verification is based on deterministic output: subscriber lists should contain one entry per published order per subscriber, the account balance should equal the replayed arithmetic, and duplicate event ids should not create duplicate shipments.

Assessment exercises

An order service publishes OrderPlaced, then the email consumer is down for ten minutes. What broker or subscription behavior determines whether the emails are eventually sent?
A projection table says an account balance is 90, but replaying the event stream gives 100. List two likely causes and how you would diagnose them.
Design an event name and payload for a subscription renewal. Which fields belong in the event metadata rather than the payload?
Why is per-order ordering usually preferable to global ordering in a high-volume commerce system?
During replay, a consumer starts charging credit cards again. Which boundary was designed incorrectly, and how would you change it?

Summary

Pub-sub distributes facts to independent consumers through topics and subscriptions. Event sourcing records the facts that changed an aggregate and derives state by replaying them. Together, they support auditability, asynchronous workflows, and rebuildable read models, but they require careful choices about ordering, idempotency, schema evolution, replay, and side effects. In system design interviews and real architectures, the strongest answer is not simply to add a broker; it is to explain which events are durable business history, which are integration messages, how consumers recover, and what consistency the user will observe.