Events, Kafka, and Transactional Messaging

Events let a Spring Boot service say that something meaningful happened without forcing every reaction to live in the same method. Kafka lets those facts move between services as ordered records. Transactional messaging is the discipline of making the database change and the message publication agree, even when a process crashes between the two.

In this Spring Boot integration lesson, the outcome is concrete: when an order is placed, the application commits the order and eventually publishes one durable OrderPlaced message. Consumers can retry safely, duplicate records do not create duplicate side effects, and an operator has a clear way to diagnose stuck messages.

How the Mechanism Works

A local Spring event is an in-process notification. Code calls ApplicationEventPublisher.publishEvent, and Spring invokes matching @EventListener methods in the same application. By default that listener runs in the publishing thread. A @TransactionalEventListener adds transaction awareness: it can run before commit, after commit, after rollback, or after completion. For integration events, AFTER_COMMIT is usually the important phase because it prevents a listener from publishing a message for a database change that later rolls back.

Kafka stores records in topics split into partitions. A producer chooses a topic, optional key, value, and headers. Records with the same key go to the same partition, so Kafka preserves their relative order within that partition. A consumer group divides partitions among group members. Each group tracks offsets, which are positions in each partition. A listener that commits an offset is saying, in effect, that records up to this point no longer need to be delivered to this group.

The difficult part is that a relational database transaction and a Kafka send are not automatically one atomic unit. If the service writes an order and crashes before sending Kafka, downstream services never hear about the order. If it sends Kafka before committing the order and the transaction rolls back, downstream services react to a fact that never became true. Kafka producer transactions can make groups of Kafka writes atomic, but they do not automatically include your database. The common Spring Boot answer is the transactional outbox: write the business row and an outbox row in the same database transaction, then have a separate publisher read unsent outbox rows and send them to Kafka.

API and Configuration Anatomy

The usual pieces are a domain operation, a domain or application event, a Kafka topic contract, producer serialization, listener container settings, retry policy, and idempotency storage. Spring Boot auto-configures Kafka support when the Kafka client and Spring Kafka are on the classpath, then binds properties under spring.kafka. The application still owns the important choices: topic names, message keys, payload schema, retries, acknowledgments, and how offsets are committed.

spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      properties:
        enable.idempotence: true
        acks: all
    consumer:
      group-id: billing-service
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring.json.trusted.packages: com.example.orders.events

The producer settings ask Kafka to acknowledge records only when the configured replicas have accepted them and enable producer idempotence so client retries do not create extra records from the same producer session. The consumer settings define one logical subscriber, billing-service, and configure JSON deserialization for event classes in a bounded package.

Example 1: Publish After Commit

The first example stays inside one Spring Boot application. The service saves an order and publishes a local event while the database transaction is active. The listener is marked AFTER_COMMIT, so the email or Kafka handoff does not happen when validation or persistence fails later in the transaction.

package com.example.orders;

import org.springframework.context.ApplicationEventPublisher;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.transaction.event.TransactionPhase;
import org.springframework.transaction.event.TransactionalEventListener;

record OrderPlaced(Long orderId, String customerId) {}

@Service
class OrderService {
    private final OrderRepository orders;
    private final ApplicationEventPublisher events;

    OrderService(OrderRepository orders, ApplicationEventPublisher events) {
        this.orders = orders;
        this.events = events;
    }

    @Transactional
    Long placeOrder(String customerId) {
        Order order = orders.save(new Order(customerId));
        events.publishEvent(new OrderPlaced(order.id(), customerId));
        return order.id();
    }
}

@Service
class OrderNotificationListener {
    @TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT)
    void on(OrderPlaced event) {
        System.out.println("committed order " + event.orderId());
    }
}

Expected behavior: if placeOrder commits order 42, the listener prints committed order 42. If the transaction rolls back, the listener is not called. This is useful for local side effects, but it is still not durable messaging. If the JVM commits and then dies before the listener completes, the event is gone because it lived only in memory.

Example 2: Send a Kafka Record

The second example turns the committed event into a Kafka record. The order id is used as the Kafka key so all records for the same order stay on one partition in the order sent by this producer.

package com.example.orders;

import java.util.concurrent.CompletableFuture;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.support.SendResult;
import org.springframework.stereotype.Component;
import org.springframework.transaction.event.TransactionPhase;
import org.springframework.transaction.event.TransactionalEventListener;

@Component
class OrderKafkaPublisher {
    private final KafkaTemplate<String, OrderPlaced> kafka;

    OrderKafkaPublisher(KafkaTemplate<String, OrderPlaced> kafka) {
        this.kafka = kafka;
    }

    @TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT)
    void publish(OrderPlaced event) {
        CompletableFuture<SendResult<String, OrderPlaced>> send =
            kafka.send("orders.placed", event.orderId().toString(), event);
        send.whenComplete((result, failure) -> {
            if (failure != null) {
                System.err.println("Kafka publish failed for order " + event.orderId());
            }
        });
    }
}

Expected behavior: a successful send appends one record to topic orders.placed with key 42 and a JSON value representing OrderPlaced. This is better than a direct call from inside the transaction, but there is still a gap: if the send fails after commit, the order remains saved and no durable reminder exists unless the application records that failure somewhere.

Example 3: Transactional Outbox

The outbox closes that gap by making the message itself part of the database transaction. The service writes both the order and an outbox row. A publisher later sends unsent rows to Kafka and marks them as sent. The outbox row should contain a stable event id, aggregate id, event type, payload, creation time, status, attempt count, and last error.

package com.example.orders;

import java.time.Instant;
import java.util.UUID;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

@Service
class OrderApplicationService {
    private final OrderRepository orders;
    private final OutboxRepository outbox;
    private final JsonWriter json;

    OrderApplicationService(OrderRepository orders, OutboxRepository outbox, JsonWriter json) {
        this.orders = orders;
        this.outbox = outbox;
        this.json = json;
    }

    @Transactional
    Long placeOrder(String customerId) {
        Order order = orders.save(new Order(customerId));
        OrderPlaced event = new OrderPlaced(order.id(), customerId);
        outbox.save(new OutboxMessage(
            UUID.randomUUID(),
            "Order",
            order.id().toString(),
            "OrderPlaced",
            json.write(event),
            Instant.now(),
            "NEW",
            0,
            null));
        return order.id();
    }
}

Expected behavior: after a successful transaction, the database has one order row and one NEW outbox row. After rollback, neither row exists. This gives the publisher a durable backlog to resume after process restart.

package com.example.orders;

import java.util.List;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import org.springframework.transaction.annotation.Transactional;

@Component
class OutboxPublisher {
    private final OutboxRepository outbox;
    private final KafkaTemplate<String, String> kafka;

    OutboxPublisher(OutboxRepository outbox, KafkaTemplate<String, String> kafka) {
        this.outbox = outbox;
        this.kafka = kafka;
    }

    @Scheduled(fixedDelayString = "${outbox.publish-delay:PT5S}")
    @Transactional
    void publishBatch() {
        List<OutboxMessage> messages = outbox.lockNextBatch(50);
        for (OutboxMessage message : messages) {
            kafka.send("orders.placed", message.aggregateId(), message.payload()).join();
            message.markSent();
        }
    }
}

Expected behavior: each scheduler run locks a small batch, sends each payload to Kafka, and marks successful rows as sent. A production implementation often separates the Kafka send from a long database transaction or uses careful row status transitions, but the invariant remains: unsent messages stay queryable and retryable.

Example 4: Idempotent Consumer

Kafka delivery to consumers is normally at least once. A listener may process a record, crash before committing its offset, and then receive the same record again. Consumers must make side effects idempotent by recording an event id or enforcing a unique business key.

package com.example.billing;

import org.springframework.dao.DuplicateKeyException;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
import org.springframework.transaction.annotation.Transactional;

@Component
class BillingOrderListener {
    private final ProcessedEventRepository processed;
    private final InvoiceRepository invoices;

    BillingOrderListener(ProcessedEventRepository processed, InvoiceRepository invoices) {
        this.processed = processed;
        this.invoices = invoices;
    }

    @KafkaListener(topics = "orders.placed", groupId = "billing-service")
    @Transactional
    void on(OrderPlaced event) {
        try {
            processed.insert(event.orderId().toString());
        } catch (DuplicateKeyException duplicate) {
            return;
        }
        invoices.createForOrder(event.orderId(), event.customerId());
    }
}

Expected behavior: the first delivery creates one invoice and records the order id as processed. A duplicate delivery returns before creating another invoice. This is more dependable than assuming Kafka will never redeliver.

Design Choices and Trade-offs

Choose local Spring events when the reaction belongs to the same application and can be reconstructed or tolerated if the process stops. Choose direct Kafka publishing when a missed message is acceptable or when another compensating mechanism already exists. Choose an outbox when the message is part of the business truth and must survive application crashes.

Keys are a design choice. Keying by order id preserves per-order ordering but spreads a busy customer’s orders across partitions. Keying by customer id preserves per-customer order but can overload one partition for a large customer. Payloads are also contracts. A small event containing identifiers forces consumers to query back, which increases coupling and latency. A richer event reduces round trips but requires schema evolution discipline.

Batch size and polling frequency affect latency and database load. Small batches publish quickly but increase query overhead. Large batches improve throughput but can hold locks longer and make one bad record block more work unless failures are isolated. Retention policy matters too: Kafka retention gives consumers replay windows, while outbox retention gives producers audit and repair windows.

Failure Modes and Troubleshooting

Symptom: an order exists but no Kafka message appears. Cause: direct publishing failed after commit, or the outbox publisher is stopped. Diagnostics: query the outbox for NEW or FAILED rows, inspect publisher logs, and check Kafka producer metrics for send failures. Correction: restart the publisher, fix the broker or serialization error, and republish unsent rows rather than recreating orders.

Symptom: downstream invoices are duplicated. Cause: the consumer treats Kafka delivery as exactly once for external side effects. Diagnostics: compare duplicate invoice rows with Kafka offsets and application restart times. Correction: add a processed-event table or a unique invoice constraint based on the source event id or aggregate id.

Symptom: a listener repeatedly logs deserialization errors and the partition stops advancing. Cause: the consumer cannot deserialize a record at the current offset. Diagnostics: inspect the failing topic, partition, offset, headers, and configured trusted packages. Correction: deploy the correct event class or schema, route poison records to a dead-letter topic, then resume the consumer group.

Symptom: messages arrive out of order for the same order. Cause: producers used inconsistent keys or multiple topics represent one ordered workflow without coordination. Diagnostics: inspect keys and partitions for the affected records. Correction: standardize the message key and document where ordering is guaranteed.

Reliability and Security Implications

Reliability comes from explicit retry boundaries. Retrying the producer is safe only when the message identity and consumer behavior tolerate duplicates. Retrying the consumer is safe only when its side effects are idempotent or guarded by unique constraints. Dead-letter topics are useful for diagnosis, but they are not a substitute for an owner and replay procedure.

Security depends on treating event payloads as shared data. Do not put secrets, raw payment details, or unnecessary personal data in Kafka messages. Use broker authentication and authorization so each service can read and write only its topics. Limit deserialization trust to known packages or schemas, because accepting arbitrary types from a message broker is an unsafe boundary.

Hands-on Lab

Prerequisites: a Spring Boot application with Spring Web, Spring Data JPA, Spring Kafka, a relational database, and access to a local Kafka broker. Create a topic named orders.placed. Enable scheduling with @EnableScheduling.

1. Create an orders table and an outbox_messages table with a primary key event id, aggregate id, event type, payload, status, attempts, and last error.
2. Implement an order service that saves an order and a NEW outbox row in one @Transactional method.
3. Implement a scheduled publisher that reads a limited batch of NEW rows, sends each payload to orders.placed, and marks successful rows SENT.
4. Implement a Kafka listener in a separate consumer group that records processed event ids before creating its side effect.
5. Place one order through an HTTP endpoint or integration test.

Verification: the order table should contain one order, the outbox table should move from NEW to SENT, the Kafka topic should contain one record keyed by the order id, and the consumer side effect should appear exactly once even if you manually replay the same event. To test recovery, stop Kafka, place an order, confirm the outbox row remains unsent, restart Kafka, and confirm the publisher sends it.

Cleanup: stop the application, delete test rows from the order, outbox, processed-event, and side-effect tables, and delete or reset the local Kafka topic if the test data should not be replayed later.

Assessment Exercises

1. An order service commits successfully and then the process exits before Kafka acknowledges the send. Explain how direct publishing and the outbox pattern behave differently.
2. Given events for the same order must be processed in order, what Kafka key would you choose, and what throughput trade-off might that create?
3. A consumer sends a confirmation email and then crashes before committing its offset. Design a guard that prevents duplicate emails after redelivery.
4. Design a dead-letter handling procedure for a poison record. Include what data you would capture and how you would replay after fixing the bug.
5. Decide whether @TransactionalEventListener(AFTER_COMMIT) is sufficient for a billing integration. State the assumption that makes your answer true.

Summary

Spring Boot events are useful for separating reactions inside one application. Kafka is useful for durable ordered records between applications. Transactional messaging is the extra design needed when a database commit and a message must not diverge. For business events that must survive crashes, write an outbox row in the same transaction as the business change, publish it with bounded retries, and make every consumer idempotent.