Kafka Brokers Topics Partitions and Consumer Groups

Published by Programming Line in Data Engineering in Depth. Last updated .

Purpose: what Kafka is organizing

In a data engineering system, Kafka is the durable handoff point between producers that create events and consumers that process them. Brokers, topics, partitions, and consumer groups are the four terms that explain most Kafka behavior. A broker is a server that stores partition files and answers producer and consumer requests. A topic is a named stream, such as orders.created. A partition is an ordered, append-only log inside a topic. A consumer group is a set of consumer processes that cooperate so each partition is read by one group member at a time.

The outcome of this lesson is practical: given a stream, you should be able to decide how many partitions it needs, predict which records preserve order, explain why adding consumers sometimes helps and sometimes does nothing, diagnose lag, and recover from common broker or consumer failures without guessing.

How the mechanism works internally

A Kafka topic is not one physical queue. It is a collection of partition logs spread across brokers. Each partition is a sequence of records identified by monotonically increasing offsets: offset 0, offset 1, offset 2, and so on. Producers append records to the end of a partition. Consumers fetch records by topic, partition, and offset. Kafka does not delete a record when one consumer reads it. Retention is time-based, size-based, or compaction-based, so many independent applications can replay the same events.

Partition placement is coordinated by the cluster controller. Each partition has one leader replica and usually one or more follower replicas. Clients read from and write to the leader by default. Followers copy the leader’s log. Replicas that are caught up enough are called in-sync replicas. When a broker fails, a follower that is in sync can become the new leader. This is why replication protects availability, while acknowledgments and minimum in-sync replica settings control how much durability a producer waits for.

Ordering is partition-scoped. Kafka can guarantee that records appended to one partition are read in offset order by a consumer group member. It cannot guarantee total order across all partitions in a topic. A producer key, such as customer id or device id, is commonly hashed to choose a partition. That makes all records for the same key land in the same partition, preserving per-key order while allowing unrelated keys to be processed in parallel.

A consumer group turns partitions into units of work. If a topic has twelve partitions and a group has three active consumers, Kafka assigns about four partitions to each consumer. If the same group grows to twelve consumers, each can receive one partition. If it grows to twenty consumers, eight consumers sit idle because a partition cannot be actively owned by two members in the same group. A different group can read all partitions independently because its committed offsets are separate.

Configuration and API anatomy

Topic creation usually requires three important choices: partition count, replication factor, and retention policy. Partition count sets the upper bound for parallelism within one consumer group. Replication factor controls how many brokers store each partition. Retention controls how long records remain available for replay. These settings are data modeling decisions, not just infrastructure defaults.

Producer behavior is shaped by bootstrap.servers, key.serializer, value.serializer, acks, batching settings, and optional idempotence. The producer first contacts a bootstrap broker to discover metadata, then sends each batch to the leader for the selected partition. A keyed record uses a partitioner. An unkeyed record is usually distributed for throughput, which means no meaningful order should be inferred across produced records.

Consumer behavior is shaped by group.id, bootstrap.servers, deserializers, auto.offset.reset, offset commit strategy, and poll loop settings. The consumer joins its group, receives partition assignments, fetches records, processes them, and commits offsets. A committed offset means “the next offset this group should read.” If a consumer processes offset 41 and commits 42, a replacement consumer starts at 42 after reassignment.

Example 1: choosing partitions with keys

The first example models a simple keyed partitioner. Real Kafka clients use a defined partitioning algorithm, and exact hashing can vary by client and configuration. The important behavior is stable mapping: the same key repeatedly chooses the same partition as long as the partition count and partitioner stay unchanged.

import hashlib

partitions = 4
keys = ["customer-7", "customer-2", "customer-7", "customer-9"]

for key in keys:
    digest = hashlib.sha256(key.encode("utf-8")).hexdigest()
    partition = int(digest, 16) % partitions
    print(f"{key} -> partition {partition}")

Expected behavior: both customer-7 records print the same partition number. That means updates for that customer can be processed in order by a single group member. The trade-off is skew: if one customer generates half of all traffic, one partition becomes hot while other partitions are underused.

Example 2: assigning partitions to a consumer group

This example approximates range assignment for one topic. Kafka has several assignors, but all of them must respect the same constraint: within one group, one partition has at most one active owner.

partitions = list(range(8))
consumers = ["consumer-a", "consumer-b", "consumer-c"]

assignments = {consumer: [] for consumer in consumers}
for index, partition in enumerate(partitions):
    owner = consumers[index % len(consumers)]
    assignments[owner].append(partition)

for consumer, owned in assignments.items():
    print(f"{consumer}: {owned}")

Expected output is deterministic for this script: consumer-a owns partitions 0, 3, and 6; consumer-b owns 1, 4, and 7; consumer-c owns 2 and 5. If you add a fourth consumer, work is redistributed. If you add a ninth consumer for eight partitions, one member receives no partitions.

Example 3: consumer lag from offsets

Lag is the distance between the latest available offset in a partition and the group’s committed offset. It is not automatically bad. A batch job may intentionally build lag and drain it later. In a near-real-time pipeline, sustained increasing lag means processing capacity is below input rate or consumers are stuck.

end_offsets = {0: 105, 1: 88, 2: 140}
committed_offsets = {0: 100, 1: 88, 2: 115}

total_lag = 0
for partition, end in end_offsets.items():
    committed = committed_offsets[partition]
    lag = end - committed
    total_lag += lag
    print(f"partition {partition}: lag {lag}")

print(f"total lag: {total_lag}")

The output is partition 0 lag 5, partition 1 lag 0, partition 2 lag 25, and total lag 30. Partition 2 deserves attention first. The next diagnostic question is whether that partition receives more records, contains slow records, or is assigned to a weak consumer.

Design choices and trade-offs

Partition count is the most visible design choice. More partitions allow more parallel consumers and higher aggregate write throughput, but they add file handles, metadata, leader election work, and recovery overhead. Increasing partitions later can change key-to-partition mapping for many keys, so systems that depend on per-key ordering should plan partition counts with future throughput in mind.

Key choice is a data modeling decision. A customer id key is good for customer-ordered workflows, such as account balance updates. A region key may be useful for regional aggregation but can create severe skew if one region dominates traffic. A null key improves distribution but gives up business ordering. Composite keys can spread load while preserving the order that actually matters, such as tenant_id:device_id.

Consumer group boundaries decide whether applications compete or independently subscribe. Two replicas of the same service should use the same group.id so work is shared. Two different downstream systems, such as fraud detection and warehouse ingestion, should use different group ids so each receives every record. Reusing a group id accidentally is a common cause of missing data in new consumers.

Failure modes and troubleshooting

Symptom: consumers are running but no records are processed. The usual causes are an unexpected group id, offsets already committed at the end of the topic, or assignment to another live member. Diagnose by describing the consumer group, checking current offset versus log end offset, and listing active members. Correct by using the intended group.id, resetting offsets only when replay is acceptable, or shutting down duplicate members.

Symptom: lag grows on one partition while others stay current. Causes include key skew, a poison record that repeatedly fails, or a slow external dependency used by the consumer handling that partition. Diagnose per-partition lag, inspect the assigned consumer, and sample records around the stuck offset. Correct by fixing the bad record handling, adding retries with a dead-letter topic, improving the slow dependency, or changing the key strategy for future topics.

Symptom: producers receive timeout or not enough replicas errors. A leader may be unavailable, in-sync replica count may be below the producer’s durability requirement, or brokers may be overloaded. Diagnose broker health, partition leader placement, under-replicated partitions, and producer acks. Correct by restoring brokers, reducing overload, or aligning min.insync.replicas and producer acks with the durability target. Lowering durability can hide the symptom but increases acknowledged-write loss risk during failures.

Symptom: processing duplicates after a restart. Kafka normally provides at-least-once processing unless the application uses transactions or idempotent sinks. If a consumer processes records and crashes before committing offsets, those records are read again. Diagnose by comparing sink writes with committed offsets and restart times. Correct with idempotent writes, committing after successful processing, or transactional consume-process-produce patterns where the use case justifies the complexity.

Security, performance, and reliability implications

Security controls should match Kafka’s named resources. Authentication proves which producer or consumer is connecting. Authorization should limit principals to the topics and consumer groups they need. A service that only writes orders.created should not be able to read payroll topics or alter offsets for a warehouse consumer group. Encryption in transit protects records and credentials between clients and brokers.

Performance comes from batching, compression, partition parallelism, and efficient consumers. Larger producer batches reduce request overhead but add latency. Compression reduces network and disk usage but costs CPU. Consumers should keep polling and avoid long blocking work inside the poll loop; slow side effects can be moved to bounded worker pools while preserving per-partition commit discipline.

Reliability depends on replication, acknowledgments, offset management, and replay design. A replicated topic with careful producer acknowledgments can survive a broker failure without losing acknowledged records. A consumer that commits too early can lose work when processing fails. A consumer that commits too late can duplicate work. Kafka gives the primitives; the pipeline’s correctness depends on matching those primitives to the sink’s behavior.

Hands-on lab: observe partitions, groups, and lag

Prerequisites: a local Kafka environment, command-line access to Kafka tools, and a disposable cluster or namespace. The commands below use common tool names; adjust the bootstrap server for your environment.

BOOTSTRAP=localhost:9092
TOPIC=lesson.orders
GROUP=lesson-workers

kafka-topics --bootstrap-server "$BOOTSTRAP" --create --topic "$TOPIC" --partitions 3 --replication-factor 1
kafka-topics --bootstrap-server "$BOOTSTRAP" --describe --topic "$TOPIC"

printf 'order-1\norder-2\norder-3\norder-4\n' | kafka-console-producer --bootstrap-server "$BOOTSTRAP" --topic "$TOPIC"

kafka-console-consumer --bootstrap-server "$BOOTSTRAP" --topic "$TOPIC" --group "$GROUP" --from-beginning --max-messages 4
kafka-consumer-groups --bootstrap-server "$BOOTSTRAP" --describe --group "$GROUP"

kafka-topics --bootstrap-server "$BOOTSTRAP" --delete --topic "$TOPIC"

Step 1 creates a three-partition topic. Verify that the description shows three partitions and a leader for each one. Step 2 produces four records. Because the console producer has no explicit key in this simple form, records may be distributed by the producer’s default behavior and should not be treated as globally ordered across partitions. Step 3 starts a consumer group and reads four messages from the beginning. Step 4 describes the group; after consumption, current offsets should have advanced and lag should be zero or near zero. Cleanup deletes the disposable topic. If deletion is disabled in your environment, use a short-retention topic name and remove any test consumers instead.

Assessment exercises

  1. A topic has six partitions and one consumer group has ten healthy members. How many members can actively process records from that topic, and why?
  2. An orders topic is keyed by customer_id. One enterprise customer creates most events, and only one partition has high lag. Propose two mitigations and explain what ordering each one preserves or gives up.
  3. A consumer writes to a database and commits offsets before the database transaction completes. Describe the failure that can lose data and change the sequence to avoid it.
  4. Two independent applications accidentally use the same group.id. What symptom would each team see, and what change should they make?
  5. You increase a topic from 12 to 24 partitions. Explain why throughput may improve and why keyed ordering assumptions should be reviewed.

Summary

Kafka brokers store replicated partition logs. Topics name streams, partitions provide ordered append-only logs and parallelism, and consumer groups coordinate which process owns each partition. The core data engineering skill is mapping business ordering, throughput, replay, and failure recovery requirements onto these primitives. Good Kafka designs choose keys deliberately, size partitions for expected parallelism, commit offsets only after successful work, and troubleshoot by looking at leaders, assignments, offsets, and per-partition lag instead of treating a topic as a single queue.