Three-broker Kafka cluster with Kafka UI

1. Create the setup

Create a folder named kafka-cluster-docker. Download the Compose file and save it there as kafka-cluster-compose.yaml.

Download the three-broker compose file ↓

The file starts three Kafka brokers and one Kafka UI container. The brokers form one KRaft cluster. The UI connects to all three brokers over Docker's internal network.

2. Start the cluster

Run each command from the folder containing kafka-cluster-compose.yaml:

docker compose -f kafka-cluster-compose.yaml config --quiet
docker compose -f kafka-cluster-compose.yaml up -d
docker compose -f kafka-cluster-compose.yaml ps

Wait until broker-1, broker-2, broker-3, and kafka-ui are running.

3. Open Kafka UI

Open http://localhost:8080 ↗. Select local-cluster to browse brokers, topics, partitions, consumer groups, and messages.

4. Create a replicated topic

Create six partitions with a replication factor of three. Kafka places each partition's replicas across the three brokers.

docker compose -f kafka-cluster-compose.yaml exec broker-1 /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker-1:29092 --create --topic ride-events --partitions 6 --replication-factor 3
docker compose -f kafka-cluster-compose.yaml exec broker-1 /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker-1:29092 --describe --topic ride-events

A partition is an ordered log. Each has one leader replica and follower replicas. Replication lets the topic remain available if one broker fails.

5. Produce events into different partitions

Use a key so related events are routed consistently. The key is before the colon; Kafka hashes it to choose a partition. Run the producer, then enter the sample lines one at a time. Stop it with Ctrl+C.

docker compose -f kafka-cluster-compose.yaml exec broker-1 /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server broker-1:29092 --topic ride-events --property parse.key=true --property key.separator=:

R101:{"event":"RideRequested","rideId":"R101"}
R202:{"event":"RideRequested","rideId":"R202"}
R303:{"event":"TripCompleted","rideId":"R303"}
R404:{"event":"RideRequested","rideId":"R404"}

These keys may land in different partitions. Events with the same key, such as two events for R101, stay in the same partition and keep their order.

6. Compare consumer assignments

Two consumers in one group

Open two terminals and run the same command, using the same group name. Start both consumers first, wait a few seconds for Kafka to assign partitions, and then produce a few new events. The options print each record's partition and key:

docker compose -f kafka-cluster-compose.yaml exec broker-1 /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server broker-1:29092 --topic ride-events --group analytics-service --from-beginning --property print.partition=true --property print.key=true

docker compose -f kafka-cluster-compose.yaml exec broker-1 /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server broker-1:29092 --topic ride-events --group analytics-service --from-beginning --property print.partition=true --property print.key=true

Because both consumers belong to analytics-service, Kafka assigns each partition to only one of them at a time. One terminal may print partition 0 and 3 while the other prints partition 1 and 4. They share the work; they do not both read the same partition at the same time. With six partitions, this group can use up to six active consumers. Stop both consumers with Ctrl+C.

Two different groups

Now run one consumer for each group in separate terminals:

docker compose -f kafka-cluster-compose.yaml exec broker-1 /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server broker-1:29092 --topic ride-events --group analytics-service --from-beginning --property print.partition=true

docker compose -f kafka-cluster-compose.yaml exec broker-1 /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server broker-1:29092 --topic ride-events --group notification-service --from-beginning --property print.partition=true

These groups have separate offsets, so both receive the same events. A group is a logical application: analytics can process every event for reports while notifications independently process every event needed for messages.

Where Kafka stores offsets

When a consumer commits progress, Kafka stores that position in an internal topic named __consumer_offsets. Think of it as Kafka's own offset queue: it records the group name, topic, partition, and the next offset that group should read. It stores positions, not copies of your ride events. Each consumer group has its own offset record, so analytics-service and notification-service can be at different positions.

docker compose -f kafka-cluster-compose.yaml exec broker-1 /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker-1:29092 --list
docker compose -f kafka-cluster-compose.yaml exec broker-1 /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker-1:29092 --describe --topic __consumer_offsets

Kafka manages this internal topic for you. Its partitions are distributed across the brokers and replicated like other Kafka data. The consumer-group describe command reads these committed positions and calculates lag from them.

8. Simulate a broker failure

Replicas make a cluster resilient. First inspect the current leaders and replicas, then stop one broker. Run these commands while the other brokers are still running:

docker compose -f kafka-cluster-compose.yaml exec broker-1 /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker-1:29092 --describe --topic ride-events

docker compose -f kafka-cluster-compose.yaml stop broker-1
docker compose -f kafka-cluster-compose.yaml ps

Wait a few seconds, then describe the topic again:

docker compose -f kafka-cluster-compose.yaml exec broker-2 /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker-2:29092 --describe --topic ride-events

For partitions whose leader was on broker-1, Kafka elects an in-sync follower on another broker. The leader number can change, while the partition and its records remain available. The exact partitions that move depend on the assignment in your run. Kafka UI also shows the new leaders.

Start the broker again and let it catch up with its replicas:

docker compose -f kafka-cluster-compose.yaml start broker-1
docker compose -f kafka-cluster-compose.yaml ps
docker compose -f kafka-cluster-compose.yaml logs --tail=50 broker-1

This exercise demonstrates why a three-broker cluster with replication factor three can continue serving data when one broker is temporarily unavailable. It is a learning setup; always wait for the broker to become healthy before stopping another one.

9. Inspect the cluster

Inspect both groups and the broker logs:

docker compose -f kafka-cluster-compose.yaml exec broker-1 /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server broker-1:29092 --describe --group analytics-service
docker compose -f kafka-cluster-compose.yaml exec broker-1 /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server broker-1:29092 --describe --group notification-service
docker compose -f kafka-cluster-compose.yaml logs --tail=50 broker-1 broker-2 broker-3

Kafka UI shows the same information visually, including partition leaders, replicas, consumer offsets, and lag. When finished, stop the cluster:

docker compose -f kafka-cluster-compose.yaml down