Kafka practical with Docker

1. Install Docker Desktop

Download Docker Desktop for your operating system, install it, and open it. Wait until the Docker engine is running.

Download Docker Desktop ↗

Docker runs Kafka in an isolated container. Docker Compose lets us describe the container in one YAML file.

2. Check Docker

Open Terminal on macOS or Linux, or PowerShell on Windows. Run these commands one at a time:

docker --version
docker compose version
docker info

The first two commands should print version numbers. The docker info command should show a Server section. If Docker cannot connect, open Docker Desktop and try again.

3. Create the Kafka setup

Create a folder named kafka-docker. Save the following file as compose.yaml inside it. Run each command block from this folder.

Download compose.yaml ↓

services:
  broker:
    image: apache/kafka:4.3.1
    container_name: kafka-broker
    restart: unless-stopped
    ports:
      - "127.0.0.1:9092:9092"
    environment:
      KAFKA_NODE_ID: "1"
      KAFKA_PROCESS_ROLES: "broker,controller"
      KAFKA_LISTENERS: "PLAINTEXT://:9092,CONTROLLER://:9093"
      KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://localhost:9092"
      KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT"
      KAFKA_CONTROLLER_QUORUM_VOTERS: "1@localhost:9093"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: "1"
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: "1"
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: "1"
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: "0"
      KAFKA_NUM_PARTITIONS: "3"

This is a single-node learning setup. Kafka runs in KRaft combined mode, where the broker and controller run in the same container. It has one broker and replication factor one, so it is not a production fault-tolerant cluster.

4. Start Kafka

Validate the Compose file, start the broker, and check its status. Run these lines one at a time:

docker compose config --quiet
docker compose up -d
docker compose ps

Expect the broker service to show as running. If it is still starting, wait a few seconds and run docker compose ps again.

docker compose logs --tail=100 broker

5. Create a topic

A topic is the named stream where Kafka stores events. Create a three-partition topic for our ride events:

docker compose exec broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic ride-events --partitions 3 --replication-factor 1

Inspect the topic:

docker compose exec broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic ride-events

You should see three partitions. Because this local setup has one broker, each partition has replication factor one.

6. Produce events

Start the console producer. It waits for one event per line. Run the command, then type the two JSON lines below one at a time. Press Ctrl+C when finished.

docker compose exec broker /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic ride-events

{"event":"RideRequested","eventId":"evt-1","rideId":"R101","pickup":"Airport","destination":"City Center"}
{"event":"TripCompleted","eventId":"evt-2","rideId":"R101"}

Here the console producer is acting as a simple Kafka producer. In the ride application, the Ride Service would publish these events.

7. Consume events

Open a second terminal in the same kafka-docker folder. Start a consumer from the beginning of the topic:

docker compose exec broker /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic ride-events --from-beginning

You should see both events. Leave this consumer running, return to the producer terminal, and type another event. The consumer prints the new event as Kafka appends it to the topic. Press Ctrl+C to stop the consumer.

8. Inspect the topic and consumer group

Give the consumer a group name so Kafka can remember its position. Stop the previous consumer, then run:

docker compose exec broker /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic ride-events --group analytics-service --from-beginning

In another terminal, inspect the group:

docker compose exec broker /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group analytics-service

The output shows the topic, partition, current consumer offset, latest offset, and LAG. Lag is the number of records the consumer has not processed yet.

When finished, stop the setup from the project folder:

docker compose down

This removes the container. Starting it again creates a fresh local broker.