System Design / Messaging
Messaging queues.
Services do not always need to call one another directly. Messaging systems let one service send work or events while another service processes them later.
01 / EVENT STREAMING
Kafka
A durable event stream where topics are divided into partitions and many consumer groups can read events independently.
Read the Kafka guide →02 / MANAGED QUEUES
Amazon SQS
A managed AWS queue for decoupling services and distributing work to consumers. Messages are received, processed, and deleted after successful handling.
Read the SQS guide →03 / MESSAGE ROUTING
RabbitMQ
A message broker where exchanges route messages to queues, allowing services to distribute work and apply routing rules.
Read the RabbitMQ guide →Choose a tool
How are Kafka, RabbitMQ, and SQS different?
All three move data between services, but they organize and deliver that data differently.
| System | Ownership and model | What happens after reading? | Best starting use |
|---|---|---|---|
| Kafka | Open-source event streaming with topics and partitions. | The message is not deleted when a consumer reads it. The consumer commits an offset, and Kafka retains the event until its retention policy removes it. | Many independent services consuming high-volume events, analytics, and replay. |
| RabbitMQ | Open-source message broker with exchanges and queues. | In a normal queue, RabbitMQ deletes the message after acknowledgement. It can requeue the message when processing fails. | Work queues, task distribution, and routing different messages to different workers. |
| Amazon SQS | Amazon AWS-managed queue service. | The consumer explicitly deletes the message after successful processing. Visibility timeout makes it available again when processing fails. | Background jobs and service decoupling when AWS should manage the queue servers. |
A simple decision guide
- Choose Kafka when an event may be read by several applications and replayed later.
- Choose RabbitMQ when broker-side routing and worker queues are central to the design.
- Choose SQS when you want a managed AWS queue and do not want to operate message-broker servers.
Key difference: Kafka keeps messages after consumers read them, until retention removes them. In normal RabbitMQ queues and SQS queues, successful processing is followed by acknowledgement or deletion, so the message is removed. Kafka and RabbitMQ are open-source projects that you operate or run through a provider; SQS is managed by Amazon.
These are useful beginner distinctions. Modern RabbitMQ and Kafka also support overlapping features, so a production choice should consider throughput, ordering, operations, cost, and the surrounding cloud platform.
We will learn one system at a time, then apply each choice to a real design.