Kafka vs RabbitMQ

RabbitMQ's strength is routing at publish time. A producer sends to an exchange, and the exchange type decides where the message goes: a direct exchange matches the routing key exactly, a topic exchange matches it against binding patterns, a fanout exchange copies the message to every bound queue and a headers exchange routes on header values. Adding a consumer with its own filter means declaring a queue and a binding, and no producer changes.

Kafka routing is simpler and sits mostly in the consumer. A producer picks a topic, and a key decides the partition, so events with the same key land in the same partition. Consumers subscribe to topics and filter what they read in their own code or in a stream processor. There is no broker-side pattern matching on message content.

Where the messaging layer owns many varied routing rules, such as sending an order event to different workers by region or priority, an exchange model fits. Where every consumer wants the whole stream of a topic and applies its own logic, a Kafka topic with a consumer group fits.

Kafka's ordering and its parallelism are the same mechanism. Within a consumer group each partition is read by exactly 1 consumer at a time, so order inside a partition is preserved and the number of partitions caps the useful number of consumers in a group. Order isn't guaranteed across partitions, so the key you choose decides which events must stay in sequence. Choosing a key such as a customer id keeps that customer's events ordered and spreads load across customers.

RabbitMQ scales a queue by adding competing consumers, and the prefetch setting limits how many unacknowledged messages each holds. A queue delivers in order, but with several consumers, or after a message is requeued, processing order is no longer the publish order. When strict order matters, single active consumer keeps 1 consumer receiving at a time and fails over to another if it disconnects. To scale beyond one queue, you split the work into several queues, which is the job of the exchange.

The rule of thumb: if you need per-key order at high volume, a partitioned log gives it by design. If you need to add workers freely to drain a backlog of independent jobs and order barely matters, competing consumers on a queue are the more direct fit.

Kafka guarantees at-least-once delivery by default, and the design documentation says a transactional producer together with a consumer using read_committed isolation can provide exactly-once processing between Kafka topics. Exactly-once into an outside system still needs that system's cooperation, so write sinks to be idempotent.

RabbitMQ with manual acknowledgements and publisher confirms is also at-least-once, and a message can be redelivered with its redelivered flag set. Its retry tools sit in the broker. A dead letter exchange receives messages that were rejected, expired, pushed out by a length limit or returned past a quorum queue's delivery limit. Since 4.0 that limit defaults to 20, so a poison message stops cycling forever.

A plain Kafka consumer has no per-message acknowledgement or dead letter queue: it commits an offset, so a message it cannot process either blocks the partition or is copied by your code to a retry or dead letter topic. Share groups change that. They track each record's acknowledgement and delivery attempt count, and Kafka states they're intended where records are processed one at a time instead of as an ordered stream. Per-message retry logic is where a queue broker saves the most code, and where share groups narrow the gap.

The gap is narrower than it was. RabbitMQ streams are a persistent, replicated append-only log that consumers can read repeatedly until it expires, with retention by age or size. A consumer can attach at the first or last message, or at an offset or timestamp. Super streams partition a stream to scale out. The documentation notes that streams don't explicitly fsync, so quorum queues offer stronger data safety.

Kafka moved the other way. In Kafka 4.2, share groups from the Queues for Kafka proposal are production-ready, so consumers can share a partition without each partition being tied to one consumer. Kafka 4.0 dropped ZooKeeper, and RabbitMQ 4.0 added native AMQP 1.0 and removed classic queue mirroring.

Durability differs in how it's built. A quorum queue is a Raft-replicated queue: with the default 3 members it tolerates 1 failure, and with 5 members 2. Its documentation lists very long backlogs and large fan-outs as better served by streams. Kafka replicates each partition to followers and counts a write committed once all in-sync replicas have it. Both are replicated, and the choice rests on the access pattern more than on safety.

Data pipelines usually want the properties of a log. Events from a database or an application land once, and a warehouse loader and a stream processor read them, along with a monitoring job, each at its own pace, with the option to reprocess a day after a bug fix. That is the pattern in Kafka vs Kinesis and in batch vs streaming designs, and it's why Kafka shows up so often in pipeline design rounds.

RabbitMQ shows up in the control plane around a pipeline: triggering jobs, fanning out notifications, handing units of work to a pool of workers with retries and a dead letter path. In a system design answer, name the access pattern first, whether messages are consumed once or read many times, whether order by key matters, and whether replay is needed. Let that pick the tool.

To practice explaining these trade-offs, work through the data pipeline interview questions and the data pipeline practice problems, read the Kafka interview questions, and see how the system design round asks for them.