Message Queues Gone Wrong: When to Reach for RabbitMQ and When Kafka Will Eat Your Lunch

Every system eventually needs to talk to itself. Order here, process there, notify someone, retry on failure, repeat. Messaging is how you stitch services together without hardcoding them into each other.

Two names come up every time: RabbitMQ and Kafka.

Both move messages. Both decouple producers from consumers. Both are “just queues” until you’ve built something on top of them and realised they’re not.

This is how to pick between them. And more importantly, how to stop picking the wrong one out of habit.

The Core Difference Nobody Explains Clearly

RabbitMQ is a queue. Kafka is a log.

That one sentence explains everything, but it takes a while to sink in.

RabbitMQ routes messages to queues. A consumer pulls a message, processes it, acknowledges it. The message is gone. If the consumer is down, the message sits waiting. If the queue fills up, producers back pressure. This is the traditional model.

Kafka appends messages to a partitioned, immutable log. Consumers read at their own pace, keeping track of their position (offset). Messages stick around for as long as you configure — hours, days, years. The log doesn’t care if anyone is reading it.

That difference sounds simple. In practice it’s the entire architecture debate.

RabbitMQ: The Smart Router

RabbitMQ is brilliant when you need routing intelligence.

A message arrives at an exchange. The exchange decides where it goes based on rules: direct, topic, fanout, headers. One message can fan out to five queues or be routed to exactly one. Consumers pull from queues they care about.

This is powerful when your system has branching logic.

Think about an order processing system. An order is placed. You need to:

• Send a confirmation email
• Update the warehouse inventory
• Trigger a payment capture
• Log the transaction
• Notify the fraud check system

Different consumers. Different destinations. RabbitMQ handles this naturally — publish once, five queues handle it, each at their own pace. If the fraud check service is down, the other four keep running.

Use RabbitMQ when:

• You need flexible routing (pub/sub, topic routing, request-reply patterns)
• Message delivery guarantees matter more than throughput
• You need RPC-style request-reply over a queue
• Your consumers expect a traditional pull-from-queue interaction model
• You want per-message TTL, dead-letter queues, and fine-grained flow control out of the box

Kafka: The Immutable Event Spine

Kafka is different. There are no “queues” in the RabbitMQ sense. There are topics split into partitions. Messages land in partitions in the order they were written. Consumers in a group read partitions — each partition goes to one consumer in the group. If a consumer falls behind, it just catches up when it can.

The log never forgets.

This changes what you can build on top of it. With RabbitMQ, once a message is consumed and acknowledged, it’s gone. You can replay it. Kafka lets you replay from any point in the log because the data is still there.

Use Kafka when:

• You need event streaming — a continuous flow of events rather than discrete request-reply exchanges
• You need replay capability — consumers that process at their own speed and can restart from a known offset
• You need very high throughput — Kafka handles millions of messages per second per topic
• You need durability by default — messages survive broker restarts and consumer downtime
• You’re building event-driven architectures, audit logs, or change data capture (CDC) pipelines

The Real-World Problem: Using Kafka Like RabbitMQ

Here’s where I see teams go wrong. They hear “Kafka is fast” and use it everywhere. They build a RabbitMQ-style architecture on top of Kafka: short-lived queues, request-reply patterns, transient messages that get processed and forgotten.

Kafka hates this.

Kafka is optimised for long-lived consumers reading long-lived topics. Retaining data “just in case” is the default behaviour — which means disk usage grows, consumer group rebalances get expensive, and monitoring becomes a nightmare when you have 400 topics with 10 messages each.

If you don’t need replay, don’t need streaming semantics, and just want to route a message from A to B reliably — use RabbitMQ. It’s simpler to operate, has better operational tooling for request-reply, and doesn’t make you think about partitions and offsets for a basic messaging pattern.

The Other Way Around: RabbitMQ at High Throughput

Kafka isn’t always the answer either.

If your workload is bursty — spikes of 10,000 messages then silence for an hour — Kafka’s persistent log is overkill. RabbitMQ with smart prefetch settings handles this cleanly without you needing to manage partition counts and retention policies.

If you need sophisticated routing — message A goes to Europe, message B goes to APAC, message C triggers a complex workflow — RabbitMQ’s exchange types handle this in configuration. Kafka makes you build it in application logic.

What “Reliability” Actually Means

Both systems offer delivery guarantees, but they work differently.

RabbitMQ: at-least-once by default. With publisher confirms and consumer acknowledgements, you get strong guarantees but the state is in-memory and queue-based. Messages that were acknowledged and the consumer died? Gone. Unless you configured dead-lettering explicitly.

Kafka: durability is structural. A message isn’t “delivered” until it’s written to all in-sync replicas. Consumers can be offline for days and pick up where they left off. The guarantee is baked into the log architecture, not configured on top.

But here’s the trap: “durable by default” doesn’t mean “the right default.” If you don’t need the durability, you’re paying for it in operational complexity and disk usage you don’t need.

The Decision Framework

Instead of reaching for the technology you know best, run through this:

Is your message a command or an event?

• Command (do this now): RabbitMQ
• Event (this happened): Kafka

Does the receiver need to process it immediately, or at their own pace?

• Immediately or with retry logic: RabbitMQ
• Own pace with replay capability: Kafka

Do you need to fan-out one message to many consumers?

• Variable routing, different queues per consumer: RabbitMQ
• Multiple consumer groups reading the same topic: Kafka

Will you need to replay historical messages?

• No: RabbitMQ
• Yes: Kafka

What’s your peak throughput?

• Under 100k messages/second (most systems): RabbitMQ is fine
• 100k+ sustained: Start evaluating Kafka

How long does a message need to live?

• Process and forget (minutes to hours): RabbitMQ
• Need to preserve for audit/compliance (days to years): Kafka

The Short Version

There is no “best” messaging system. There is the right system for the problem you have right now.

RabbitMQ is the smart courier — it routes, retries, and delivers with good operational defaults for most service architectures.

Kafka is the event ledger — it records everything once and lets consumers process at their own pace, forever.

If you’re integrating a few services with standard request-reply patterns, RabbitMQ is almost always the simpler starting point.

If you’re building an event-driven platform where services produce events and downstream services consume them independently — including retroactively — use Kafka.

And if you find yourself adding RabbitMQ-style acknowledgements and routing logic on top of Kafka because “we already have it running,” stop. You’ve chosen the expensive option to save a few days of setup. The setup cost is the last thing you should optimise.

The honest answer is usually: start with RabbitMQ. Move to Kafka when you have a specific reason — not because it’s the new default.

Pick the one that fits your architecture. Not the one you read about last.