Distributed Systems

Message Queues Explained: Kafka, RabbitMQ, and When to Use Them

A queue sits between two services so the sender doesn't have to wait for the receiver to finish. That sounds simple until you pick a specific queue and its retry, ordering, and delivery guarantees start shaping your whole architecture.

Published July 6, 2026

The pitch for a message queue is always the same: service A produces work, service B consumes it, and if B is slow or down, A doesn't have to know or care. What differs enormously between queue systems is what happens to a message between being produced and being consumed, and that's where most production incidents originate.

Two different mental models

RabbitMQ is a traditional message broker built on AMQP. A producer publishes to an exchange, the exchange routes the message to one or more queues based on bindings, and a consumer pulls from the queue and acknowledges it. Once acknowledged, the message is gone. This is the model most people picture when they hear "queue": a to-do list that shrinks as work gets done.

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
channel.queue_declare(queue='orders', durable=True)

channel.basic_publish(
    exchange='',
    routing_key='orders',
    body='{"order_id": 4821, "amount": 49.99}',
    properties=pika.BasicProperties(delivery_mode=2)
)
connection.close()

Kafka is not a queue in that sense at all. It's a distributed, append-only log split into partitions. Producers write to the end of a partition; consumers track their own offset and read forward from wherever they left off. Nothing is deleted when a consumer reads it — messages stick around until a retention policy (time-based or size-based) expires them, regardless of whether anyone consumed them yet. The log.retention.hours and log.retention.bytes broker settings, documented in the official Kafka broker configuration reference, control how long that window stays open.

from confluent_kafka import Producer

producer = Producer({'bootstrap.servers': 'localhost:9092'})

def delivery_report(err, msg):
    if err is not None:
        print(f'delivery failed: {err}')
    else:
        print(f'delivered to {msg.topic()} partition {msg.partition()}')

producer.produce('orders', key='4821', value='{"amount": 49.99}', callback=delivery_report)
producer.flush()

That difference has real consequences. Because Kafka keeps the log, you can replay it: add a new analytics consumer next month and it can read every event since the beginning of retention, not just new ones. RabbitMQ has no equivalent — once a message is acked, it's simply not there anymore. If you need audit trails or the ability to rebuild a downstream system's state from history, that alone can decide the choice.

Ordering and partitioning

Kafka guarantees order only within a single partition, not across a topic. If you need all events for a given customer processed in order, you key the message on customer ID so it always lands on the same partition — the producer example above keys on order_id for the same reason. Get the key wrong and you'll get intermittent out-of-order bugs that are miserable to reproduce because they depend on partition assignment and consumer timing.

RabbitMQ queues are ordered by default (FIFO per queue), but that guarantee breaks down the moment you add multiple consumers competing for the same queue, since messages get round-robined and a slow consumer can finish after a faster one that grabbed a later message.

Delivery guarantees, and why "exactly once" is mostly marketing

Both systems default to at-least-once delivery: if a consumer crashes after processing a message but before acknowledging it, the message gets redelivered. Your consumer logic has to be idempotent — processing the same order confirmation twice shouldn't double-charge anyone. Kafka has transactional producers that get you close to exactly-once within Kafka itself, but the moment your consumer writes to an external database or calls another API, you're back to needing idempotency keys or upserts, because there's no way to make the Kafka commit and the external side effect atomic across two separate systems.

Consumer groups vs competing consumers

In RabbitMQ, multiple consumers on one queue simply compete: each message goes to exactly one of them. In Kafka, a consumer group achieves a similar effect but at the partition level — each partition is assigned to exactly one consumer within the group, so parallelism is capped by partition count. Add an 11th consumer to a topic with 10 partitions and it sits idle. This is a common surprise for teams migrating from RabbitMQ, where scaling out consumers is a non-issue.

When a queue is the wrong answer

Not every async problem needs infrastructure this heavy. If the workload is a background job triggered by a single web request — resize an image, send a welcome email — a lighter task queue like Sidekiq, Celery, or even a database-backed job table is easier to operate and debug. Reach for Kafka when you have multiple independent consumers that need the same event stream, need replay, or are building an event-sourced system. Reach for RabbitMQ when you want flexible routing (topic exchanges, dead-letter queues, priority queues) and don't need long-term retention. Reach for neither when a synchronous call would be simpler and the extra latency of an async hop isn't buying you anything.

One operational note that catches teams off guard: both systems need capacity planning around consumer lag, not just producer throughput. A queue that fills faster than it drains looks fine in a load test that only runs for five minutes and becomes an outage on the day traffic sustains for five hours. Track consumer lag as its own metric, not just queue depth at a point in time.