Message Queues and Streaming

Why Queues: Decoupling and Load Leveling

A message queue sits between producers and consumers so that neither needs the other to be up, fast, or scaled the same way. Producers write and move on; consumers process at their own pace. This buys temporal decoupling (the email service can be down for 10 minutes and no orders are lost), load leveling (a spike of 50,000 requests per second gets absorbed by the queue and drained at the consumers' sustainable 5,000 per second), and independent scaling of the two sides.

The two classic models are point-to-point and publish/subscribe. In point-to-point, each message is consumed by exactly one worker from a shared queue, which is the model for task distribution: resize this image, send this email, charge this card. In pub/sub, each message is delivered to every subscriber, which is the model for event broadcast: OrderPlaced fans out to inventory, analytics, and notifications, each with its own subscription. Kafka unifies both with consumer groups: within one group, partitions are divided among members (point-to-point semantics for scaling), while multiple independent groups each get the full stream (pub/sub semantics).

Queues are also the standard answer to write spikes in system design interviews: put a queue in front of the slow or expensive operation (video transcoding, third-party API calls, fan-out on write) so the user-facing path stays fast and the backlog is observable.

Kafka vs RabbitMQ vs SQS

Kafka is a distributed, partitioned, replicated commit log. Messages are appended to partitions and retained for a configured period (days or forever), regardless of consumption; consumers track their own offsets and can rewind and replay. This makes Kafka a streaming platform, not just a queue: the same topic feeds real-time consumers, batch jobs, and a new service backfilling from history. It was built at LinkedIn, where it grew to handle on the order of 7 trillion messages per day, and sequential disk I/O plus zero-copy transfer let a modest cluster sustain millions of messages per second. Choose Kafka for event streams, high throughput, replay, and multiple independent consumers of the same data.

RabbitMQ is a traditional smart broker implementing AMQP. Messages are routed through exchanges (direct, topic, fanout, headers) into queues, pushed to consumers, and deleted on acknowledgment. It offers rich routing, per-message TTLs, priorities, and delayed delivery, with typical throughput in the tens of thousands of messages per second per node, and generally lower single-message latency at low volume than Kafka. Choose RabbitMQ for task queues and complex routing where you do not need replay or log retention.

SQS is AWS's fully managed queue: effectively unlimited throughput on standard queues, no brokers to operate, pay per request. Standard queues give at-least-once delivery with best-effort ordering; FIFO queues give exactly-once processing and strict order within a message group, capped at 300 transactions per second per API action (3,000 with batching). Its visibility timeout model (a consumed message becomes invisible, then reappears if not deleted in time) is the mechanism behind its at-least-once behavior. Choose SQS when you are on AWS and want zero operational burden; pair with SNS for fan-out.

Delivery Guarantees: At-Most-Once, At-Least-Once, Exactly-Once

At-most-once means fire and forget: the producer does not wait for acknowledgment, or the consumer acks before processing, so a crash loses the message but nothing is ever duplicated. It is acceptable for metrics and logs where a small loss rate is tolerable. At-least-once means the message is retried until acknowledged after processing, so nothing is lost but duplicates occur: the consumer processes, crashes before acking, and the message is redelivered. This is the practical default for almost all real systems.

Because at-least-once is the default, consumers must be idempotent: processing the same message twice must have the same effect as once. Standard techniques are a deduplication table keyed by message ID (insert the ID in the same transaction as the side effect), natural idempotency (set status to shipped is safe to repeat, increment balance is not), or idempotency keys passed to downstream APIs the way Stripe accepts them on charge creation.

Exactly-once is best understood as exactly-once processing effect, not exactly-once delivery, which is impossible over an unreliable network in the general case (see the Two Generals problem). Kafka gets close within its own ecosystem: idempotent producers (broker deduplicates by producer ID and sequence number) plus transactions that atomically write output messages and commit consumer offsets, giving end-to-end exactly-once for Kafka-in, Kafka-out stream processing. The moment a side effect leaves Kafka (an HTTP call, a database write outside the transaction), you are back to at-least-once plus idempotency. Saying exactly that sentence in an interview is a strong signal.

Ordering, Consumer Groups, and Partitioning

Global ordering across a distributed queue does not scale, so systems offer ordering per partition or per message group instead. In Kafka, messages with the same key (say, user ID or order ID) hash to the same partition, and each partition is consumed by exactly one consumer within a group, so all events for a given order are processed in order. Choosing the partition key is a real design decision: it must match the entity whose order matters, and it must distribute well, since a hot key (one celebrity user, one huge tenant) creates a hot partition that caps throughput.

Consumer groups are Kafka's unit of horizontal scaling. A topic with 32 partitions supports up to 32 active consumers in one group; adding a 33rd does nothing, which means partition count sets your parallelism ceiling and is expensive to change later, so it is typically overprovisioned (32 or 64 partitions for a topic that needs 8 consumers today). When consumers join or leave, the group rebalances, which briefly pauses consumption, and a consumer that is slow to heartbeat gets kicked out and its partitions reassigned, causing duplicate processing of in-flight messages, another reason idempotency is mandatory.

RabbitMQ and SQS standard queues do not guarantee order once you have multiple consumers or redeliveries; SQS FIFO restores order per message group ID at the cost of throughput. A useful rule: require ordering only where the domain truly needs it, scope it to the narrowest key possible, and design consumers to tolerate reordering everywhere else.

Dead Letter Queues and Backpressure

A poison message that always fails (malformed payload, bug in a handler) will be redelivered forever under at-least-once semantics, blocking a FIFO queue entirely or wasting capacity on a standard one. The fix is a dead letter queue: after N failed attempts (SQS maxReceiveCount, RabbitMQ x-dead-letter-exchange, Kafka usually via an application-level retry topic chain like retry-5m, retry-1h, then DLQ), the message is shunted aside for inspection. A DLQ must be monitored and alerted on; an unmonitored DLQ is just silent data loss with extra steps. Design the operational loop: alert on DLQ depth, inspect, fix the bug, redrive messages back to the main queue.

Backpressure is what happens when producers outrun consumers. In broker-based systems the queue absorbs the difference for a while, so the key metric is consumer lag (Kafka: offset lag per partition; SQS: ApproximateNumberOfMessagesVisible plus oldest message age). Growing lag means you must scale consumers, shed load, or slow producers. Kafka pushes backpressure naturally because consumers pull at their own rate; push-based systems like RabbitMQ use prefetch limits and publisher flow control, and brokers protect themselves with retention limits or max queue length, after which they drop or refuse messages.

Sizing sanity check for interviews: if producers emit 10,000 messages per second and each consumer handles 500 per second, you need at least 20 consumers, so at least 20 partitions, plus headroom to drain backlog after an outage (draining a 1-hour outage backlog at 1.5x capacity takes 2 more hours). Walking through that arithmetic unprompted is exactly what senior candidates do.

Tradeoffs