Message Queues — Basics

Message Queues — Basics (RabbitMQ, SQS, Redis Streams, NATS)

Kafka covered separately. This file = traditional brokers + cloud queues.

Why a queue at all

• Decouple producer from consumer (different scale, language, deploy).
• Smooth bursty traffic (level-load).
• Async processing (long jobs, side effects).
• Reliable delivery + retries on transient failure.
• Fan-out to many consumers.
• Ordering / FIFO when needed.

RabbitMQ

AMQP 0.9.1 broker (also MQTT, STOMP via plugins). Mature, flexible routing.

Concepts

• Producer publishes to exchange, not queue.
• Exchange routes to one or many queues based on bindings.
• Exchange types:
  • direct — exact routing key match.
  • topic — wildcard match (order.*.created).
  • fanout — broadcast to all bound queues.
  • headers — match on message headers.
• Consumer subscribes to a queue.
• Acknowledgment — manual ack required for at-least-once. Reject → nack → requeue / DLQ.

Reliability features

• Persistent messages + durable queues survive restart.
• Publisher confirms — broker ack to producer.
• Dead-letter exchange — auto-route rejected/timeout messages.
• Mirrored / quorum queues for HA (quorum = Raft-based, recommended).
• Lazy queues — keep messages on disk for huge backlogs.

Fits

• Complex routing (microservice events with topic patterns).
• Per-message ack and redelivery semantics.
• Heterogeneous consumers, RPC-over-MQ patterns.
• Smaller throughput than Kafka but richer per-message control.

Amazon SQS

Fully managed. Two queue types:

Standard

• At-least-once, best-effort ordering, virtually unlimited throughput.
• Low latency, easy to scale.

FIFO

• Exactly-once processing within a deduplication window (default 5 min, via MessageDeduplicationId).
• Ordered within a MessageGroupId.
• 300 TPS (or 3000 with batching) per group.

Concepts

• Message visibility timeout — consumer “rents” a message; if not deleted in window, re-appears.
• Long polling (WaitTimeSeconds=20) — reduces empty calls.
• DLQ via RedrivePolicy after N receives.
• Max retention 14 days. Max msg size 256KB (or via S3 extended client for bigger).

Fits

• AWS-native, want zero ops.
• Simple producer-consumer queue.
• Decoupling Lambda invocations.
• Cron-like with EventBridge / SQS scheduled.

Redis Streams

Persistent log inside Redis (not pure pub/sub). Lighter than Kafka, broker-grade than SQS in some ways.

Concepts

• XADD to append. Each message gets an id (<ms>-<seq>).
• XREAD to read; consumer groups via XREADGROUP.
• XACK to ack. XPENDING to inspect unacked.
• XCLAIM to reassign stuck messages.
• Replay possible (read from any id).

Fits

• You already run Redis.
• Microsecond-latency queues.
• Bounded backlog (limited by RAM mostly).
• Lightweight job queue (BullMQ, Sidekiq use Redis).

NATS / JetStream

CNCF lightweight messaging. Subjects (NATS core pub/sub) + JetStream (durable streams + consumers).

Fits

• IoT / edge — small footprint.
• Polyglot mixed pub/sub + queue + KV + Object Store.
• Lower-than-Kafka latency, simpler ops.

Apache Pulsar

Tiered storage (broker + bookkeeper segments). Multi-tenant, geo-replication built-in.

Fits

• Multi-tenant SaaS.
• Want Kafka-like at huge scale with separated storage.

Comparison cheat-sheet

	Kafka	RabbitMQ	SQS	Redis Streams	NATS JS
Ordering	per partition	per queue	per group (FIFO)	per stream	per subject
Delivery	at-least-once (EoS w/ tx)	at-least-once + per-msg ack	at-least-once	at-least-once	at-least-once
Persistence	replicated log	durable+mirrored	managed	RAM (+ AOF)	replicated
Throughput	very high	high	very high	very high	very high
Latency	low	low	medium (10-100ms)	very low	very low
Replay	yes	no (by design)	no	yes	yes
Routing	partition by key	direct/topic/fanout	none built-in	none	subject hierarchy
Mgmt overhead	high	medium	none	low	medium

Common patterns

• Work queue: producer drops jobs, N workers compete.
• Pub/sub: producer broadcasts events to multiple subscribers (fanout exchange / multiple SQS queues / Kafka consumer groups / Redis pub/sub).
• Request-reply: send request with replyTo queue + correlationId.
• DLQ: reject after N attempts; manual replay.
• Delayed delivery: SQS native (15min max), RabbitMQ delayed-msg plugin, Redis ZSET with scheduler.
• Priority: RabbitMQ priority queues, separate streams per priority.

When to use what

• Need huge throughput, replay, log semantics: Kafka.
• Rich routing, per-message ack, RPC-over-MQ: RabbitMQ.
• AWS, simple, low ops: SQS.
• Already on Redis, low latency, simple: Redis Streams + BullMQ.
• Edge / IoT / polyglot: NATS JetStream.
• Geo-replicated multi-tenant: Pulsar.