Intermediate16 min

Messaging & Queues

Decouple producers from consumers with a buffer that absorbs bursts and survives failures.

By the end of this lesson, choose queue vs log semantics and reason about delivery guarantees.

How deep?

How the pieces actually move.

A message broker sits between producers and consumers so they don't call each other directly. The producer hands off a message and moves on; the consumer processes it when ready. This decoupling absorbs traffic bursts, survives consumer downtime (messages wait), and lets you scale the two sides independently.

First, the whole system

First, see the whole system. Then we’ll open it up.

Queue vs log

A producer sends to a broker. A queue delivers each message to one competing consumer; a log (e.g., Kafka) retains messages and lets consumer groups read at their own offsets and replay.

flowchart LR
  prod[Producer] --> broker[(Broker)]
  broker -->|queue: each msg to one consumer| c1[Consumer]
  broker -->|log: replayable, per-partition offset| c2[Consumer group]

Two families: queues (RabbitMQ, SQS) deliver each message to one consumer and typically delete it after ack — great for work distribution. Logs (Kafka, Redpanda) retain an ordered, replayable stream partitioned across consumers — great for event streaming, replay, and multiple independent readers. Delivery is practically at-least-once, so consumers must be idempotent; ordering is only guaranteed within a partition.

Abstraction leak

This is where the abstraction starts leaking.

'Exactly-once delivery' is marketing. Under the hood it's at-least-once delivery plus idempotent processing (or transactional offsets). If your consumer isn't idempotent, redelivery will eventually double-process a message.

Queue backlog & poison messages

Trigger: Consumers can't keep up, or a message that always fails blocks its partition.
Symptom: Growing lag and rising end-to-end latency; a stuck poison message stalls progress.
Blast radius: Every consumer of the backed-up stream.
Mitigation: Autoscale consumers, backpressure, dead-letter queues, retry limits.

Deep dive: Queue Backlog

How it connects

When would you choose a log (Kafka) over a queue?

When you need retention, replay, ordering within partitions, or multiple independent consumers reading the same stream at their own offsets.