Message Queues, Pub/Sub and Event Streams: How Asynchronous Systems Work

In article 05, Pixly's upload request saved the photo and then "put a message on a queue" so that thumbnails, moderation, feed updates and notifications could happen later. That one sentence hides a lot of machinery. Where does the message live while it waits? What happens if a worker crashes halfway through making a thumbnail? What if the message is delivered twice, or a malformed message fails forever? What if uploads arrive faster than workers can keep up?

This article answers those questions. Queues are one of the most useful tools in system design, and also one of the most frequently misused ("add Kafka") so the goal is to understand what they guarantee, what they do not, and how to choose between a work queue, pub/sub and an event stream.

• The problems queues solve, and the vocabulary: producer, consumer, broker, acknowledgement, consumer group, offset.
• Three models: work queues, publish/subscribe, and event streams.
• The life of a message: delivery, acknowledgement, visibility timeout and redelivery.
• Retries, poison messages and dead-letter queues.
• Delivery guarantees: at-most-once, at-least-once and "effectively once".
• Ordering, partitions and the trade-off with parallelism.
• Backlogs and backpressure, and how to design good workers and messages.
• How to choose, and how each option fails.

A message queue is a durable buffer between parts of a system. A producer writes messages to it; a consumer reads and processes them. The software that stores and delivers messages is the broker.

Queues solve five problems:

1. Decoupling in time. The producer does not wait for the consumer, and the consumer does not have to be running when the message is sent.
2. Load leveling. A burst of 10,000 messages is absorbed by the queue; consumers process them at a steady rate (article 05).
3. Reliable retries. If processing fails, the message is still there to try again, without failing the user's original request.
4. Fan-out. One event ("photo uploaded") can trigger several independent pieces of work.
5. Independent scaling. You scale consumers by adding workers, without touching producers.

Each message is processed by one consumer. Many workers read from the same queue, and the broker hands each message to just one of them. Microsoft calls this the Competing Consumers pattern: multiple concurrent consumers process messages from the same channel, to increase throughput and availability (Competing Consumers pattern).

Producer --> [ m5 m4 m3 m2 m1 ] --> Worker A gets m1                                --> Worker B gets m2                                --> Worker C gets m3

Use for: background jobs where each job should run once, such as "make thumbnails for photo 9001" or "send this email". Examples: Amazon SQS, RabbitMQ queues, Azure Service Bus queues, Google Cloud Tasks.

A producer publishes a message to a topic. Every subscription to that topic gets its own copy. Each subscription may itself be read by several competing workers. Microsoft's Publisher-Subscriber pattern describes this: an application announces events to multiple interested consumers asynchronously, without coupling senders to receivers (Publisher-Subscriber pattern).

                         +--> subscription "thumbnails"   --> thumbnail workersProducer --> topic ------+--> subscription "moderation"   --> moderation workers "photo.uploaded"        +--> subscription "notifications" --> notification workers

Use for: one event that several independent services care about. Adding a new consumer means adding a subscription; the producer does not change. Examples: Amazon SNS with SQS queues behind it, Google Cloud Pub/Sub, Azure Service Bus topics. Google describes Pub/Sub as an asynchronous, scalable messaging service that decouples services producing messages from services processing them (Pub/Sub overview).

An event stream is an append-only, ordered log that the broker keeps for a configured time (hours, days, or longer), whether or not anyone has read it. Consumers do not delete messages; each consumer tracks its own position (an offset) in the log, and can rewind to replay history.

Apache Kafka is the best-known example. Its documentation describes events stored durably in topics, topics split into partitions spread across brokers, events with the same key written to the same partition, and a guarantee that any consumer of a given partition reads its events in exactly the order they were written; events are retained for a configurable period rather than deleted after consumption (Introduction to Kafka).

Topic "orders", partition 0:  [o1][o4][o7][o9] ...   <-- group "billing" at offset 3                                                      <-- group "analytics" at offset 1Topic "orders", partition 1:  [o2][o3][o5][o6][o8]...

Several consumer groups read the same stream independently, each at its own pace; within a group, each partition is read by one consumer at a time, which is how a stream scales out while keeping per-partition order.

Use for: high-throughput event pipelines, multiple consumers that each need the full history, replay (rebuild a search index from events), stream processing, and event-driven architectures (article 27). Examples: Apache Kafka (and managed versions such as Amazon MSK), Amazon Kinesis Data Streams, Azure Event Hubs, Redis Streams.

Here is what happens to one message in a typical work queue:

1. Producer sends message  --> broker stores it durably, replies "stored"2. Consumer receives it     --> broker hides it from other consumers (it is "in flight")3. Consumer processes it4a. Success: consumer ACKNOWLEDGES (deletes) it --> gone for good4b. Crash or error: no acknowledgement arrives     --> after a timeout, the message becomes visible again and is redelivered

The key idea is the acknowledgement. The broker only forgets a message when the consumer says it is done. Amazon SQS implements step 2 with a visibility timeout: a received message stays in the queue but is hidden from other consumers; if it is not deleted before the timeout expires (30 seconds by default, adjustable), it becomes visible again and can be received by another consumer. The SQS documentation also states that, because of its at-least-once delivery model, there is no absolute guarantee a message will not be delivered more than once (Amazon SQS visibility timeout).

In a stream like Kafka, the equivalent of acknowledging is committing the offset: the consumer records "I have processed up to here". If it crashes before committing, it resumes from the last committed offset and processes some messages again.

• Too short: a slow but healthy worker is still processing when the message reappears; a second worker processes it too. Duplicates rise.
• Too long: when a worker really crashes, its message waits a long time before anyone retries it.

For long tasks, a worker can extend the timeout periodically (a heartbeat) while it works, so the message stays hidden only as long as the worker is alive.

Some failures are transient: the email provider timed out, the database was briefly busy. Retrying later works. Others are permanent: the message is malformed, or refers to a photo that was deleted. Retrying forever wastes capacity and can block other messages. Such a message is called a poison message.

The standard answer is a dead-letter queue (DLQ): after a message has been received a set number of times without success, the broker moves it to a separate queue for investigation. SQS, for example, moves a message to the configured DLQ when its receive count exceeds a maximum you set (Using dead-letter queues in Amazon SQS).

Good practice:

• Retry with backoff. Do not retry instantly in a tight loop; wait longer after each failure (article 20 adds random jitter).
• Classify errors. Retry transient ones; send clearly permanent ones to the DLQ immediately.
• Alert on DLQ growth. A DLQ nobody watches is a place where data quietly disappears.
• Make redrive easy. After fixing a bug, move messages from the DLQ back to the main queue.

Most systems choose at-least-once because losing work is usually worse than doing it twice, and then make the consumer idempotent: processing the same message twice has the same effect as processing it once. Some brokers offer "exactly-once" features within their own boundaries (for example, transactions inside Kafka), but as soon as your consumer touches an outside system (sends an email, charges a card, writes to another database) you need idempotency in your own code. Article 20 covers the techniques; the simulation below shows the basic idea.

This Java program models a work queue in memory. Message m2 fails once with a transient error and succeeds on retry. Message m3 is malformed and fails every time, so it ends in the DLQ. Message m1 is delivered twice (as at-least-once delivery allows), and the consumer skips the duplicate because it remembers which ids it has processed.

// File: QueueDemo.javaimport java.util.ArrayDeque;import java.util.ArrayList;import java.util.Deque;import java.util.HashMap;import java.util.HashSet;import java.util.List;import java.util.Map;import java.util.Set;public class QueueDemo {    record Message(String id, String body) {}    static final Deque<Message> queue = new ArrayDeque<>();    static final List<Message> deadLetters = new ArrayList<>();    static final Map<String, Integer> attempts = new HashMap<>();    static final Set<String> processed = new HashSet<>(); // in real systems: durable storage    static final int MAX_ATTEMPTS = 3;    static void handle(Message m) {        if (m.body().equals("corrupt")) {            throw new IllegalArgumentException("cannot parse");                     // permanent        }        if (m.id().equals("m2") && attempts.get("m2") == 1) {            throw new IllegalStateException("timeout calling email provider");    // transient        }        if (!processed.add(m.id())) {            System.out.println("  " + m.id() + " already processed, skipping side effect");            return;        }        System.out.println("  " + m.id() + " processed: " + m.body());    }    public static void main(String[] args) {        queue.add(new Message("m1", "send welcome email to u1"));        queue.add(new Message("m2", "send welcome email to u2"));        queue.add(new Message("m3", "corrupt"));        queue.add(new Message("m1", "send welcome email to u1")); // duplicate delivery        while (!queue.isEmpty()) {            Message m = queue.poll();            int attempt = attempts.merge(m.id(), 1, Integer::sum);            System.out.println("deliver " + m.id() + " (attempt " + attempt + ")");            try {                handle(m);                       // success: acknowledge, message is gone            } catch (RuntimeException e) {                System.out.println("  failed: " + e.getMessage());                if (attempt >= MAX_ATTEMPTS) {                    deadLetters.add(m);                    System.out.println("  moved to dead-letter queue");                } else {                    queue.add(m);                // no acknowledgement: redelivered later                }            }        }        System.out.println("dead letters: " + deadLetters.stream().map(Message::id).toList());    }}

deliver m1 (attempt 1)  m1 processed: send welcome email to u1deliver m2 (attempt 1)  failed: timeout calling email providerdeliver m3 (attempt 1)  failed: cannot parsedeliver m1 (attempt 2)  m1 already processed, skipping side effectdeliver m2 (attempt 2)  m2 processed: send welcome email to u2deliver m3 (attempt 2)  failed: cannot parsedeliver m3 (attempt 3)  failed: cannot parse  moved to dead-letter queuedead letters: [m3]

One simplification matters: the program keeps processed ids in memory. A real consumer must store them durably, and ideally in the same transaction as the side effect, or a crash between "send email" and "record id" still produces a duplicate. Article 20 shows how.

Many queues do not preserve order: with competing consumers, message 2 may finish before message 1. Often that is fine (thumbnails for different photos are independent). Sometimes it is not: "account opened" must be processed before "account closed".

Brokers offer ordering within a group or partition, never across a whole high-throughput system:

• Kafka: order is guaranteed within a partition. Events with the same key (for example account_id) go to the same partition, so all events for one account are read in order (Introduction to Kafka).
• SQS FIFO queues: messages with the same message group id are processed in strict sequence; the next message in a group is not available until the in-flight one is deleted or its visibility timeout expires (Amazon SQS visibility timeout).

The trade-off is direct: order within a key means only one consumer works on that key at a time. Choose the narrowest key that needs ordering (per account, per conversation, per order), not a global order, or you lose parallelism. Also note that retries can reorder messages: if message 1 fails and is retried while message 2 succeeds, 2 is processed first unless the broker blocks the group.

When strict ordering is too expensive, design consumers to tolerate reordering: include a version or timestamp in each message and ignore updates older than what you have already applied.

A queue hides a mismatch between arrival rate and processing rate, but only for a while. If producers send 2,000 messages per second and consumers process 1,500, the backlog grows by 500 per second: 1.8 million messages after an hour. Messages wait longer and longer, and the work may no longer be useful when it finally happens (a "your order is out for delivery" notification three hours late).

Watch two metrics above all:

• Queue depth (or consumer lag in streams): how many messages are waiting.
• Age of the oldest message: how long the oldest one has waited. This one tells you the user impact.

Responses:

• Scale consumers on lag or age, as far as downstream systems can take it.
• Protect downstream systems: more workers can overload the database they write to. Limit concurrency.
• Prioritise: separate queues for urgent and bulk work (for example, password-reset emails versus newsletters).
• Drop or expire stale work when it is no longer useful, instead of processing it late.
• Push back on producers (backpressure): if the queue is full or too old, reject or slow new work at the edge rather than accept work you cannot do.

The Amazon Builders' Library has an article on exactly this problem, describing how backlogs can grow until they are effectively insurmountable and discussing techniques such as separating workloads and dropping messages that are too old to matter (Avoiding insurmountable queue backlogs).

• Idempotent. Every handler must be safe to run twice.
• Stateless. Any worker can process any message (article 08); state lives in shared stores.
• Bounded. Each message is a small, bounded unit of work. Split "process all 2 million users" into one message per user or per batch.
• Timeouts on every call, and heartbeats or timeout extension for long tasks.
• Graceful shutdown: stop receiving, finish or release in-flight messages, then exit, so deployments do not cause mass redelivery.
• Concurrency limits that protect the database, provider APIs and rate limits.

{  "message_id": "evt_7b2f9c",  "type": "photo.uploaded",  "version": 2,  "occurred_at": "2026-10-04T18:22:07Z",  "photo_id": 9001,  "owner_id": 42,  "object_key": "photos/u42/p9001/original.jpg"}

• A unique id for deduplication.
• A type and version, so the format can evolve without breaking old consumers.
• References, not large payloads. Put the photo in object storage and send its key. (Passing a reference to a large stored payload is sometimes called the claim-check pattern.) Brokers have message size limits, and large messages slow everything.
• No secrets, and as little personal data as necessary: messages are copied, retained and logged.

Pixly needs four things to happen after an upload: thumbnails, moderation, feed updates and notifications. Two reasonable designs:

Option A: pub/sub topic with a queue per consumer.

API server --> topic "photo.uploaded" --+--> queue "thumbnails"    --> thumbnail workers                                        +--> queue "moderation"    --> moderation workers                                        +--> queue "feed-fanout"   --> feed workers                                        +--> queue "notifications" --> notification workers                                              (each queue has its own DLQ)

Each consumer scales and fails independently. A broken notification worker fills its own DLQ without affecting thumbnails. Simple to operate with managed services.

Option B: an event stream, topic photo-events partitioned by photo_id, with one consumer group per service.

Adds replay: if Pixly later builds a search service, it can read the topic from the beginning (within retention) to index existing photos. Ordering per photo is preserved (useful if "uploaded" and "deleted" events for the same photo must be applied in order). Costs more operational effort.

Choice: start with Option A, unless the requirements include replay or many future consumers of the full history. That is a defensible, requirement-driven answer, and you can say exactly what would make you switch.

• Access control per queue or topic: which services may publish and which may consume.
• Encryption in transit and at rest; minimise personal data in messages, because retention keeps it around.
• Metrics: queue depth or lag, age of the oldest message, processing rate, error rate, DLQ size, end-to-end latency from event to effect.
• Tracing: pass a trace id in message metadata so a user action can be followed from the API through each consumer (article 25).

• Using a queue where the user needs the answer now.
• Acknowledging before processing, then losing work on crashes.
• Assuming "exactly-once" and writing non-idempotent consumers.
• No dead-letter queue, or one that nobody monitors.
• Asking for global ordering when per-key ordering is enough.
• Putting large payloads in messages instead of references.
• Scaling consumers without considering the database they all write to.
• Choosing Kafka for a handful of background jobs.

• Introduce a queue with its reason: "thumbnails are slow and can fail; a queue makes uploads fast and retries failures".
• Say "at-least-once, so consumers are idempotent", and how (unique constraint, processed-ids table, idempotent writes).
• Mention a DLQ and monitoring of lag or oldest-message age.
• If ordering matters, name the partition or group key.
• Justify pub/sub or a stream only when there are several consumers or a need for replay.

• Next: Relational Databases and Data Modeling for System Design starts Module 5, the data layer every one of these consumers writes to.
• Going deeper: idempotency and delivery guarantees in article 20, the outbox pattern in article 21, and event-driven architecture in article 27. The notification system (article 33) is a full design built on these ideas.