Design a Notification System: Email, SMS and Push at Scale

Every product eventually sends messages to its users: a one-time password at login, "your order has shipped", "someone commented on your photo", a weekly digest, a festival-sale announcement. At first each team calls an email or SMS provider directly from its own code. Then the problems arrive: a provider outage breaks checkout, a marketing campaign delays login codes by twenty minutes, users receive the same message three times, nobody can tell whether a message was delivered, and someone in legal asks how unsubscribes are honoured across six codebases.

The answer is a shared notification system: one platform that accepts "notify this user about this event" from any service and takes responsibility for preferences, templates, channels, providers, retries and tracking. It is one of the most common Level 2 interview questions because it combines queues, workers, idempotency, third-party integration and prioritisation.

• How to scope a multi-channel notification platform.
• How to estimate normal load and campaign spikes.
• An asynchronous API with idempotency, and the data model behind it.
• Priority and channel separation, workers, provider rate limits and failover.
• Retries, dead-letter queues and deduplication at several layers.
• Preferences, quiet hours, frequency caps and templates.
• Campaign fan-out, delivery tracking, failures, security and observability.

Build a platform service that other internal services call to notify users. It delivers through mobile push, SMS, email and an in-app inbox, using external providers for push, SMS and email. It must handle both transactional notifications (one-time passwords, order updates, security alerts) and marketing notifications (campaigns to millions of users).

Functional requirements

1. Accept a notification request (recipient user, notification type, data, optional schedule) from internal services.
2. Choose channels from the type and the user's preferences; render templates in the user's language.
3. Deliver through providers; record and expose delivery status.
4. Support campaigns targeting a segment of users.
5. Maintain an in-app inbox per user.

Out of scope: the campaign-authoring user interface, A/B testing of content, rich analytics of open rates beyond basic counts.

Non-functional requirements

50 million DAUPer active user per day: 5 push, 0.3 email, 0.05 SMS (mostly OTPs), 2 in-appPush:   250 million/day -> about 2,900/s averageEmail:   15 million/day -> about 175/sSMS:    2.5 million/day -> about 30/sIn-app: 100 million/day -> about 1,160/sTotal about 370 million notifications/day, about 4,300/s; peak factor 3 -> about 13,000/sCampaign spike: one push campaign to 20 million users within 10 minutes-> 20,000,000 / 600 s = about 33,000 pushes/s on top of normal traffic

Storage: about 370 million notification records a day at roughly 500 bytes (including delivery rows) is about 185 GB a day. Kept for 90 days, that is about 17 TB: a partitioned, time-ordered store (article 16), not one relational table.

Provider limits: external providers impose their own rate limits and quotas per account (a fact of life, though the numbers vary by provider and contract). Our design must never exceed them, so throughput to each provider is something we control.

Conclusions: steady load is moderate; campaign spikes and provider limits drive the design, together with the requirement that OTPs are never delayed by bulk traffic.

POST /v1/notificationsIdempotency-Key: order-77-shipped                 (chosen by the calling service){  "recipient_user_id": 42,  "type": "order_shipped",                         (maps to a template and category)  "priority": "transactional",                     (or "marketing")  "data": {"order_id": "ord_77", "eta": "2026-10-07"},  "send_at": null                                   (or a timestamp)}202 Accepted{ "notification_id": "ntf_9c1a", "status": "accepted" }Errors: 400 unknown type or missing data; 403 caller not allowed to send this type; 429 caller over quota

POST /v1/campaigns                       { "segment_id": "...", "type": "diwali_sale", "send_at": "..." } -> 202GET  /v1/notifications/{id}              -> status per channel (accepted, sent, delivered, failed, suppressed)GET  /v1/users/me/inbox?cursor=...       -> in-app notifications, newest first (cursor pagination)PUT  /v1/users/me/notification-preferences  { "marketing": {"email": false, "push": true}, "quiet_hours": "22:00-08:00" }POST /v1/users/me/devices                { "platform": "android", "push_token": "..." }   (register a device)

Key decisions:

• 202 Accepted: delivery happens later; the caller does not wait for providers (article 05).
• The idempotency key is mandatory for transactional types, scoped per calling service, so a caller's retry never creates a second notification (article 20).
• The caller names a type, not channels. The platform decides channels from type rules and preferences, so channel logic lives in one place.

notifications      (notification_id PK, sender_service, idempotency_key, recipient_user_id,                    type, priority, data, send_at, status, created_at)                   UNIQUE (sender_service, idempotency_key)deliveries         (notification_id, channel, status, provider, provider_message_id,                    attempts, last_error, updated_at)                   PRIMARY KEY (notification_id, channel)          -- at most one delivery per channelinbox_items        (user_id, created_at, notification_id, title, body, read_at)                   partition key user_id, sorted by created_at     -- "my inbox, newest first"preferences        (user_id, category, channel, enabled)  + quiet_hours, locale, timezone per userdevices            (user_id, device_id, platform, push_token, last_seen_at)templates          (type, channel, locale, version, subject, body)

• Uniqueness on (sender_service, idempotency_key) deduplicates requests; the primary key on (notification_id, channel) guarantees one delivery record per channel.
• notifications and deliveries are written at thousands per second and kept for 90 days: partition by time (daily partitions you can drop) or use a wide-column store partitioned by notification_id.
• inbox_items is read by user, newest first: partition by user_id, cluster by time (article 17).
• Preferences, devices and templates are small, read constantly, and cached.

Calling service --> Notification API --> database --> call provider synchronously --> respond

The API saves the request, renders the template, calls the email or SMS provider, and responds. It works in a demo. In production:

• the caller waits for the provider (seconds), and a provider outage fails the caller's request;
• a campaign of 20 million calls the provider 20 million times inline;
• retries happen in the request path, multiplying load;
• nothing stops a campaign from delaying OTPs.

1. Coupling to providers: slow or failed providers stall every caller.
2. Spikes: campaigns create bursts far above normal load, and above provider limits.
3. Priority inversion: bulk messages delay urgent ones in shared queues.
4. Duplicates: retries by callers, redelivery by queues and timeouts with providers all risk sending twice.
5. Per-message lookups: preferences, devices and templates for every message.

The API validates the request, enforces the caller's quota, inserts the notification (the unique idempotency constraint makes retries return the original notification_id), writes an outbox event, and returns 202. Everything else is asynchronous (article 21's outbox guarantees the event is published if the row committed).

REQUEST PATHCalling service --> Notification API --> DB (notification + outbox) --> 202ASYNC PATHOutbox relay --> [intake queue: transactional] --\             --> [intake queue: marketing]     ----> Routing workers:                                                       - load preferences, quiet hours, frequency caps (cached)                                                       - choose channels; render templates per locale                                                       - create delivery rows (one per channel)                                                       - enqueue per channel and priority   [push-transactional] [push-marketing] [sms-transactional] [email-transactional] [email-marketing] [inbox]            |                 |                  |                    |                  |             |       Push senders      Push senders       SMS senders         Email senders      Email senders   Inbox writer       (token bucket per provider; circuit breakers; retries with backoff; DLQ per queue)            |                                    |                    |     Push providers (FCM, APNs)           SMS providers A, B     Email providers A, B            |   Delivery receipts and webhooks --> Status workers --> deliveries table

• Routing workers do the per-user work (preferences, channel choice, templates) once, then hand off to channel queues.
• Channel queues are split by priority. Transactional queues have dedicated workers and reserved provider capacity; marketing queues use what is left. Microsoft's Priority Queue pattern describes this approach: prioritise requests so higher-priority messages are received and processed sooner, for example with separate queues and consumer pools (Priority Queue pattern).
• Queue age, not only depth, matters: the Amazon Builders' Library describes how backlogs can grow until they are effectively insurmountable, and recommends separating workloads and dropping work that is too old to be useful (Avoiding insurmountable queue backlogs); expired marketing sends are dropped rather than delivered hours late.
• Sender workers are competing consumers (article 11; Competing Consumers pattern) that scale with queue depth.

• Push: Android and iOS devices receive push messages through platform services: Firebase Cloud Messaging and the Apple Push Notification service, documented by Google and Apple (Firebase Cloud Messaging, User Notifications). The sender needs each device's push token, registered by the app. When a provider reports a token as invalid, delete it.
• SMS and email: external providers accept messages over APIs and report delivery later through callbacks (webhooks) or polling.
• In-app: written directly to inbox_items; the app reads it with cursor pagination, and an open app can receive a live update over an existing real-time connection (article 05).

Each sender pool uses a token bucket per provider account (article 31), shared across workers (in Redis), set slightly below the provider's limit. Transactional traffic gets a reserved share of each bucket. When a provider returns 429, honour its Retry-After and slow down. This keeps campaigns from triggering throttling that would also hit OTPs.

• Transient failures (timeouts, 5xx, 429): retry with exponential backoff and jitter, a limited number of times (article 20).
• Permanent failures (invalid phone number, unsubscribed address, invalid token): mark the delivery failed or suppressed; do not retry.
• Exhausted retries: move to the queue's dead-letter queue for inspection (SQS, for instance, moves messages after a configured maximum number of receives (Dead-letter queues)).
• Provider outage: a circuit breaker per provider (article 23) opens after repeated failures; transactional SMS and email fail over to a secondary provider. Marketing traffic simply waits in its queue.

A small window remains: the sender can call the provider, the provider can send, and the sender can crash before recording sent. On redelivery, the sender sees sending (not sent) and must decide. With provider-side deduplication ids, retrying is safe; without them, a duplicate is possible. For OTPs a duplicate is harmless (both carry the same code if the code is generated once and stored with the notification); for payment alerts, wording such as "your payment of ₹499 was received" is still correct if seen twice. State this residual risk in an interview rather than claiming exactly-once.

Routing workers check, for each notification:

• Category and channel opt-in (marketing email off, for example). Security-critical transactional messages (an OTP the user requested, a fraud alert) may bypass marketing opt-outs, by policy.
• Quiet hours in the user's time zone: non-urgent messages are delayed to the end of the quiet period (scheduled through the job scheduler, article 37), not dropped.
• Frequency caps: at most N marketing pushes per user per day, enforced with a per-user counter (a fixed-window limit, article 31).

These lookups happen for every message, so preferences and devices are cached (cache-aside with invalidation on change, article 09).

A campaign to 20 million users is not one request but 20 million messages:

1. A campaign job reads the segment (from an analytics store or a precomputed list) in pages of, say, 1,000 users.
2. For each page, it enqueues batch messages to the marketing intake queue (one message per batch of users, rather than 20 million tiny messages).
3. Routing workers expand batches, apply preferences and caps, and enqueue per-user sends.
4. Throughput is set deliberately: 20 million over an hour is about 5,600 per second; over 10 minutes, about 33,000 per second, which must fit within provider limits and the reserved share for transactional traffic. The campaign job paces itself.
5. Each campaign records progress (last page processed), so a crash resumes instead of restarting, and per-user idempotency keys (campaign_id + user_id) prevent duplicates on resume.

Providers report outcomes asynchronously (delivered, bounced, failed). Status workers receive these reports (via webhooks, verified with the provider's signature, or by polling), map the provider message id back to (notification_id, channel), and update deliveries. Callers and support tools read status through the API.

• Delivery semantics: at-least-once through queues, with deduplication to make sends effectively once in nearly all cases; the residual window is stated.
• Preferences: cached with short TTLs and invalidated on change; an opt-out takes effect within a minute or two. For legal opt-outs (marketing email unsubscribe), the routing step can re-check the source of truth for marketing sends to be safe.
• Status: eventually consistent; provider receipts arrive seconds to minutes later.
• Ordering: generally not guaranteed across notifications; where order matters (two updates about the same order), include a version or timestamp and let the app display the latest state.

• Who may send what: each calling service authenticates with its own identity (article 28) and may only send types it owns; marketing types require additional approval.
• Personal data: phone numbers and email addresses are sensitive; store them in the user service, fetch them at send time, and avoid copying them into logs and queues where possible (send user ids, resolve contact details in the sender).
• Provider credentials in a secrets manager, rotated (article 28).
• Template safety: escape user-supplied data in templates (HTML email) to prevent injection; restrict links to approved domains.
• Signed unsubscribe links so they cannot be forged to unsubscribe others.
• Cost abuse: attackers can trigger OTP SMS to many numbers to run up costs or harass people. Rate-limit OTP requests per phone number, per account and per IP, and require extra checks after repeated requests.
• Webhook verification: verify provider signatures on status callbacks.

"The notification platform lets any service notify a user by type; the platform decides channels from type rules and preferences, across push, SMS, email and an in-app inbox. We handle about 370 million notifications a day, around 13,000 a second at peak, plus campaign spikes of 30,000 a second, and transactional messages must reach providers within 5 seconds.

The API validates, deduplicates with a per-caller idempotency key, stores the notification with an outbox event and returns 202. Routing workers apply preferences, quiet hours and frequency caps from cache, render localised templates and create one delivery record per channel. Messages go to per-channel queues split by priority, so campaigns can never delay OTPs.

Sender workers respect provider rate limits with shared token buckets, retry transient failures with backoff and jitter, send poison messages to dead-letter queues, and fail transactional traffic over to a secondary provider when a circuit breaker opens. Delivery records and provider deduplication ids make sends effectively once, with a small stated residual window. Campaigns are expanded in pages, paced to provider capacity and resumable. Status comes back through verified provider webhooks. We alert on transactional queue age, provider health and DLQ growth."

• Next: Design a Chat System Like a Messaging App, where messages must be delivered in real time, in order, to online and offline users.
• Related: the job scheduler (article 37) for scheduled and delayed sends.