Design Interviews · lesson 10 of 15 · about 28 min
A notification service end to end: the channels and their limits, why a login code must not wait behind a sale, how retries and failover trade reliability for duplicates, how to avoid spamming people, when to send, and what a mid-run restart does to a campaign. Tap any for a plain-language card.
One message, three very different roads
So the design is a pipeline of queues: accept and validate, check user preferences, render, then a separate queue and worker pool per channel, so that one slow provider cannot block the others. Size the load first, with the peak that campaigns create.
100 million users receiving 5 notifications a day is 5,800 a second on average and 29,000 at a 5× peak: about 20,000 push, 7,200 e-mail and 1,400 SMS per second.
100 million users at 5 a day is 500M notifications a day: 5.79K a second, 28.9K at a 5× peak, split about 20.3K push, 7.23K e-mail and 1.45K SMS a second. SMS is the smallest volume but the biggest bill, so it is the one to protect.
Arithmetic with illustrative shares; e-mail is whatever is left after push and SMS.
Your login code is stuck behind a sale
The fix is to separate traffic by priority: distinct queues (or priority levels) for transactional and security messages, bulk marketing, and low-priority digests, with workers always taking from the highest non-empty queue, and a reserved share of provider capacity for the top class. Campaigns are also throttled deliberately so they never use the whole channel. Compare the passcode's wait.
A 10-million-message campaign through a channel that sends 5,000 a second takes 33 minutes to drain. In one FIFO queue a passcode sent right after waits that long; in a priority queue it waits milliseconds.
A 10-million campaign at 5,000 messages a second occupies the channel for 33 min. In one FIFO queue a passcode sent just afterwards waits 33 min: the user gave up long ago.
Fluid arithmetic: wait = backlog ÷ send rate for FIFO; with strict priority the high-priority message waits for at most one send slot.
“Sent” is not “delivered”
The cost is duplicates. When the provider did send the message but the acknowledgement was lost, a retry sends it again. The cure is an : a unique id per notification, passed to the provider (many support it) and checked in your own sent-log before sending, so a repeat is recognised and dropped. Reliability and duplicates must be tuned together. Try the numbers.
5% failure per attempt, 2 retries: 0.0125% of messages are never sent, but a 2% acknowledgement-loss rate causes 1.9% duplicates without an idempotency key. A second provider drops failures to about one in 60 million and the key removes duplicates.
With 5% failure per attempt and 2 retries, 0.013% of messages are never sent. With no key, about 1.90% of users get the message twice.
Exact probabilities assuming independent failures: undelivered = p^(retries + 1), squared with a second independent provider; duplicates ≈ (1 − p) × acknowledgement-loss rate when a retry exists.
Twelve pings an hour is a mute button
Batching needs a short delay and a store that groups pending events by user and kind; urgent events skip it. The saving depends on how often events arrive compared with the window: if events are rare each is alone in its window and nothing is saved; if they are frequent, a window of an hour can collapse dozens into one. Try the arrival rate and window length.
A user whose posts earn 0.5 events an hour gets 12 notifications a day unbatched; a 1-hour window sends about 9.4 (21% fewer), a 6-hour window about 3.8 (68% fewer), and a daily digest 1.
12 events a day for one user become 9.4 notifications with a 1.0 h batching window: 21% fewer buzzes. Urgent events (a security alert) bypass the window.
Exact for Poisson events: expected notifications = (24 ÷ window) × (1 − e^(−rate × window)).
9 a.m. happens twenty-four times
Inside each zone, spread the sends over a window rather than firing them in one second: a gentle ramp protects your providers and your own servers, which would otherwise face a thundering herd of users opening the app together when the push lands. Compare the peak rates.
50 million messages all at once over 30 minutes is 27,800 a second. Sent at 9 a.m. local across 24 zones with each zone spread over 30 minutes, the peak is 1,160 a second, 24 times lower.
50 million messages over 30 minutes need 27.8K a second if sent at one moment. Spread across 24 time zones, each zone only needs 1.16K a second, 24× lower, and nobody is woken at 3 a.m.
Arithmetic assuming users are spread evenly over the zones; real audiences cluster in a few zones, so the peak is higher than shown.
Pick the mechanism
Each problem has one mechanism that fits best. Each answer is worth 20 XP.
Match the symptom: late urgent messages, double sends, too many buzzes, or a spike at one moment?
A login passcode arrives 30 minutes late behind a marketing campaign.
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Public facts about notification infrastructure.
collapse identifiers
Apple's push service and Firebase Cloud Messaging both let a sender give a message a collapse key, so that a newer notification replaces an older undelivered one for the same device, which is how apps avoid piling up stale alerts while a phone is offline.
message status callbacks
Twilio's messaging API reports a message moving through statuses such as queued, sent, delivered and undelivered through webhooks, a reminder that 'accepted' is not the same as 'delivered'.
bounce and complaint handling
E-mail services such as Amazon SES track bounces and spam complaints and can suspend a sender whose rates are too high, so a notification system must remove bad addresses and honour unsubscribes.
exponential backoff over days
Stripe retries failed webhook deliveries with exponential backoff for up to about three days in live mode, an example of a long retry window paired with the expectation that receivers handle duplicates.
Interview angle
Six questions, each worth 20 XP. Strong answers protect urgent messages and avoid spamming users.
Name the queues, the retry and duplicate policy, and the user-facing limits.
Why separate queues per channel and per priority?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
Defences in layers. Checkpoint the job's progress every K messages so a restart resumes near where it stopped (it can still repeat up to K). Give every message a deterministic (campaign id plus user id) and record sent keys in a store checked before each send, so a repeat is skipped exactly. Prefer fan-out into small independent tasks (one per batch of a few thousand) so a failure repeats only a small task. Compare what a mid-run crash costs.
A 10-million-message job that crashes at a random point and restarts from the beginning sends about 5 million duplicates on average. With a checkpoint every 100,000 messages it is about 50,000. With a sent-key check, none.
Restarting from the start sends on average 5M duplicates (up to 10M) for a 10-million job: half the audience is messaged twice.
Expected duplicates when the crash point is uniform: half the checkpoint interval (or half the job with no checkpoint). Idempotency keys make the send itself safe to repeat.
Where to go next