Design Interviews · lesson 3 of 15 · about 30 min
A chat service end to end: the connections it must hold, how a message is delivered exactly once on a flaky network, who decides the order, what a group message costs, why presence is expensive, and how a reconnect storm can take down a healthy service. Tap any for a plain-language card.
Millions of open lines
Messages per second then size the rest: each message is small (a few hundred bytes) but arrives in huge numbers. A tells the gateway holding the recipient's connection which message to push, since sender and recipient are usually on different servers. Estimate yours; the per-connection and per-server figures below are assumptions.
500 million daily users with 20% online at once is 100 million connections; at 500,000 per server that is 200 gateway servers holding about 1 TB of connection state. 40 messages per user per day is 231,000 messages a second.
500 million daily users with 20% online need 100M open connections: 200 gateway servers at 500K each. Traffic is 231K messages a second (694K at peak) but only about 370 Mbps, because messages are tiny. The hard part is connections, not bytes.
Arithmetic. Per-server connection counts and per-connection memory are assumptions you should measure; they depend on language, runtime and TLS settings.
One tick, two ticks, blue ticks
The side effect is duplicates: if the message arrived but the ack was lost, the server pushes it again. The fix is to give every message a unique message id chosen by the sender and have the receiving device ignore ids it has already seen, so repeats are harmless (). Retries stay; duplicates disappear. Raise the loss rate and compare.
10% of acknowledgements lost, 10,000 messages: the server makes 11,060 delivery attempts. Without an id check 1,060 messages show twice; with it, none.
With 10% of acks lost the server retried until it got one: 11,060 attempts for 10,000 messages, and 1,060 extra copies appeared on the screen.
Seeded simulation: after each push the ack is lost with the chosen probability and the server pushes again (up to 50 times).
“Yes” before the question
Order by something the server controls: it assigns each message in a conversation the next when it accepts it, and every device sorts by that number. The client timestamp is kept only for display. Sequence numbers are cheap when each conversation lives on one shard. Ordering across different conversations is not needed at all, which is what makes this scale.
With one phone's clock 500 ms ahead and a message every 3 seconds, sorting by client time puts 6.3% of neighbouring messages in the wrong order. At a 2-second skew it is 21%. Server sequence numbers: 0%.
With one phone 0.5 s ahead and a message every 3 s on average, 6.3% of neighbouring messages appear in the wrong order when sorted by client timestamps. A per-conversation sequence number assigned by the server never misorders.
Seeded simulation of two people alternating 600 messages with exponential gaps; one phone's timestamps are shifted by the skew.
One message, many inboxes
End-to-end encryption adds a client-side cost: encrypting separately for each recipient (pairwise) makes the sender do 255 encryptions per message, while a scheme encrypts once with a shared group key and distributes that key securely. Group size limits (hundreds to a few thousand) exist for exactly these costs. Compare both.
1,000 group messages a second in groups of 256 means 255,000 deliveries a second. Pairwise encryption would need 255,000 encryptions a second; a sender key needs only 1,000.
1,000 messages a second to groups of 256 make 255K deliveries a second. Pairwise encryption means 255K separate encryptions on senders' devices, versus 1K with a sender key.
Arithmetic: deliveries = sends × (members − 1). The body is stored once; only the per-member delivery state multiplies.
“Online” is expensive
Real systems cut it down: send presence only to contacts who currently have a chat with you open or are looking at your profile (subscribe on demand), pull the status when a screen opens instead of streaming it, debounce short flaps (a phone dropping for 5 seconds is not an event), and keep it approximate. Last-seen can be minutes stale without anyone noticing. Compare pushing everything with pulling on demand.
500 million users flipping state 20 times a day to 200 contacts each is 23 million pushes a second if everything is pushed; pulling only when a chat opens (15 opens a day each) is about 87,000 lookups a second.
500 million users changing state 20 times a day make 116K changes a second. Pushing each to 200 contacts would be 23.1M messages a second; pulling status only when a chat opens is 86.8K lookups a second, 267× less.
Arithmetic: push = users × changes per day × contacts ÷ 86,400; pull = users × chat opens per day ÷ 86,400.
Pick the design
Each symptom points to one design choice. Each answer is worth 20 XP.
Match the symptom: wrong order, repeats, too much status traffic, or a herd of reconnects?
A reply appears above the question it answers.
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Published engineering facts about messaging systems.
Erlang gateways
WhatsApp engineers have described running its chat servers on Erlang and reported holding about two million simultaneous TCP connections on a single server, a figure that showed how far connection-oriented servers can be pushed with the right runtime.
used by WhatsApp
WhatsApp completed end-to-end encryption using the Signal Protocol in 2016; for groups it uses a sender-key design in which a sender encrypts once for the whole group instead of once per member.
Elixir gateways, then Rust
Discord's engineering blog describes real-time gateways written in Elixir holding millions of concurrent connections, and later work using Rust to scale them further, as well as migrating its message storage to a database better suited to its access pattern.
MQTT
Facebook described adopting MQTT, a lightweight publish-subscribe protocol designed for constrained devices, for Messenger in 2011 in order to save battery and bandwidth on phones.
Interview angle
Six questions, each worth 20 XP. Strong answers cover connections, delivery guarantees and ordering.
Name the connection layer, the delivery guarantee and who decides the order.
What dominates the capacity planning of a chat service?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
The cure is on the client: reconnect after a randomised, exponentially growing delay ( plus ), so the arrivals spread over minutes. On the server side keep headroom for reconnect storms, shed excess handshakes quickly with a clear “try later” instead of letting them queue, and have clients resume from the last message they saw instead of re-downloading everything.
5 million clients, 200 gateways that can each complete 2,000 handshakes a second: capacity is 400,000 a second, so even a perfect queue needs 12.5 s. Reconnecting all at once is a 12.5× overload; spread over 60 s it is 21% of capacity.
5M clients arriving at once offer 5M handshakes a second to servers that can do 400K: 12.5× overload, so handshakes time out and clients retry, making it worse.
Arithmetic: arrival rate = clients ÷ spread; recovery time ≥ clients ÷ capacity. A real overload is worse than this, since timed-out handshakes are retried.
Where to go next