Networking & Traffic · lesson 7 of 7 · about 30 min
How a server tells a browser something the moment it happens: what polling really costs, why push breaks the delay-versus-load trade, how many connections a machine can hold, how one message reaches a room spread over many servers, what happens when a phone drops its connection, and the reconnect storm after a restart. Tap any for a plain-language card.
Four ways
The costs differ by orders of magnitude. Polling's cost is set by the interval, not by how often anything happens; its delay averages half the interval. The push methods cost in proportion to events, at the price of holding a connection open per client. Header overhead of ~1,000 bytes per HTTP request is an illustrative typical figure; a WebSocket frame header is 2 to 14 bytes by the RFC.
Set 100,000 clients polling every 5 seconds with 2 events a minute each: 20,000 requests a second, 83% of them empty.
| Method | Requests/s | Bandwidth | Average delay |
|---|---|---|---|
| Short polling 83% of polls are empty | 20K | 20.7 MB/s | 2.5 s |
| Long polling holds a request open | 6.67K | 7.3 MB/s | 50 ms |
| Server-Sent Events one-way, auto-reconnect | 27.8 | 760 KB/s | 50 ms |
| WebSocket two-way | 27.8 | 707 KB/s | 50 ms |
100K clients polling every 5 s make 20K requests a second, 83% of them empty, and users wait 2.6 s on average. Pushing over a WebSocket or SSE needs 27.8 requests a second and delivers in about 50 ms, but must keep 100K connections open.
Computed from your inputs. Events arrive randomly per client; ~1,000 bytes of HTTP headers per poll, 6 bytes per WebSocket frame, ~20 bytes per SSE event (illustrative framing).
The polling curve
Push (SSE, WebSocket, or long polling) breaks the link: the server speaks when it has something, so delay is about half a round trip and load follows events. Set the delay you need, or the request budget your servers can afford, and see which one you can meet with polling.
Ask for a 1-second average delay with a 5,000 requests-a-second budget: polling cannot do both.
A 1.0 s average delay needs a 2.0 s poll, which is 100K requests a second: 20.0× your budget. With that budget the best polling can do is a 20.0 s average delay. Push delivers in about half a round trip.
Arithmetic: average delay = interval ÷ 2; requests per second = clients ÷ interval.
Connections at scale
Two hazards. Idle timeouts: load balancers and proxies close connections that carry no data for a while (an AWS Application Load Balancer defaults to 60 seconds), so the connection must send a small more often than that. Heartbeat load: a million users pinging every 30 seconds is 33,000 messages a second that carry no information. So the interval is a trade between surviving the timeout and the pings' own cost.
Set 1 million users, 32 GB machines, 20 KB per connection: memory allows 1.6 million, so two or three servers hold everyone.
Each 32 GB server holds 1M connections (limited by its descriptor limit: raise ulimit), so 1M users need 1 server. Heartbeats every 30 s keep the 60 s timeout happy and cost 33.3K tiny messages a second across the fleet.
Capacity = min(RAM ÷ state per connection, descriptor limit). State per connection varies by server and runtime; the figures are illustrative round numbers.
Fan-out
How many servers must receive each message depends on where connections land. If users are placed at random (whichever server the load balancer picked), a 50-person room is spread over most of 20 servers and every message crosses the backplane to nearly all of them. If connections are placed by room (routing by room id, a form of ), the whole room sits on one server and a message needs no cross-server hop. The price is uneven load when one room is huge.
Compare random placement with room placement for a 50-person room on 20 servers.
Random placement spreads a 50-person room over about 18.4 of 20 servers, so each message crosses the backplane 18.4 times (369K deliveries a second). Room-based routing would need 1.0.
Simulated: 400 rooms each placed over the servers at random or pinned by hash of the room id; deliveries = messages × distinct servers reached.
Dropped connections
If the gap is longer than the buffer, resume is impossible and the client must resync from scratch: fetch the full state through the normal API. That costs far more, and when thousands do it at once it is another load spike. So the buffer size is a design parameter: how long a drop do you want to survive cheaply?
Events at 2 a second, a 10-second average drop, a 20-event buffer: about 4 in 10 reconnects can resume. Raise the buffer to 100.
With 2 events a second and drops averaging 10 s, a client misses about 19 events. A buffer of 20 lets 66% of reconnects resume cheaply; the other 34% must resync from scratch: about 17.2K full resyncs a minute across 500K users if 10% reconnect each minute.
20,000 seeded drops with exponentially distributed length; missed events = rate × length. Resume works when the missed count fits the buffer.
Pick the technique
Choose the simplest technique that meets the need. Each answer is worth 20 XP.
Ask: one-way or two-way? How fresh must it be? What sits between client and server?
An admin dashboard whose numbers may be up to a minute old.
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Four published sources about running them at scale.
WebSocket → HELLO with heartbeat interval → IDENTIFY or RESUME
Discord's developer docs describe a WebSocket gateway: the server's first message (HELLO) tells the client how often to send a heartbeat, every event carries a sequence number, and a dropped client can send a RESUME with its last sequence number and session id to have the missed events replayed instead of starting over.
idle timeout 10 minutes · connection duration 2 hours
AWS documents quotas for API Gateway's WebSocket APIs: an idle connection is closed after 10 minutes, and a connection can last at most 2 hours, after which the client must reconnect. Any managed push service has limits like these, so clients must be written to reconnect and resume.
phone → Erlang server holding the connection
WhatsApp's engineers published that single servers held more than a million simultaneous connections, tuning the OS (file descriptors, buffers) and the Erlang runtime. It is the standard reference that persistent connections at huge scale are an engineering problem you can solve, not a reason to avoid push.
EventSource → text/event-stream → Last-Event-ID
The HTML standard defines Server-Sent Events: the EventSource API opens a long-lived text/event-stream response, reconnects automatically after a delay the server can set, and sends the last event id in a Last-Event-ID header so the server can resume. It is the simplest standard for one-way push.
Interview angle
Six questions, each worth 20 XP. “How would you push live updates?” follows any design with chat, feeds or notifications.
Say which method, how many connections each server holds, how you resume after a drop, and how you avoid a reconnect storm.
Why does short polling scale badly?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
The cure is jitter: each client waits a random time within its backoff window, spreading retries across the whole window so the server's capacity is used every second. Servers can also shed load early (refuse quickly), and rolling deploys avoid dropping everyone at once.
With 200,000 clients and 5,000 accepts a second, run without jitter: after three minutes most clients are still locked out. Turn jitter on.
Without jitter the retries bunch into waves: the busiest second brings 195K attempts to a server that accepts 5K. After three minutes only 35K of 200K clients are connected. 1.1M attempts were wasted.
Count-based simulation: failed clients retry after 2, 4, 8 … up to 64 s; with jitter they spread evenly over the window. Capacity is used first-come within each second.