Networking & Traffic · lesson 2 of 7 · about 28 min
How traffic is shared across servers: five algorithms measured on the same requests, how a dead server is noticed (and what that costs), what layer 4 and layer 7 can see, why sessions should not live on one server, and what happens when the balancer itself fails. Tap any for a plain-language card.
Why a load balancer
It also makes everyday work safe: you can take a server out to deploy or patch it without anyone noticing. This only works well when the servers are stateless, meaning any server can answer any request, a point the sticky-session scene returns to.
Each server here handles 100 requests a second. Raise the traffic, turn the balancer on, then knock servers down.
Try 180 requests a second with no balancer: one server can do 100. Then add the balancer and kill a server.
Capacity is 100/s but 180/s are arriving: 80/s are dropped or time out. The single server is the ceiling.
How the balancer chooses
The catch is that requests are not equal: most take 20 ms, but one in ten takes 300 ms. Round robin hands the slow ones out blindly and sometimes stacks several on one server while another sits idle. Each server here does one request at a time, so a slow one makes the line behind it wait.
The bars show the wait (the slowest 1% of requests). Raise the load, or make one server half as fast.
At high load the gap between “ignores load” and “looks at load” grows fast.
Least connections wins here: slowest 1% wait 681 ms, against 8496 ms for IP hash (sticky). Looking at load beats any fixed pattern when request sizes vary.
Simulated: 4,000 requests, 90% take 20 ms and 10% take 300 ms, arrival times random, clients skewed so a few are heavy.
Noticing a dead server
Active checks are simple but slow: until N checks have failed, the balancer keeps sending that server its share of traffic, and every one of those requests fails (or waits for a timeout). Frequent checks detect faster but add load and can flap a healthy but busy server out of the pool.
One of four servers dies. See how many requests it eats before it is removed.
Passive detection reacts in a fraction of a second at this traffic, because real requests fail constantly. Active checks are bounded by interval × failures.
For up to 15 s the balancer keeps sending the dead server its quarter of the traffic: 750 requests fail. Try passive detection, or a shorter interval.
What the balancer can see
A layer 7 balancer understands the application protocol, usually HTTP. It ends the connection, reads the request, and chooses a backend using the path, host, headers or cookies: send /api to the API servers, /static to a cache, uploads to big-disk machines. It can also terminate (decrypt once at the edge), add or strip headers, compress, rewrite URLs and stick users by cookie. The price: more CPU per request, and it must hold a full request in memory.
Many designs use both: an L4 balancer spreads load over a fleet of L7 balancers, which route by content.
Switch layers and watch the same five requests. At layer 4 they all look the same, so routing by path is impossible.
A layer 4 balancer is the fastest and simplest, and fine when every backend can answer every request or the protocol is not HTTP. It cannot tell an upload from an image.
Where does the session live?
Stickiness also skews the load: a freshly added server gets only new users, because existing users stay pinned, and a few heavy users can overload one machine. The usual fix is to make servers stateless: keep sessions in a shared store such as Redis (or in a signed token the client carries). Then any server can serve any request and stickiness is unnecessary.
Kill a server in each mode. Then add a server and see how few users it gets.
Everything is fine until a server dies. Kill one and watch the logged-out counter.
Who balances the balancer?
The maths shows why this matters. Things in a chain multiply: the service is up only if the balancer and enough servers are up. Things that are redundant combine the other way: with two independent balancers, both must fail together. Here each balancer is 99.9% available and each server 99%.
Go from one balancer to two and watch yearly downtime fall. Then raise the server count while keeping how many you need.
With one balancer the whole service can never be better than the balancer: 99.90% at best, however many servers sit behind it.
Assumes failures are independent. Real outages are often correlated (same rack, same bad deploy, same region), so treat these as best-case numbers.
Pick the algorithm
There is no best algorithm, only one that fits how your requests behave. Each answer is worth 20 XP.
Ask: are requests and servers alike? and does a client need to come back to the same server?
Ten identical stateless servers and short, uniform requests of a few milliseconds each.
Tip: press 1 to 3 to answer, Enter for the next one.
Who lives by this
Four real designs, from the packet level to the planet.
router (ECMP) → fleet of Maglev machines → consistent hashing → backends
Google's Maglev paper describes a software network load balancer running on ordinary servers instead of dedicated hardware. The router spreads packets across the Maglev machines, which use consistent hashing so a connection keeps going to the same backend even as machines change.
client → ALB (HTTP, path rules) or NLB (TCP/UDP) → targets across zones
AWS offers an Application Load Balancer that routes HTTP by path, host and header (layer 7) and a Network Load Balancer for TCP/UDP at layer 4 that AWS documents as handling millions of requests per second. Both run health checks and spread targets across availability zones.
traffic → GLB director (consistent hashing) → HAProxy tier → application servers
GitHub's load balancer posts describe a layer 4 director that uses consistent hashing so that most connections survive when a proxy server is added or drained, in front of a tier of HAProxy servers that do the HTTP routing.
one IP announced from every data centre → network routes you to the nearest
Cloudflare announces the same IP addresses from all its data centres. The internet's routing sends each visitor to a nearby one, so load is spread geographically and an overloaded or failed site is simply routed around.
Interview angle
Six questions, each worth 20 XP. Balancer questions are about what it can see, how it notices failure, and who balances the balancer.
When you draw a load balancer, add a second one next to it. Interviewers look for exactly that.
What is the main difference between a layer 4 and a layer 7 load balancer?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
The cure is detecting the failure. A shallow health check (/health returns 200 if the process is up) cannot see a server whose real requests fail, so it never ejects it. Passive outlier detection watches real responses and removes a server after a few errors in a row. Better still, make health checks deep (touch the database or the real code path) and combine both.
Break it with each algorithm, then add the passive check and watch the errors disappear.
39% of all requests hit the broken server, more than its fair 25%, because it always looks the least loaded. Nothing ever removes it: its /health endpoint still answers OK.
Simulated: 6,000 requests through 4 servers; at request 2,000 server 4 starts failing instantly. Fair share is 25%.