Foundations · lesson 4 of 5 · about 30 min
Why latency and throughput are different, how Little's law connects them, why the average lies and what p99 means, why queues make latency climb a wall, why you cannot average percentiles, and how one slow backend in a hundred can make most pages slow. Tap any for a plain-language card.
Latency vs throughput
They are tied together by : in flight = arrivals per second × seconds each request stays. A server taking 400 requests a second, each lasting 50 ms, always has about 20 requests in progress. That number is what your thread pool, connection pool and memory must hold.
Turn it round and you get a ceiling: with P workers and latency W, the most you can ever serve is P / W. If latency doubles (a slow dependency, say) the same pool can only serve half as many requests, which is how a small slowdown becomes an outage.
Set the pool to 40 and latency to 50 ms: the ceiling is 800/s. Now raise latency to 200 ms without touching anything else.
20 requests are in flight at any moment and 40 workers can hold them. Spare room: 50%. If latency rose to 100 ms the pool would be exactly full.
Little's law holds for any stable system, whatever the arrival pattern: it is arithmetic, not a model.
The price of throughput
Here each send costs 5 ms plus 0.05 ms per item (illustrative costs; the shapes are what matter). A batch leaves when it is full or when the linger time is up, whichever comes first. With few arrivals, a big batch size is irrelevant because the linger timer fires first.
At 2,000 items a second, compare batch 1 with batch 100. Then cut the linger time and see which of the two limits you.
One sender can do only 1.82K items/s at this batch size, less than the 2K/s arriving: the queue grows. Larger batches amortise the 5 ms fixed cost.
The formulas are exact for the stated costs: latency = (b−1)/(2·rate) + F + c·b; capacity = b / (F + c·b).
Averages lie
10,000 simulated requests below: a body with a median of 50 ms, plus a share of requests that hit a slow path. Raise the tail and watch which numbers move. The mean drifts a little, p50 not at all, p99 jumps.
Why does the tail matter if it is only 1%? A page that makes 10 independent calls is slow whenever any one of them is, so the share of slow pages is 1 − (1 − p)10, far more than p.
Set the tail to 3%: p50 does not move, p99 leaps past a second. That is why teams alert on p99, not the average.
The mean is 63 ms, the median 50 ms, but 1 request in 100 takes 454 ms or more (9.1× the median). A page that makes 10 such calls is slow 9.6% of the time, not 1%.
10,000 seeded samples: a log-normal body (median 50 ms) plus the slow path. Percentiles are measured from the samples, not assumed.
A latency budget
Here every hop is a log-normal random delay (illustrative medians and spreads) and 20,000 requests are simulated through the chain. The goal is 150 ms at the 99th percentile. Switch hops off (as if cached or removed) and widen or narrow how variable they are.
Remove the partner API call: the median barely moves, but p99 collapses. The most variable hop owns the tail.
Sum of medians 67 ms, measured median 76 ms: medians add. Sum of hop p99s 472 ms but the real end-to-end p99 is 361 ms: tails do not add. The 150 ms goal is missed by 11.5% of requests (allowed: 1%).
20,000 seeded requests through independent log-normal hops. Medians and spreads are illustrative; the statistics are computed.
The hockey stick
The tail is worse: the 99th percentile of that queue is 4.6 × S / (1 − ρ). The chart plots both exact curves; the dots are a real simulation of 40,000 requests at your chosen load, with a mean service time of 20 ms. Because it is random, the simulated dot lands close to the curve, not exactly on it.
This is the engineering reason for , and the reason “the CPU is only at 85%” is not reassuring.
Slide from 50% to 95%. The last 10 points of load cost more latency than the first 80.
At 70% busy, a 20 ms job takes 66 ms on average (3.3×) and 1 in 100 takes 303 ms or more (15×). Still manageable, but the curve is bending.
M/M/1 queue: random (Poisson) arrivals, exponential service times, one server. Real systems differ in detail but share the shape.
Measuring it right
Two servers here: A is fast (median 30 ms) and B is slow (median 120 ms, more variable). B handles only a small share of the traffic. The real p99 across all requests depends heavily on that share. The average of the two p99s ignores it, so it can be badly wrong in either direction. The fix is to store histograms (counts of requests per latency bucket) and add the counts, then read the percentile off the merged histogram. That is what HdrHistogram and Prometheus histogram buckets do.
Set B's share to 1%: the true fleet p99 sits near A's, while the average of p99s says something very different.
With B on 5% of traffic the true p99 is 205 ms. Averaging the two p99s gives 321 ms, off by 57% too high. Weighting by traffic gets 92 ms, which is also wrong: only the merged data gives the right answer.
20,000 seeded requests split between two log-normal servers. “True” is the 99th percentile of the combined list.
Which lever fixes it?
Slow systems are slow in different ways, and each way has its own fix. Read the symptom, then pick the lever. Each answer is worth 20 XP.
Ask: is the system overloaded, is every request long, or is it just a few?
Median 40 ms, mean 60 ms, but 1 request in 100 takes 3 seconds because of long garbage-collection pauses.
Tip: press 1 to 3 to answer, Enter for the next one.
Who lives by this
Four published sources, and what each says about measuring latency.
request → 100 servers → wait for all → slowest wins
Dean and Barroso's paper in Communications of the ACM (2013) shows that when a request needs responses from 100 servers and each has a one-in-a-hundred chance of being slow, 63% of requests are slow. It describes hedged requests: in a BigTable benchmark of reading 1,000 keys, sending a second request after a 10 ms delay cut the 99.9th percentile from 1,800 ms to 74 ms at the cost of about 2% extra requests.
service SLA → 300 ms at p99.9 → at 500 requests/s
Amazon's Dynamo paper (2007) describes service-level agreements stated at the 99.9th percentile, such as a response within 300 ms for 99.9% of requests at a peak load of 500 requests per second, rather than on the average or median. Their reason: the slowest requests often belong to the customers with the most data, who are the most valuable.
latency · traffic · errors · saturation
Google's SRE book lists latency, traffic, errors and saturation as the four signals to monitor, and advises tracking latency as a distribution with buckets rather than a single average, and separating the latency of failed requests from successful ones, because a fast error can make averages look healthy.
load test pauses → no samples during the stall → tail hidden
Gil Tene's talk How NOT to Measure Latency shows that many load testers stop sending while a request is stuck, so they record few slow samples exactly when the system is slowest. His HdrHistogram library stores latencies in high-resolution buckets so percentiles can be merged and reported correctly.
Interview angle
Six questions, each worth 20 XP. Latency numbers and percentiles come up in nearly every design interview, usually as “what is your p99 target?”
Always say which percentile, over which window, for which requests.
A service handles 200 requests per second and each takes 150 ms. About how many requests are in flight at once (Little's law)?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
The fix is not to make every server perfect, which is impossible, but to tolerate the slow ones. A sends a second copy to another replica if the first has not replied after a short delay (about the 95th percentile), and takes whichever answers first. Because it only fires for the slow few, the extra load is small.
Pages here are 10 ms per call normally; “slow” means over 100 ms for the whole page.
Fan-out 100, slow rate 1%, no hedging. Then switch hedging on and compare p99 and the extra load.
50 calls with a 1% slow chance each: the formula says 39.5% of pages hit at least one slow call; the simulation sees 39.5% over 100 ms. The page p99 is 564 ms although the median is 22 ms.
Seeded Monte Carlo: 4,000 pages, each call log-normal (median 10 ms) with the slow chance above. The 63% figure is the formula from the Tail at Scale paper.