Reliability & Operations · lesson 1 of 9 · about 28 min
How a small blip becomes an outage and the rules that prevent it: choosing timeouts, why retries multiply through a call chain, how backoff and jitter spread a thundering herd, why a retry budget limits the total, hedged requests for the slow tail, and the metastable failure where retries keep a service down after the cause is gone. Tap any for a plain-language card.
Timeouts
The usual rule: set it just above a high percentile (p99 or p99.9) of how long the call takes when healthy, so a few percent of normal calls are not killed, and add nothing to cover a truly hung server, which should be cut off quickly. Below, calls take a median of 50 ms with a long tail, and 1% hang for 5 seconds. Compare how many calls each timeout kills and how long workers are held.
At 75 ms (1.5× the median) many healthy calls die; at 5 s every hang holds a worker for the full 5 s. Look for the knee in between.
A 200 ms timeout kills 1.08% of calls; 0.02% of all calls were healthy ones that were merely slow. Workers are held 56 ms on average against a 50 ms median. In the sweet spot: hangs are cut off quickly and healthy calls survive.
30,000 seeded calls: log-normal body (σ = 0.4) around the median, plus the chosen share of 5-second hangs.
Retries
The danger multiplies in a call chain. If A calls B calls C calls D and each layer retries 3 times, then when D is struggling a single user request can become 4 × 4 × 4 = 64 calls to D. Each layer thinks it is being sensible; together they form a aimed at the service that is least able to cope. The rule of thumb is to retry in one layer (usually the outermost or the one closest to the failing call) and nowhere else.
Set 3 layers and 3 retries and read off the multiplier; then set layers to 1.
With a 10% random failure rate and 2 retries, only 0.100% of calls fail for good. But if the failure is an outage rather than bad luck, every attempt fails and 3 retrying layers send up to 27 times the normal load to the service that is already in trouble.
Independent failures: P(all fail) = f^(retries+1). Worst-case amplification = (1 + retries)^layers when every attempt fails at every layer.
Backoff and jitter
The chart shows the offered load as a multiple of capacity (the dashed line at 1×) over the first 45 seconds, with 500 requests a second of ordinary traffic on a service that can take 1,000. Pick a policy and compare the peak, the total work the service was asked to do and how many requests failed for good. AWS's well-known measurements of the same idea found full jitter did the least total work.
Compare “Retry immediately”, “Backoff, no jitter” and “Backoff + full jitter” with the same herd.
Retry immediately: the service sees a peak of 7.9× its capacity, does 1.28 times the useful work, and 5.5% of requests fail for good (5.0% with no retries at all). The crowd returns instantly, again and again.
A fluid model in 0.5 s steps: the herd arrives at t = 0 on top of 500 req/s; the service handles 1,000/s, and above that its goodput falls as (capacity ÷ load)² because overloaded work times out. The same model underlies the next scenes.
Retry budgets
Google's SRE book recommends exactly this (a per-request retry limit plus a retry ratio), and gRPC's retry design ships a token-bucket throttle that stops retries when too many calls are failing. Below, the service is completely down for a while and then comes back. The table compares the five policies for the same outage: how much extra work each created, how overloaded the service got, and how many requests were lost.
Raise the traffic to 90% of capacity and see which policies leave the service buried while the budget keeps it near 1×.
After a 10-second outage at 700/s with 3 retries each, backoff plus jitter still makes the service do 1.46× the work and loses 11% of requests, while the 10% retry budget keeps extra work to 1% and loses 8%. With this much headroom even plain jitter recovers; raise the traffic to see why a budget matters.
Same fluid model as the previous scene (0.5 s steps, goodput falls as (capacity ÷ load)² when overloaded). The budget policy earns 0.1 retry tokens per fresh request and spends one per retry.
Hedged requests
The conditions matter: the request must be safe to send twice (a read or an write), another replica must exist, and the hedge delay must be tuned. Hedge too early and you double the load; hedge too late and you save nothing. Here 1% of requests land on a slow server, 20× slower than normal.
Hedge at about 100 ms and see p99 fall from around 480 ms to around 130 ms for 2% extra requests. Then drag the hedge delay down to 60 ms and watch the extra load jump.
Sending a second copy after 100 ms cuts p99 from 483 ms to 131 ms and p99.9 from 1407 ms to 177 ms, at the cost of 2.0% extra requests. A cheap trade for a much better tail.
40,000 seeded requests: median 50 ms, log-normal spread, plus the chosen share on slow servers. A hedge is an independent draw from the same distribution.
Which fix?
Each tool solves a different part of the problem: timeouts stop waiting, jitter spreads retries, budgets cap them, hedging trims the tail. Each answer is worth 20 XP.
Ask: is the problem waiting too long, retrying together, retrying too much, or one slow server?
A service calls a dependency that sometimes never answers, and threads pile up waiting until the service stops responding.
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Four published sources on timeouts, retries and tail latency.
'Exponential Backoff And Jitter'
Marc Brooker's AWS Architecture Blog post simulated many clients contending for a resource and compared plain exponential backoff, full jitter, equal jitter and decorrelated jitter. Backoff alone still produced clustered retries; adding jitter finished the work with far fewer calls, and full jitter was among the best.
retry limits and retry budgets
The chapter on addressing cascading failures advises limiting retries per request, using exponential backoff with jitter, and keeping a per-client retry budget, for example only retrying while retries stay under a small fraction of requests, so that a struggling backend is not buried by its own callers.
retry policy with throttling
gRPC's retry design lets a service config set maximum attempts and exponential backoff, and an optional retryThrottling policy (maxTokens and tokenRatio) in which each failure drains a token bucket and each success refills it, so clients stop retrying when many calls are failing.
'The Tail at Scale'
The paper describes hedged requests and tied requests as ways to cut tail latency, and reports a benchmark in which sending a second request after a short delay reduced the 99.9th-percentile latency of a large Bigtable read from 1,800 ms to 74 ms at the cost of only about 2% more requests.
Interview angle
Six questions, each worth 20 XP. Retry and timeout behaviour is one of the most common follow-ups to any design that calls other services.
Say: timeout on every call, retry only idempotent calls, backoff with jitter, a retry budget, and retry in one layer.
Why must every network call have a timeout?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
The tell-tale sign is that fixing the original cause does not fix the outage; operators typically recover by cutting the load (shedding, disabling clients, restarting with traffic off). Prevention is what the previous scenes taught: cap retries with a , use deadlines, back off, and keep headroom so a blip does not tip the system. Pick a policy, set how busy the service normally is, and watch whether the load ever returns below the dashed capacity line.
With “Retry immediately” at 800/s the load never comes back down. Switch to the budget and it returns within seconds.
The service was down for 10 s but retry immediately keeps offering 3.2× its capacity afterwards: it never recovers on its own, and 54% of requests are lost. The cause is gone; the retries are the outage now.
Fluid model (see earlier scenes). Recovery = first time fresh requests succeed more than 95% of the time for five seconds straight. The model is a simplification of the metastable failures described in the research literature.
Where to go next