Reliability & Operations · lesson 2 of 9 · about 28 min
How to stop one failing part from taking down everything: why a slow dependency exhausts a thread pool, bulkheads that give each dependency its own compartment, circuit breakers that stop calling a service that is clearly broken, how to tune them so they do not trip on noise, shedding load by priority, and the bug of counting the wrong errors. Tap any for a plain-language card.
How outages spread
Here is the nasty part. When threads run out, requests that have nothing to do with the slow dependency (the product page, the login) also find no free thread and fail. One slow non-critical call has become a of the whole service. Nothing crashed; the service is simply full of threads waiting. Try making the slow dependency slower.
Leave 20% of traffic going to the dependency and raise its latency from 50 ms to 5 s: the pool saturates and the healthy 80% of requests start failing too.
All requests together want 25 of 200 threads, so every request gets one. Make the dependency slower or the pool smaller to see the cascade.
Steady-state arithmetic: threads wanted = rps × latency. When total demand exceeds the pool every class gets the same fraction of what it asks for.
Bulkheads
It costs some flexibility: unused threads in one pool cannot help another, and the split needs tuning to real traffic. The same idea applies to connection pools, queues, rate limits per tenant and even whole deployments (cell-based architecture). The simulation uses the same 500 requests a second as the previous scene, 20% of them calling the dependency.
Make the dependency take 5 s with a shared pool, then switch to split pools and see the healthy requests recover.
Separate pools: the dependency, at 5 s, can only fill its own 40 threads, so 8% of its requests are served (the rest fail fast), while 100% of healthy requests are untouched. The damage is contained to one compartment.
Same Little's-law arithmetic as the previous scene; each pool serves min(1, pool ÷ demand) of its own class.
Circuit breakers
Failing fast is already useful, because an instant error is better than a thirty-second wait. It becomes much better with a fallback: a cached answer, a default list, a degraded page. This simulation sends 100 calls a second; the dependency is completely down from second 20 to second 50. The line on the chart is how many calls per second actually reach it.
Compare the dependency's calls with and without the breaker, then raise the fallback to 80% and watch user-visible errors fall.
During the 30-second outage the dependency was asked 506 times instead of 3,000; the breaker opened 4.9 s in and closed 5.0 s after the dependency came back. Users saw 30% errors against 26% without it, but the errors are instant, not 30-second hangs.
Seeded tick simulation (100 ms ticks): 100 calls/s, 1% baseline errors, 100% errors from 20 s to 50 s. Minimum 20 calls in the window before the breaker may open.
Tuning
Three dials control it: the threshold (higher = fewer false trips, slower to notice), the window and minimum volume (bigger windows average out noise but react later), and the cooldown. The chart of numbers below runs one hour of normal-but-noisy traffic to count false trips, and a real total outage to measure how long the breaker takes to notice.
Start at 5% with 5 calls a second: dozens of false trips an hour. Raise the threshold to 20% and they vanish, while a real outage is still caught within seconds.
In a healthy hour (3% of calls failing at random, 5 calls a second) the breaker tripped 107 times and spent 1130 s rejecting good traffic. A real outage was detected 0.4 s after it began. Too sensitive: raise the threshold, widen the window or require more calls.
One simulated hour of healthy but noisy traffic and a separate 150 s run with a total outage from second 60. Minimum 20 calls in the window before opening; 10 s cooldown; 3 probes.
Load shedding
This model says an unprotected overloaded server serves each request with probability (capacity ÷ load)², because most of its work is wasted on requests that have already timed out. At twice capacity only a quarter of requests succeed, and half the capacity does useful work. A shedding server always uses its full capacity, and critical traffic is served first.
Set load to 2× capacity with 30% critical traffic: without shedding everybody gets 25%; with shedding all critical requests succeed.
At 2.0× capacity an unprotected server serves 25% of every kind of request and wastes 50% of its effort. With priority shedding 100% of critical requests are served, 29% of the rest, and the server does useful work all the time.
A model, not a measurement: unprotected goodput = load × (capacity ÷ load)²; shedding serves critical requests first and spends all capacity on useful work.
Which defence?
Timeouts bound the wait, breakers stop calling the broken, bulkheads contain the damage and shedding refuses excess. Pick the one that fits. Each answer is worth 20 XP.
Ask: is the problem waiting, a broken dependency, shared resources, or too much incoming traffic?
Image-resizing calls sometimes hang and are using up the worker pool that checkout shares.
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Four published sources on containing failure.
stability patterns
Nygard's book popularised the circuit breaker, bulkhead and timeout as named stability patterns for production systems, using case studies of outages in which one slow integration point exhausted threads and took down an entire application.
circuit breaker + thread-pool bulkhead
Netflix's Hystrix library wrapped each dependency call in a command with its own thread pool or semaphore (the bulkhead), a timeout, a fallback and a circuit breaker that opened on error percentage. It is now in maintenance mode, with newer libraries such as Resilience4j taking the same ideas forward.
circuit breaking and outlier detection
Envoy applies circuit-breaking limits (maximum connections, pending requests, requests and retries per cluster) and outlier detection that ejects misbehaving hosts, so that these protections live in the proxy beside every service instead of in each application.
'Using load shedding to avoid overload'
Amazon's article describes rejecting excess requests cheaply and early, prioritising important work, and testing for overload, because a server that tries to serve everyone while overloaded spends its effort on requests that have already timed out.
Interview angle
Six questions, each worth 20 XP. Failure containment is the follow-up to any design that depends on other services.
Name the pattern, say what it protects, and say what the user sees instead (a fallback).
By Little's law, how many threads does a dependency tie up?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
The fix is classification: only server-side faults and timeouts are failures; client errors are successes as far as the breaker is concerned. Related rules: do not count requests rejected by your own rate limiter, and give each tenant or endpoint its own breaker so one noisy caller cannot open it for the rest. Here the dependency is perfectly healthy; a share of requests are bad ones.
With 40% bad requests and a 30% threshold, counting all errors keeps the breaker open nearly all the time. Switch to server errors only and the breaker never trips.
The dependency is perfectly healthy, yet 100% of requests failed (only 40% were bad requests): the breaker opened 5 times and rejected 299 s of good traffic because it counted the caller's mistakes as the dependency's failures.
Healthy dependency, 100 calls/s for 300 s. With “every non-200” the bad requests count toward the threshold; probes need every call to succeed to close the breaker again.
Where to go next