Foundations · lesson 3 of 5 · about 25 min
What 99.9% and 99.999% really allow, why a chain of dependencies is weaker than any link, how redundancy works and where shared causes cap it, what failover costs in seconds, and how one lost server can take down the rest. Tap any for a plain-language card.
The nines
Over a year, three nines allow about 8.8 hours of downtime, four nines about 53 minutes, five nines about 5 minutes. Each nine is ten times harder, because every extra nine means failures must be rarer and repaired faster, with less and less time for a human to even notice.
Move the slider, then try the second one: a single 10-minute incident, as a share of the month's allowance. This is the idea behind the used by many teams.
At four nines the whole month's budget is about 4 minutes. One slow rollback eats all of it.
Bar length is the logarithm of the yearly downtime: each step down is ten times less. Year = 365.25 days.
A 10-minute outage uses 23% of a month's allowance at 99.9% (44 min). Plenty of room left.
Dependencies in a chain
Two ways out. Make the individual links more reliable (expensive: each nine is ten times harder), or make some dependencies soft: if the recommendations service is down, show the page without recommendations. A soft dependency does not appear in the product at all.
Set 10 services at 99.9% and read the total. Now mark 6 of them soft.
Bars show availability from 90% (empty) to 100% (full) for 99.9% links.
8 hard dependencies at 99.9% give 99.203%, about 8.0× the downtime of a single link (5 h 49 min a month). Try making some of them soft.
Redundancy
Two things stop the maths from working in real life. First, : a bad deploy, a full disk on the shared volume, a mistake in the config, a power cut in the one room: they take the copies down together. Use the second slider to say what share of each server's downtime is shared. Second, everything the copies depend on in common (the load balancer, the database, the DNS provider) sits in series with them, so you cannot beat its availability.
Set the common-cause share to 20% and watch the third replica stop helping.
Bar length is the number of nines (minus one). The dashed line is the ceiling set by the shared component.
2 copies at 99% would reach 99.99% if they failed independently. With 0% of their downtime shared and a 99.99% component in the path, the system reaches 99.98%. The shared component is now the ceiling: more copies cannot help.
Failover
The cleanest way to count it is : availability = MTBF ÷ (MTBF + MTTR). Failing less often (longer MTBF) and recovering faster (shorter MTTR) both help, and recovering faster is usually the cheaper lever. Taking a human out of the loop is the biggest single win.
Compare Automatic with Manual at 6 failures a year. Then see how many failovers a four-nines budget can afford.
Detection 15 s + switch-over 30 s = 45 s per failure. At 6 failures a year that is 4.5 min, or 99.999144% (check: MTBF/(MTBF+MTTR) = 99.999144%). A four-nines budget (53 min a year) allows 70 such failovers. This setup fits.
Pure arithmetic from your inputs. Real detection also has a tail: the first check after a crash can land anywhere in the interval.
A simulated lifetime
With independent failures the simulation lands close to the formula. Then add common-cause events (the same bug, the same zone, the same bad config push), each one taking every copy down for 30 minutes. A few of those a year can cost more than all the independent failures combined.
Set copies to 3 with 0 shared events, then add 2 shared events a year and compare the observed downtime.
With independent failures, 2 copies reach 99.99559% (formula 99.99695%). The two should be close; any gap is randomness, because overlaps of rare events are rare.
Seeded Monte Carlo over 20 simulated years: same inputs always give the same result.
Which design meets the target?
More availability costs more money and more complexity, so the right design is the cheapest one that meets the target. Each answer is worth 20 XP.
Turn the target into minutes of downtime per year first, then ask what a failover costs.
An internal wiki used in office hours. A few minutes of downtime a week is fine (about 99%).
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Four real examples, and what each has said publicly.
your instance → Region-level SLA → service credits
AWS publishes a Region-Level SLA of 99.99% monthly uptime for EC2 (and a lower 99.5% for a single instance), with service credits if it is missed. The gap between those two figures is the point of this lesson: multiple instances in multiple zones are what turn 99.5% into 99.99%.
object → many devices across ≥3 zones → 99.99% / 11 nines
AWS documents S3 Standard as designed for 99.99% availability and 99.999999999% (11 nines) durability by storing objects redundantly across multiple devices and at least three availability zones. Availability and durability are different promises: one is about being reachable, the other about not losing data.
SLO target → error budget → ship or stabilise
Google's SRE book argues that 100% is almost never the right target: users cannot tell 99.99% from 99.999% when their own phone and network are less reliable. It proposes an error budget, the allowed failure, to be spent on launches and risk, and to pause launches when it runs out.
business hours → random instance terminated → service keeps running
Netflix built Chaos Monkey to terminate production instances at random during business hours, so engineers must design services that survive losing any single machine. It is redundancy tested continuously, instead of trusted on a diagram.
Interview angle
Six questions, each worth 20 XP. “How would you make this highly available?” follows almost every design.
Say the target in minutes, remove single points of failure one by one, and then name what the copies still share.
Roughly how much downtime per year does 99.9% availability allow?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
The defences are old and dull. Headroom: run the cluster cool enough that losing one is survivable (the maths of “N+1” from the Scaling lesson). Load shedding: when over capacity, refuse the excess cleanly (), so the survivors keep working at full speed for most users instead of drowning for all of them.
Each minute, if the load per server is above the “falls over at” threshold, one more server fails.
Start at 80% with 5 servers and watch it unravel. Then lower the load to 70%, or switch shedding on.
One server lost and the cluster settled: the survivors run at 100%, below 105%. This is what headroom buys.
Deterministic model: load per server = total load ÷ servers alive; above the threshold one server fails each minute.
Where to go next