Reliability & Operations · lesson 7 of 9 · about 28 min
What happens when a whole region fails: how RPO and RTO define the four disaster-recovery strategies, why distance costs a fixed number of milliseconds, how long failover really takes with DNS or anycast, what active-active does to write conflicts, why untested plans fail, and why the surviving region must be sized for both. Tap any for a plain-language card.
Disaster recovery
Every step up buys a smaller RPO and RTO at a higher bill, because you pay for more capacity sitting ready in another region. The figures below simulate a region lost at a random moment: backups run nightly, the replicated options copy asynchronously with the lag you set, and recovery time for backup-and-restore grows with the data size.
Make the data 200 TB and see backup-and-restore take days; the replicated options barely change. Then raise the replication lag.
Warm standby: losing the region costs on average 3 s of recent writes (23 s in the worst 1% of cases) and about 8 min to be serving again, at roughly 1.60 times the cost of a single region.
20,000 simulated disasters. Backup: nightly. Replicated options: log-normal lag with an occasional tenfold burst. Recovery: 30 min provisioning plus data ÷ restore speed; pilot light 30, warm standby 8, active-active 1 minute (illustrative). Cost multipliers are rough illustrations.
Distance has a price
This sets the design rules. A synchronous write that waits for another region to confirm adds a full round trip to every commit, so only about 1,000 ÷ RTT commits a second can be done one after another on one connection. A page that makes ten sequential calls across regions spends a second just in transit. So systems keep requests within a region, replicate asynchronously across regions, and accept some data loss on disaster (the RPO) in exchange for speed. The distances are approximate great-circle figures.
Choose London ↔ Singapore: ten sequential cross-region calls cost more than a second of pure travel time.
N. Virginia ↔ Frankfurt is 6,500 km apart, so a round trip takes at least 98 ms. Waiting for the far region to confirm every write limits one connection to about 10 commits a second, and 10 sequential cross-region calls add 975 ms of unavoidable travel time. This is why cross-region replication is normally asynchronous.
RTT = 2 × distance × route factor ÷ 200,000 km/s. Distances are approximate; real-world latency is higher because of routing and equipment.
Failover mechanics
That is why failover with a 5-minute TTL takes five minutes after detection even if everything works. Shorter TTLs help at the cost of more DNS queries (and some resolvers ignore low TTLs). Global load balancers and addresses avoid the wait entirely: the same IP is announced from several regions and routing, or the balancer, simply stops sending to the failed one. Compare them.
Leave 30 s × 3 checks and a 300 s TTL on DNS: it takes about six minutes for 95% of users to move. Switch to anycast and it takes under two minutes.
Detection takes 90 seconds. With a 300-second TTL half of the users have moved after 250 s and 95% after 380 s; the rest keep hitting the dead region until their cached answer expires.
Model: clients' cached answers expire uniformly over one TTL after detection; anycast moves traffic within about a minute. Real resolvers can be slower or ignore short TTLs.
Active-active
Conflicts then need a policy: (simple, silently loses one update), merge with a (counters, sets, text), keep both versions and ask the application (as in the Quorums lesson), or avoid them by design by assigning each record a home region and routing its writes there, which most real systems do. The figures are expected conflicts per day for a table of hot keys.
With 1,000 keys each written 10 times per 1,000 seconds in each region and a 1-second lag, there are about 17,000 conflicts a day, one every five seconds. Double the lag and the count doubles.
1,000 hot keys, each written about 10 times per thousand seconds in both regions, with 1 s of lag between them produce roughly 17,280 conflicting write pairs a day. Under last-write-wins each one silently discards an update. That is too many to ignore: give each record a home region, or use merge-friendly data types.
Expected overlaps for independent Poisson writes: keys × a × b × 2 × lag, per second, scaled to a day. Skewed access to a few very hot keys makes it worse.
Testing the plan
The remedy is to practise: scheduled where you fail over for real, automation so that the plan is code that is exercised, and regular drills that reset the clock. Netflix's Chaos Kong exercise evacuates an entire region of traffic on purpose. Set the plan's size, how reliable each step is when fresh, how fast it drifts and when it was last tested.
Leave 15 steps, 99% fresh, 2% monthly drift: a plan last tested a year ago works about 2% of the time. Drop the months to 1.
A 15-step plan where each step works 99% of the time succeeds 86% of the time when fresh. With 2% of steps breaking every month, one last tested 12 months ago works only 2% of the time. You do not have a disaster-recovery plan; you have a document.
P(plan works) = (p × (1 − drift)^months)^steps. Independent steps assumed; a simplification, but the direction is the point.
Which strategy?
Backup and restore is cheapest and slowest; pilot light keeps a core running; warm standby keeps a small full copy; active-active serves from everywhere. Match the strategy to the RPO, RTO and budget. Each answer is worth 20 XP.
Ask: how much data can we lose, how long can we be down, and what will we pay?
An internal reporting tool: a day of lost data and a day of downtime are acceptable, and budget is tight.
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Four published sources on multi-region resilience.
'Disaster Recovery of Workloads on AWS'
The AWS paper lays out backup and restore, pilot light, warm standby and multi-site active/active as increasing investments that reduce RTO and RPO, and recommends choosing from the business's recovery objectives instead of using the most expensive option by default.
US-EAST-1 outage
A mistyped command during debugging removed more S3 servers than intended in us-east-1 and the restart took hours, taking down many sites and services (and even AWS's own status page icons) that depended on that one region. The postmortem is a standard example of hidden regional dependencies.
regional evacuation drills
Netflix has described running an active-active architecture across AWS regions and regularly rehearsing evacuating all traffic from one region to the others, so that region-level failover is a practised operation rather than an emergency improvisation.
multi-region configurations
Spanner offers multi-region instance configurations that replicate synchronously across regions using Paxos to survive the loss of a region, advertised with a higher availability SLA than regional configurations, at the cost of higher write latency.
Interview angle
Six questions, each worth 20 XP. Multi-region comes up whenever an interviewer asks what happens if a whole data centre or region fails.
State RPO and RTO first, then pick a strategy that meets them, and mention testing and capacity.
What does RPO measure?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
Two further traps. The refugees arrive with cold caches: the survivor's caches were warm for its own users, not for the other region's, so the database behind it takes several times its normal read load. And autoscaling may be too slow, or capped by quotas in the surviving region. The rule for N regions is to provision so that losing one still leaves each survivor under about 60 to 70% (for two regions, each below 50% in normal operation), and to rehearse it.
With each region at 60% and no spare capacity, B can serve only about 92% of the traffic and its database sees 3.6 times its normal load. Switch to full capacity in the survivor.
Each region ran at 60%, so the survivor must take 120% of one region's capacity but only has 110%: about 8% of requests cannot be served, and with cold caches its database sees 3.6 times its normal load. The failover has become a second outage.
Load on the survivor = 2 × normal utilisation; capacity = 1 + headroom. Cold-cache database load = load × the multiple you choose (illustrative).
Where to go next