Reliability & Operations · lesson 8 of 9 · about 26 min
How to find weaknesses before they find you: chaos engineering as a scientific method with a steady state and a hypothesis, which faults reveal which defences, why you widen the blast radius in stages, how many experiments a rare bug needs, why a finding is only done when fixed and re-tested, and what happens when an experiment is left running. Tap any for a plain-language card.
Chaos engineering
The statistics are the same as for a canary. To see a small change in error rate against normal noise you need enough requests; with a small blast radius that takes time. Here the normal error rate is 0.1% and the experiment may raise it. Choose how much traffic is exposed and how large an effect you are looking for.
Expose 5% of 2,000 req/s and look for a 0.5-point rise in errors: seconds. Shrink the effect to 0.1 points and the wait grows to minutes.
To tell a rise from 0.1% to 0.6% from normal noise needs about 2,796 requests in the experiment group. With 5% of 2,000 req/s that is 28 s. Quick enough to run, watch and stop within minutes.
Two-proportion sample size (1% false alarm, 80% power), the same arithmetic as the canary scene in the deployments lesson.
Faults
Pick a fault, then switch the defences on and off to see what the experiment would reveal. This is the use of chaos in miniature: you do not learn that the defence is good, you learn whether it is there and works before an outage teaches you.
Choose “Dependency becomes slow” with no defences: even requests that never touch the dependency fail. Add a timeout and a bulkhead.
Dependency becomes slow: Calls wait for the full response. Only 38% of requests succeed, including ones that never needed the faulty part. The experiment found a missing defence.
Illustrative models built from the earlier lessons: thread-pool arithmetic for slowness (200 threads, 500 req/s, 20% use the dependency), proportional capacity for a zone loss, independent retries for packet loss.
Blast radius
Netflix's approach grew from killing an instance (Chaos Monkey) to a zone (Chaos Gorilla) to a whole region (Chaos Kong): the scope widens as the system proves itself. Below, a weakness breaks a share of the traffic it touches; monitoring needs a few hundred failed requests to be sure. Compare plans for the same hidden weakness.
With a weakness that breaks 30% of exposed requests, going straight to 100% fails over 100,000 requests. The ramp catches it at 1% for about 1,400.
1% → 5% → 25% → 100%: the weakness was noticed at 1% exposure, after about 1,440 failed requests. Going straight to 100% would have caused 108,000. Learning the same thing cost 75 times less.
Stages last ten minutes each; monitoring needs 500 failed requests to be sure; after detection the experiment keeps running for the abort time. Illustrative.
Finding weaknesses
That argues for cadence and automation: run many small experiments continuously (Chaos Monkey kills random instances during business hours for this reason), vary the faults and the conditions, and treat each new architecture change as a reason for new experiments. A scheduled is a good start; continuous experiments are what finds the long tail.
At a 5% chance per experiment and four a month, a weakness has a 71% chance of being found within six months, but only 18% in the first month.
With a 5% chance per experiment, 24 experiments over 6 months find the weakness with probability 71%. To be 95% sure you need about 59.
P(found) = 1 − (1 − q)^n, assuming independent experiments each with the same chance q.
Closing the loop
Treat findings like production incidents: prioritise by blast radius, give them owners and dates, and verify the fix with the same experiment that exposed the problem. When the finding rate outruns the fix rate, slow the experiments down rather than pile up knowledge you cannot act on, or invest in fixing the classes of problem (a standard client library with timeouts, a platform default) rather than each instance.
Set finds to 4 and fixes to 2 a month: after 18 months there are 36 known unfixed weaknesses. Raise the fix rate to match.
Finding 4 and fixing 2 a month leaves 36 known weaknesses after 18 months, each a failure the team has been warned about and has not fixed. The programme is generating risk reports, not resilience.
Open items after each month = max(0, previous + found − fixed). A model of throughput, not of difficulty.
Which experiment?
Start small and pick the experiment that tests exactly the thing you claim is true. Each answer is worth 20 XP.
Ask: what exactly am I claiming is safe, and what is the smallest fault that tests it?
You have never run chaos in production and want a first low-risk experiment.
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Four published sources on chaos engineering.
random instance termination in production
Netflix described Chaos Monkey as a tool that randomly terminates virtual machine instances in production during business hours, so that engineers build services that tolerate the loss of any single instance. It grew into a family of tools: Chaos Gorilla for an availability zone and Chaos Kong for a whole region.
Principles of Chaos Engineering
The published principles state a method: build a hypothesis around steady-state behaviour, vary real-world events, run experiments in production, automate them to run continuously, and minimise the blast radius.
Disaster Recovery Testing
Google runs company-wide disaster recovery exercises called DiRT, in which teams deliberately simulate outages of data centres, networks and dependencies to see whether services, runbooks and people cope, and then fix what breaks.
experiments with stop conditions
AWS Fault Injection Service lets teams run fault experiments (stopping instances, adding network latency, throttling APIs) against their own resources, with stop conditions tied to CloudWatch alarms so that an experiment ends automatically if a guardrail metric is crossed.
Interview angle
Six questions, each worth 20 XP. Chaos engineering shows up as a follow-up to 'how do you know it is resilient?'.
Describe the loop: steady state, hypothesis, small blast radius, automatic abort, fix and re-test.
What is the first step of a chaos experiment?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
Safety rules for the experiment itself: every fault has an automatic expiry (a time-to-live after which it removes itself, enforced by the platform rather than the person), faults are visible (annotated on dashboards and announced to the on-call), and there is a global stop that reverts every active experiment. Compare an unbounded experiment with one that expires after 15 minutes.
Leave 50% failures on 20% of 3,000 req/s for 16 hours: more than 17 million failed requests. A 15-minute expiry limits it to a few hundred thousand.
A fault touching 20% of 3,000 requests a second and breaking 50% of them, left running for 16 hours, fails about 17,280,000 requests. The experiment designed to make the system safer is now the biggest incident of the quarter.
Failed requests = traffic × 3,600 × hours × share touched × share broken, capped by the expiry if there is one.
Where to go next