Reliability & Operations · lesson 5 of 9 · about 28 min
How to ship without breaking production: five rollout strategies compared on the same bad release, why a small canary may be blind, how expand-and-contract keeps two versions working against one database, feature flags that separate deploying from releasing, automated rollback, and why a destructive migration makes rollback impossible. Tap any for a plain-language card.
Shipping safely
The simulation ships a release that breaks a share of the requests it receives. Monitoring needs a few minutes and a few hundred bad requests to notice, a person needs a few more minutes to act, and each strategy has its own rollback time. Compare how many failed requests each produces, then pick one to see its exposure over time.
The same bug costs 138,000 failed requests as a big bang and about a thousand behind a flag.
Big bang (replace all at once): the bad release was noticed after 2 minutes, reached at most 100% of users, and caused about 138,000 failed requests before it was fully rolled back. The same bug as a big bang would have caused 138,000.
One-minute steps. Bad requests = traffic × share exposed × defect rate. Noticed when 500 bad requests have been seen and 2 minutes have passed. Rollback times: big bang 10, rolling 15, blue-green 1, canary 2 minutes, flag instant (illustrative).
Reading the canary
That is the hidden cost of a tiny canary on a quiet service: 5% of 200 requests a second is only 10 a second, so doubling the error rate from 0.1% to 0.2% takes about 50 minutes to confirm, and subtler regressions take hours. Gates should compare the canary to the current version running at the same time (not to yesterday), on a few metrics that matter (errors, latency percentiles, saturation), and should be backed by a fixed minimum bake time. The figures use a standard two-proportion test (1% false alarm, 80% power).
Keep 200 req/s at 5% and try to detect 0.1% → 0.5%. Then raise the traffic or the canary share and watch the wait shrink.
To tell 0.1% errors from 0.5% with confidence the canary must handle about 3,748 requests. At 10.0 requests a second that takes 6.2 min. Fast enough to gate each stage of the rollout.
n ≈ (z_α + z_β)² [p₀(1−p₀) + p₁(1−p₁)] ÷ (p₁ − p₀)², with z_α = 2.33 and z_β = 0.84. Time = n ÷ canary requests per minute.
Mixed versions
The standard recipe for schema changes is : add the new column first (old code ignores it), deploy code that writes both and reads the new one, backfill, and only much later remove the old column. It takes four deploys instead of one, and it makes every step safe to roll back. The same applies to APIs: add fields, do not rename them; remove them only when nothing uses them.
Pick the one-step rename and a 20-minute rollout: about 300,000 requests fail. Then choose expand and contract.
During the 20-minute rollout the new code expects contact_email while old instances still write email (and the reverse). On average half of the 600,000 requests land on an instance that disagrees with the database, so about 300,000 fail. A rollback would break the other half.
A simple model: during the overlap about half of requests hit an instance whose code disagrees with the schema. Real impact depends on which queries are affected.
Feature flags
Flags also allow targeting (a region, a customer, a percentage chosen by hashing the user id, so each user gets a consistent experience) and kill switches for risky dependencies. They are not free: every flag is a branch that must be tested both ways, and old flags left in the code are a maintenance hazard, so remove them once a feature is fully released. Compare a canary and a flag ramp against a big bang for the same bug.
Lengthen the person's response time and watch the canary's harm climb while the flag's stays small: with a flag the kill switch is instant once someone reacts.
The same bug costs 594,000 failed requests as a big bang, 40,500 as a canary and 10,800 behind a flag, because the flag exposes only 1% of users to the new path while monitoring catches up and then turns it off with no redeploy.
Same one-minute-step model as the first scene; a flag exposes 1% for 10 minutes then ramps, and the kill switch takes effect immediately after a person acts.
Automated rollback
Automation is not magic. It needs trustworthy metrics, thresholds chosen so normal noise does not trigger a revert, and a rollback path that is exercised often enough to work. Even so it is the cheapest way to cut incident minutes. Compare a human reaction time with an automated revert for any strategy.
Choose the canary and make the response 30 minutes: that is a bad night. Automate it and the damage collapses to the detection time.
With a 15-minute human reaction the release caused about 42,000 failed requests and was fully undone after 21 minutes. An automated rollback acting at detection brings that to 3,600 and 6 minutes, avoiding 38,400 failures.
Same simulation as scene 1 at 1,000 req/s with a bug breaking 20% of the requests it sees. Automated rollback means a response time of zero.
Which strategy?
Rolling replaces instances in batches, blue-green swaps whole fleets, canary tests on a slice of live traffic, and flags decide per user at runtime. Each answer is worth 20 XP.
Ask: do I need an instant switch back, a trial on real traffic, a gradual replacement, or control per user?
Move to a new major version of the base operating-system image, with the ability to switch back within seconds.
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Four published sources on safe delivery.
BlueGreenDeployment
Martin Fowler's bliki describes blue-green deployment, credited to Dave Farley and Jez Humble's book Continuous Delivery: run two identical production environments, deploy to the idle one, test it, switch the router, and keep the old one for an instant rollback.
'Automating safe, hands-off deployments'
Amazon describes deploying in waves, starting with a single small unit, with bake time between stages, automatic alarms that stop the rollout and roll back without human action, and a rule against deploying to multiple regions at once.
Spinnaker's canary judge
Kayenta, open-sourced by Netflix and Google as part of Spinnaker, compares metrics from a canary against a baseline using statistical tests and produces a score that decides whether a deployment is promoted or rolled back automatically.
'Rapid release at massive scale'
Facebook engineering describes shipping web code continuously through stages that begin with employees and a small share of production before reaching everyone, with automatic monitoring, using runtime configuration and gating so that features can be switched off without a new push.
Interview angle
Six questions, each worth 20 XP. Deployment safety is a standard follow-up to any design with frequent releases.
Name the strategy, say how fast you can undo it, and mention schema compatibility.
What is the main benefit of a canary release?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
Defences: make migrations so the old shape stays valid until you are sure; keep destructive steps in their own later release, after a soak period; take a backup or snapshot before irreversible changes; and rehearse the rollback. Here a service writes 200 records a second. Compare a destructive one-step migration with the expand-and-contract approach after the bug has been live for some hours.
Keep 200 writes a second and 6 hours: more than four million rows exist only in the new shape. Switch to expand and contract and the rollback is clean.
After 6 hours at 200 writes a second, 4,320,000 rows exist only in the new format. The old code cannot read them, so rolling back the build trades one outage for another; the only way out is a risky roll-forward fix and a data repair.
Rows = writes per second × 3,600 × hours. With expand-and-contract the old shape is retained, so no rows are stranded.
Where to go next