Reliability & Operations · lesson 6 of 9 · about 28 min
How to match capacity to load without paying for idle servers or dropping requests: what a peak-sized fixed fleet costs, the control loop behind autoscaling and its three delays, why spikes beat reactive scaling and what to do about it, flapping and cooldowns, capacity planning with lead times, and the case where adding servers makes things worse. Tap any for a plain-language card.
Why scale automatically
adds instances when load rises and removes them when it falls, so capacity follows demand and you pay for what you use. It only works if the service is enough for instances to come and go, and if scaling is fast enough to keep up. Pick a traffic shape and see what a fixed fleet costs.
Choose “Sudden spike”: the fleet needed for the peak is 42 instances, but average demand needs only a quarter of that.
Peak demand of 4,200 req/s needs 42 instances; the average of 1,156 would need only 12. A fixed fleet sized for the peak runs at 28% utilisation and costs 84 instance-hours over two hours. More than half of what you pay for is idle.
Seeded two-hour demand curves in 10-second steps, base level about 1,000 req/s. Instances needed = ceil(peak ÷ capacity per instance).
The control loop
The target is a safety margin. At a 70% target there is 30% headroom to absorb growth while new instances start. Chart: demand in black, capacity in dashed red. Any time the black line is above the red one, requests are being dropped. Move the target and the two delays.
With the daily wave, capacity tracks demand easily. Change to the sawtooth or noisy shape and increase the delays to see the gaps open up.
Targeting 70% with a 60-second startup and a 60-second metric delay, the fleet dropped 0.00% of requests and used 39.0 instance-hours against 27.1 for a perfectly clairvoyant fleet. The loop keeps up with this traffic.
10-second steps; each instance serves 100 req/s; scale-in waits 5 minutes below the desired level; limits 2 to 80 instances.
Spikes
Four common defences. More headroom (a lower target) absorbs more before the new instances arrive, at the price of idle capacity. A warm pool of pre-initialised instances cuts startup to seconds (AWS offers warm pools for this). Scheduled or predictive scaling adds capacity before a known event (Netflix described a predictive engine, Scryer, for its daily pattern). And a queue turns an unservable burst into a delay instead of an error. Compare them on the same spike.
Reactive-only drops about 6% of requests. Try each fix and watch the dashed capacity line arrive earlier.
Reactive only: 5.84% of requests dropped (5.84% with reactive scaling only) for 36.5 instance-hours. By the time the extra instances boot, the spike has taken the old fleet to its limit.
Same simulation as before on a five-minute 4.2× spike at 1,000 req/s base. The scheduled option adds 45 instances ten minutes before the spike.
Flapping
The cure is asymmetry: scale out fast, scale in slowly. A (a stabilisation window) makes the scaler wait, for example five minutes of consistently lower demand, before removing capacity. Kubernetes' HPA defaults to a 5-minute stabilisation window for scale-down, and its Cluster Autoscaler waits 10 minutes before removing an unneeded node. The simulation uses noisy traffic around 1,800 req/s.
Set the cooldown to 0 and count the scaling events; then 5 minutes. Notice the dropped-request figure barely changes while the churn collapses.
With a 5.0-minute cooldown the fleet changed size 44 times in two hours (396 with none), dropping 0.59% of requests and using 73.1 instance-hours. Few, deliberate changes: the fleet follows the trend, not the noise.
Noisy demand: 1,800 ± 450 req/s per 10-second step. Scaling actions = scale-out batches plus scale-in steps.
Capacity planning
Keep headroom for spikes and for failures (if one of five zones fails, the other four must carry its load: that is 25% headroom for N+1). Subtract the lead time to get the resource: cloud quota increases can take days, hardware weeks to months. The latest safe order date is when utilisation will reach your ceiling, minus the lead time. Set your numbers.
At 60% utilisation growing 8% a month with 25% headroom and a 3-month lead time, you have to order in about 2 months, not when the alarm goes off.
Growing 8% a month from 60%, utilisation reaches your 75% ceiling in 2.9 months (100% in 6.6). With a 3-month lead time you must order right now: you are already late.
Months to reach a utilisation u₂ from u₁ at g per month = ln(u₂ ÷ u₁) ÷ ln(1 + g). Compound growth, no seasonality.
Which approach?
Reactive scaling follows load, scheduled scaling anticipates it, headroom absorbs it, and a queue defers it. Each answer is worth 20 XP.
Ask: is the load predictable, how fast does it arrive, how fast can capacity start, and can the work wait?
A stateless API with a smooth daily wave and instances that start in under a minute.
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Four published sources on autoscaling.
desired replicas = ceil(current × current metric ÷ desired metric)
The Horizontal Pod Autoscaler documentation gives the algorithm as desiredReplicas = ceil(currentReplicas × currentMetricValue ÷ desiredMetricValue), evaluated by default every 15 seconds, with a stabilisation window of 5 minutes before scaling down to avoid flapping.
predictive autoscaling
Netflix wrote about Scryer, an engine that forecasts the next day's load from historical patterns and provisions capacity ahead of demand, because purely reactive scaling lagged the daily ramp and the startup time of instances.
pre-initialised instances
Warm pools keep a set of already-launched, pre-initialised instances stopped or hibernated, so that when the group scales out it takes instances from the pool in far less time than a full launch and initialisation.
slow, careful scale-down
The Cluster Autoscaler adds nodes when pods cannot be scheduled and removes a node only after it has been unneeded for a configurable time, ten minutes by default, so a brief dip in demand does not cause nodes to be removed and re-added.
Interview angle
Six questions, each worth 20 XP. Autoscaling is a standard follow-up to any 'how does it handle ten times the traffic?' question.
Mention metric lag and startup time, scale out fast and in slowly, and the bottleneck you are not scaling.
What formula does the Kubernetes Horizontal Pod Autoscaler use?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
The fix is to find the bottleneck before scaling: cap the fleet at the size the database can support, use a connection pooler or proxy that multiplexes thousands of app connections onto a few dozen database ones, add read replicas or caching, and scale the constrained tier. Move the sliders: the chart shows total throughput against instance count.
Leave 50 connections per instance against a database comfortable at 1,000: throughput peaks near 20 instances and then collapses. Cap the fleet there.
Throughput peaks at 23 instances (2,268 requests a second). At 60 instances there are 3,000 connections against a database that copes with 1,000, and throughput has fallen to 333. Every extra server is making the incident worse.
Illustrative model: each instance serves 100 req/s; the database delivers its capacity while connections ≤ M and (M ÷ connections)² of it beyond.
Where to go next