Reliability & Operations · lesson 4 of 9 · about 28 min
How reliable is reliable enough, and how to spend the difference: defining an SLI as good events over valid events, turning an SLO into an error budget, alerting on burn rate instead of raw errors, choosing how many nines you can afford, using the budget to decide how fast to ship, and measuring from where users actually stand. Tap any for a plain-language card.
What to measure
Averages are a poor SLI. The mean latency hides the tail; a threshold-based SLI (“share of requests faster than 300 ms”) does not. Choose a threshold and the system's behaviour, and see how the three ways of looking at the same traffic disagree. The next scenes turn an SLI into a target (an ) and a budget for failing.
Leave the error rate at 0.1% and tighten the threshold from 1 s to 100 ms: the availability number never moves, but the SLI falls to 40%.
99.93% of requests succeeded, which sounds fine, but only 96.74% were faster than 300 ms, so the SLI (succeeded and fast) is 96.67%. The mean latency of 136 ms hides a p99 of 379 ms.
50,000 seeded requests: log-normal latency (σ = 0.5) around the median, plus the chosen share of outright failures.
Error budgets
The budget turns arguments about reliability into arithmetic. While there is budget left, ship features and take risks. When it runs out, slow down and fix reliability. The chart shows the budget remaining day by day through one simulated month in which a few incidents occur at random times. The percentage at the right is how often a month like this would meet the SLO.
Try three incidents of 20 minutes at 99.9%: the budget is usually gone. Then try 99%.
A 99.9% SLO allows 43 minutes of full outage in 30 days. This month's 2 incidents overspent it by 24%, running out on day 21. Across many simulated months with the same pattern, 57% meet the target.
Seeded months: incidents start at random times with exponential durations; a small constant background error rate (0.01%) is also charged to the budget. Budget = (1 − SLO) × 43,200 minutes.
Burn-rate alerts
Google's SRE Workbook recommends several alerts at once: a fast burn (14.4× over 1 hour, page), a slow burn (6× over 6 hours, page) and a leak (1× over 3 days, ticket). The key property: whatever the severity, each page fires before it has burned more than a fixed share of the budget (2%, 5%). Set an incident below and read which alerts fire and when.
An outage of 50% errors trips the fast-burn alert in seconds. A quiet 0.5% leak at 99.9% never pages, but the ticket alert catches it.
| Alert | Fires when | Fires after | Budget spent by then |
|---|---|---|---|
| Page: fast burn | 14.4× over 1 h | 43 min | 2% |
| Page: slow burn | 6× over 6 h | 1.8 h | 5% |
| Ticket: leak | 1× over 3 d | 3.6 h | 10% |
At 20× burn the fast alert pages 43 min into the incident, after spending only 2% of the month's budget.
Idealised: the incident begins from zero errors; an alert fires when the error ratio averaged over its window reaches burn × (1 − SLO). Windows and burn rates follow the SRE Workbook's example.
Choosing a target
It also depends on what users experience end to end. If their phone connection is 99.5% reliable, nothing you do beyond 99.9% is visible. Higher targets cost disproportionately: redundancy across zones, automated failover, slower releases. Pick the SLO users actually need, not the largest number you can say. Below, choose a target and your typical time to recover.
At 99.9% with a 30-minute recovery you can afford 1.4 incidents a month. At 99.99% you can afford 0.14: less than one incident every seven months.
99.9% leaves 43 min a month. With 30-minute incidents that is 1.44 incidents a month. Combined with a 99.5% connection, users see 99.40%: the connection already costs more than your service does.
Budget = (1 − SLO) × 43,200 minutes; affordable incidents = budget ÷ time to recover. User experience = SLO × connection reliability.
Budget policy
Release techniques change the arithmetic. A staged rollout or canary sends the new version to a fraction of users first, so a bad deploy hurts only that fraction before it is rolled back. The simulation runs thousands of four-week months with your deploy pattern and reports how much budget they burn and how often the SLO is broken.
With 10 deploys a week, 5% failing, 20 minutes each, at 99.9%: the SLO is broken in over a third of months. Switch to a 5% canary and the risk disappears.
10 deploys a week at 5% failure, each taking 20 minutes to roll back, burn on average 92% of a 99.9% budget and break the SLO in 36.6% of months. Under an error-budget policy the team would freeze launches here and invest in testing, canaries or faster rollback.
3,000 seeded four-week months. Each failing deploy causes an outage of exponential length (mean as set) multiplied by the share of users exposed (100%, 25% or 5%).
Which term?
An SLI measures, an SLO sets the target, an SLA is the contract with consequences, and the error-budget policy says what to do when the budget runs out. Each answer is worth 20 XP.
Ask: is it a measurement, an internal target, a customer promise, or a rule about what to do?
The share of requests in the last 30 days that succeeded in under 300 ms.
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Four published sources on SLOs and error budgets.
SLI, SLO and SLA, and error budgets
The Service Level Objectives and Embracing Risk chapters define an SLI as a quantitative measure of service behaviour, an SLO as a target for it and an SLA as an agreement with consequences, and introduce the error budget as the allowed unreliability that product and reliability teams share.
multiwindow, multi-burn-rate alerts
The Workbook's 'Alerting on SLOs' chapter recommends paging when the budget burns at 14.4 times the sustainable rate over one hour, or 6 times over six hours, and opening a ticket at 1 times over three days, with a shorter window to confirm so alerts clear quickly.
example error budget policy
The Workbook includes an example error-budget policy: if the service has used up its budget over the window, changes other than urgent bug and security fixes are halted until the service is back within its SLO, and a postmortem is required for large single incidents.
service credits by monthly uptime
The S3 service level agreement defines a monthly uptime percentage and promises service credits in tiers: for example 10% of the monthly charge if uptime falls below 99.9%, 25% if below 99.0% and 100% if below 95%. The credit is the whole remedy, which is why engineers care more about the stricter internal SLO.
Interview angle
Six questions, each worth 20 XP. SLOs come up whenever an interviewer asks how you would know the system is reliable enough.
Define the SLI as good over valid events, pick a target with a budget, and alert on burn rate.
How is an SLI usually expressed?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
The remedy is to measure as close to the user as practical: at the load balancer or edge, from run from outside the network, and from real-user monitoring in the client. Compare the two numbers and treat the gap as a bug in the SLI. Here the server sees only 0.01% errors, while a small share of attempts fails before it ever gets a request.
Leave the edge failures at 0.4% and switch between the two measurements: 99.99% on the dashboard, but nearly 99.6% for users, a 20-fold difference in lost minutes.
Measured from server logs the service looks like 99.990%, but users get 99.590%. About 173 minutes a month of failures never appear in any log, so the dashboard stays green while customers are shut out.
User availability = (1 − server error rate) × (1 − pre-server failure rate). Minutes = difference × 43,200.
Where to go next