Reliability & Operations · lesson 3 of 9 · about 28 min
How to see inside a running system: what logs, metrics and traces each give you, why log volume gets expensive, why you cannot average percentiles, how to read a trace and find its critical path, how sampling decides which traces survive, how to write alerts that are quiet and fast, and how one high-cardinality label can destroy a metrics system. Tap any for a plain-language card.
Three signals
Logs are the richest and the most expensive, because their volume grows with traffic × lines per request × services. Thirty services at 500 requests a second each, writing 8 lines of 300 bytes per request, produce about 3 TB a day. Move the sliders to see how quickly a harmless-looking log.info on the hot path becomes terabytes. The cost line uses an illustrative price of $0.50 per GB ingested; real prices vary a lot between vendors and self-hosting.
Double the lines per request: volume and cost double. Debug logging in production is a bill.
30 services × 500 requests a second × 8 lines × 300 bytes is 3.1 TB of logs every day, 43.5 TB kept for 14 days, about $46,656 a month at the illustrative price. This is why teams sample, log at the edge, and use metrics for everything countable.
Volume = services × rps × lines × bytes × 86,400 s. The price is a labelled assumption, not a quote.
Metrics
The standard fix is to record a : count how many requests fell in each latency bucket, which can be added across hosts, and compute percentiles from the combined counts. The accuracy then depends on the bucket boundaries: coarse buckets give a blurry p99. The simulation has a fast host and a slow host (the slow one takes 5% of traffic).
Compare “average of per-host p99” with the true fleet p99, then switch the buckets to coarse.
The mean is 63 ms but the true fleet p99 is 619 ms. Averaging the two hosts' p99s gives 736 ms, which is wrong by 19%. Summing histograms and reading the percentile from the total gives 735 ms with 8 buckets (+19% off): too coarse to trust.
40,000 seeded requests: a fast host (median 40 ms) and a slow host (median 400 ms). Histogram percentile uses linear interpolation inside the bucket, as Prometheus does.
Traces
Here the cart service calls Pricing, Inventory and Recommendations at the same time, then writes to the orders database. Choose a service and change its latency: make Pricing ten times faster and the total hardly moves, because Recommendations (200 ms) is the slowest of the three. Metrics would tell you the endpoint is slow; only the trace tells you why.
Shrink Pricing to 10 ms and see nothing change. Then make Recommendations 50 ms and watch the critical path move to Inventory.
Changing Pricing to 80 ms leaves the total at 317 ms. Pricing runs in parallel with a slower call (Recommendations), so it is not on the critical path and optimising it buys nothing.
Spans in red are on the critical path. The gateway adds 5 ms before and 2 ms after; the three middle calls run in parallel.
Sampling
Tail sampling needs a collector that holds every span of a trace until the trace completes, which is more machinery, but it stores far more of what you will actually look at. Set the base rate, the error rate and the share of slow requests, and compare how many of the rare, valuable traces each strategy keeps.
Keep 0.1% with a 0.1% error rate: head sampling captures roughly no errors at all, while tail sampling captures every one for about 1% of the storage.
Out of 200,000 requests, 212 failed and 2024 were slow. Head sampling at 1.0% stored 1.01% of traces and kept 0 of the 212 errors. Tail sampling stored 2.10% and kept 212: every error and every slow request, for 2.1× the storage of head sampling.
200,000 seeded requests. Head: keep with probability = base rate. Tail: keep all errors and slow requests, plus the base rate of the rest.
Alerting
A metric that wobbles around 100 will cross a threshold of 130 by chance now and then; requiring it to stay there for five minutes removes the blips but delays the page when something is really wrong. Below, a metric has a baseline of 100 and some noise; at minute 60 an incident adds a real increase. The numbers come from 30 simulated days of noise and 300 simulated incidents.
Set the threshold to 115 with 1 minute: false pages every hour. Raise the duration to 10 minutes and they stop, but detection takes longer.
Paging above 130 for 1 minute with noise of ±10: 1.9 false pages a day, and a real +40 incident is noticed after 0.2 minutes (0% never reach the threshold in four hours). People will start ignoring this alert.
The metric is 100 plus Gaussian noise each minute. False pages are counted over 30 simulated days; detection is averaged over 300 incidents that start at minute 60.
Which signal?
Logs answer 'what exactly happened?', metrics answer 'how much, how fast, how often?', traces answer 'where did the time go?'. Each answer is worth 20 XP.
Ask: one event, a trend over time, or one request across services?
Is the error rate above 1% right now, and how has it trended this week?
Tip: press 1 to 3 to answer, Enter for the next one.
Who lives by this
Four published sources on observability.
low-overhead distributed tracing
Google's Dapper paper described always-on tracing across thousands of services, made cheap by sampling (initially about one request in 1,024) and by instrumenting common libraries rather than application code. Twitter's open-source Zipkin was modelled on it.
latency, traffic, errors, saturation
The SRE book's monitoring chapter names four golden signals to watch for any user-facing system: latency, traffic, errors and saturation. It also argues for alerting on symptoms users feel and for paging only on conditions that need a human right now.
dimensional metrics
Prometheus stores each unique combination of metric name and label values as its own time series, and its documentation warns not to use labels for high-cardinality values such as user IDs or email addresses, because each new value creates a new series.
vendor-neutral instrumentation
OpenTelemetry, a CNCF project formed by merging OpenTracing and OpenCensus, defines common APIs, SDKs and a wire protocol for traces, metrics and logs, so applications can be instrumented once and send data to any compatible backend.
Interview angle
Six questions, each worth 20 XP. Observability is the standard follow-up to 'how would you debug this in production?'.
Pair each signal with its job, mention percentiles and sampling, and alert on user-visible symptoms.
Which signal gives per-event detail such as the exact error text for one user?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
user_id as a metric label. The monitoring system dies by lunchtime.The rule: labels must have a small, bounded set of values. Per-user and per-request detail belongs in logs or traces, which are built for high cardinality, not in metrics, which are built for aggregation. Prometheus's documentation warns about exactly this. The memory figure assumes about 3 KB per active series, an illustrative order of magnitude; real systems vary.
Add the user_id label with 100,000 users and watch 5,000 series become 500 million, about 1.5 TB of memory.
5 thousand series (20 services × 50 endpoints × 5 statuses) cost about 15 MB: comfortable. The labels each have a small, bounded set of values.
Series = product of the label cardinalities. Memory per series (about 3 KB) is an illustrative round number.
Where to go next