Design Interviews · lesson 11 of 15 · about 30 min
An observability platform end to end: how much data logs and metrics create, how to survive an incident's log burst, how long to keep what and where, how rollups shrink metrics, how to set alerts that wake people only when they should, and how logging itself can cause an outage. Tap any for a plain-language card.
What are my machines doing?
The volumes are surprising. Logs are measured in terabytes a day and are the expensive part. Metrics are measured in series (one per unique combination of metric name and labels) and samples per second, and their cost is driven by , not by size. Estimate both.
100,000 hosts writing 20 lines a second of 200 bytes produce 0.4 GB/s, 35 TB of logs a day. 100 metrics per host every 10 seconds is 10 million series and a million samples a second.
100K hosts writing 20 lines a second of 200 bytes produce 0.40 GB/s, 35 TB a day. 100 metrics each every 10 s make 10M series and 1M samples a second: lots of events, but each sample is only a few bytes, so logs dominate the bytes and metrics dominate the query load.
Arithmetic, before compression and indexing. Real log lines compress well (often 5 to 10 times), and many systems sample or drop debug lines at the source.
Logs spike exactly when you need them
Three numbers decide the outcome: how far the burst exceeds indexer capacity, how long it lasts, and how much the buffer can hold. If the backlog fits, the indexers catch up afterwards and you merely see logs a few minutes late. If it overflows, you lose data, and you lose the oldest or the newest, which is a policy choice to make deliberately. Try different bursts.
Ingest 300 MB/s normally, 1,200 MB/s for 10 minutes, indexers handling 800 MB/s and a 150 GB buffer: the backlog hits the buffer limit and drops about 90 GB, then clears 4 minutes after the burst ends.
A 4× burst for 10 minutes against 800 MB/s of indexing fills the 150 GB buffer: 90 GB of logs are dropped, from the incident itself.
Fluid arithmetic per minute: backlog += (ingest − indexer rate) × 60 s, capped at the buffer size.
Fast to search or cheap to keep
Retention is also a legal and privacy question: some data must be kept for years, other data (personal information in logs) must be deleted promptly, which is why sensitive fields should be removed or masked at the source. Estimate the bill. The prices are illustrative, in dollars per GB per month; use your own.
40 TB of logs a day, hot 7 days, warm 30 and cold 365 at $0.10, $0.03 and $0.004 per GB-month costs about $49,000 a month. Keeping all 402 days hot would cost about $2.1 million.
Keeping 7 days hot, 30 warm and 365 cold for 40 TB a day costs about $49K a month; holding all 402 days on the hot tier would cost $2.09M, 43 times more.
Assumptions: index overhead 1.3× on hot and warm, compression to 15% for warm (index at half overhead) and cold, warm $0.03 and cold $0.004 per GB-month. All illustrative.
Last week in detail, last year in outline
The price is precision: a one-second spike disappears into an hourly mean (keep max as well as mean, and percentiles must be stored as histograms because averages of percentiles are meaningless). The other cost lever is : adding a label such as a user id to a metric multiplies the series count, which blows up memory far faster than retention does. Compare storage with and without rollups.
10 million series sampled every 10 s: raw for 15 days, one-minute for 90 days and hourly for 730 days takes about 5.5 TB. Keeping every 10-second sample for all 835 days would take about 144 TB.
10 million series every 10 s kept for 835 days at full detail would need 144.3 TB; with 15 days raw, 90 days of minute rollups and 730 days of hourly ones it needs 5.5 TB, a 96% saving.
Arithmetic at 2 bytes per compressed sample. Each rollup stores several aggregates per interval in practice, so real savings are somewhat lower.
The pager that cried wolf
The aim is alerts that are urgent, actionable and rare. Tie them to user-visible symptoms (error rate, latency) rather than causes (CPU), use that combine a fast and a slow window, route by severity (page versus ticket), and review every page that was not worth waking someone for. The simulation runs a noisy metric for two weeks with six 20-minute incidents.
With a 3-sigma incident, an alert at 1σ for 1 minute fires 189 false alarms a day. At 2σ for 3 minutes there are none, all six incidents are caught, and detection takes 4.3 minutes on average.
Alerting at 2.0σ for 3 minutes: 0.0 false alarms a day, 6 of 6 real incidents caught, after 4.3 minutes on average. A good balance.
Seeded simulation: 14 days of independent per-minute noise with six 20-minute incidents that shift the mean. Real metrics have trends and daily cycles, so use baselines that follow them.
Pick the mechanism
Each problem has one mechanism that fits best. Each answer is worth 20 XP.
Match the symptom: bursts, cost of old logs, huge metric history, or logging slowing the app?
An incident quadruples log volume and the indexers fall behind.
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Published designs and ideas.
an in-memory time-series database
The Gorilla paper described compressing metric samples by storing timestamps as deltas of deltas and values as XORs with the previous value, averaging about 1.4 bytes per data point, which made keeping recent data entirely in memory practical.
labels and a time-series database
Prometheus, started at SoundCloud, scrapes metrics from services and stores them as labelled time series; its documentation warns that every unique combination of label values is a new series, the root of cardinality problems.
symptoms over causes
Google's SRE book recommends alerting on symptoms users feel, such as errors and latency, and treating pages as precious: every page should be urgent, actionable and need a human.
distributed tracing
The Dapper paper described always-on, low-overhead request tracing that samples a small fraction of requests, the origin of today's tracing systems and an example of controlling observability cost by sampling.
Interview angle
Six questions, each worth 20 XP. Strong answers keep observability off the critical path.
Say how bursts, cost, cardinality, alert noise and failure of the pipeline are handled.
Why put a durable buffer between log agents and indexers?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
The rule is that observability must never be on the critical path. Write logs asynchronously to a bounded in-memory buffer drained by a separate thread or local agent, and when the buffer fills, drop log lines (or sample them) rather than block. Dropping logs is bad but visible (count the drops); blocking requests is worse. Also rate-limit noisy log statements, make log levels adjustable at runtime, and alert on log volume itself. Compare the two modes.
An application producing 50,000 log lines a second into a pipeline that accepts 20,000, with 5 lines per request: synchronous logging cuts throughput to 40% of normal. Asynchronous logging keeps requests at full speed and drops 60% of the lines.
Waiting on a pipeline that accepts 20K lines a second while 50K are produced slows the application to 40% of its normal throughput: the logging system is now the outage.
Rate arithmetic: when log production exceeds pipeline capacity, synchronous logging limits requests to capacity ÷ lines per request; asynchronous logging drops the excess lines.