Scaling Data · lesson 1 of 7 · about 30 min
Why we copy data, when the leader says “saved” and what that risks, why followers show old data and how to fix it, how lag builds, what a failover costs in downtime and lost writes, why clocks make conflict resolution dangerous, and what happens when two nodes both think they lead. Tap any for a plain-language card.
The write path
The simulation sends 10 seconds of writes, then crashes the leader at 200 random moments and counts acknowledged writes that no follower had yet received. The network delays are illustrative round numbers: 0.25 ms in a rack, 1 ms between zones, 35 ms between regions (one way).
PostgreSQL defaults to asynchronous streaming replication and lets you require synchronous confirmation with synchronous_commit and synchronous_standby_names. MySQL offers a semi-synchronous plugin.
Put the follower in another region and compare. Safety is cheap in a rack and expensive across an ocean.
Asynchronous: the client waits only 1.3 ms, but a leader crash loses about 1.0 acknowledged writes (up to 2): the ones still in flight to the followers.
Seeded simulation of 10 seconds of writes and 200 crash moments. Network delays are illustrative; the ratios between modes are the lesson.
Replication lag
Simulated here: one leader and two followers whose lag varies randomly around the median you choose. A user writes, then reads after the “think time”, and again half a second later. Three policies: read from any replica (scales reads, shows staleness), always the leader (correct, no scaling), or the cheap fix: send a user to the leader for a short window after they write, then let them go back to followers.
Set lag to 500 ms with a 200 ms think time and “Any replica”: a third of reads are stale. Then switch to the pinned policy.
Reading from any replica: 12.5% of users do not see their own write 200 ms later, and 0.4% see data go backwards. Only 33% of reads load the leader, so reads scale.
20,000 seeded users. Follower lag is log-normal around the median; the same rule gives the same numbers every time.
Why lag grows
A batch job that doubles the write rate for 100 seconds is enough. If the follower's spare capacity is small, a 100-second burst takes minutes or hours to drain, and every read served by that follower is that many seconds stale. Fixes: parallel apply workers (here, each extra one adds 70% of one, illustrative), throttling the batch job, or routing reads away from a far-behind replica.
Writes at 800/s against 1,000/s of apply capacity: fine, until the 2× burst. Then add apply workers.
During the burst 1600 writes a second arrive against 1000 of apply capacity, so lag climbs to 60.0 s. It takes 299 s after the burst ends to drain, using only the 200 a second of spare capacity.
Second-by-second backlog arithmetic: backlog = max(0, backlog + arrivals − applied). Lag = backlog ÷ apply rate.
Failover
Detection is a trade-off too. Declare death after one missed heartbeat and a brief network hiccup triggers needless failovers (and risks two leaders, next scene); wait for many and the outage lasts longer. This is the same health-check arithmetic as in the Availability lesson, now with data at stake. Semi-synchronous replication means the promoted follower has every acknowledged write, so nothing acknowledged is lost.
Compare “Most up to date” with “First to answer” when the followers lag by 50 ms and 400 ms.
The promoted follower was 50 ms behind at 2000 writes a second: about 100 acknowledged writes are lost, and clients were told they succeeded. That is the best choice available. The outage lasted 19 s.
Lost writes = write rate × how far the promoted follower trailed. Downtime = detection + promotion. Both are exact for the inputs.
Multi-leader
Here user A saves a cart containing a book at one data centre. A moment later (the gap you choose) the same person, having seen that cart, adds a pen at another. The second write is truly later and should win. But if the second data centre's clock runs behind by more than the gap, its timestamp is older, and last-write-wins discards the pen. No error, no log line.
The alternative is to merge instead of choosing: keep both items (a set union, the idea behind CRDTs such as an add-wins set), or keep both versions and let the application resolve them, as Dynamo's shopping cart did.
Set the gap to 100 ms and the skew to 150 ms: the pen vanishes under last-write-wins. Switch to merging and it survives.
Last-write-wins with clocks 150 ms apart: when the second write follows within 500 ms, about 30.0% of the time the truly later write carries the older timestamp and is dropped without any error. Concurrent writes are always dropped: one of the two simply loses.
The loss rate is a Monte Carlo of 20,000 write pairs with gaps spread evenly up to your limit; the cart outcome above it is exact for your two sliders.
Which setup?
There is no best replication scheme, only the one that fits what must never be lost and what may be a little late. Each answer is worth 20 XP.
Ask: can an acknowledged write be lost? and must every region accept writes?
A product catalogue read 100 times for every write, in one region. Pages a few seconds out of date are fine.
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Four real systems, and what each has published about how it replicates.
primary → WAL stream → standbys
PostgreSQL's streaming replication ships the write-ahead log to standby servers, asynchronously by default. The synchronous_commit setting and a list of synchronous standbys let you make commits wait for one or more of them, and its documentation spells out the trade: stronger durability at the cost of commit latency.
writer → 6 storage copies in 3 zones → 4 of 6 to commit
AWS documents that Aurora keeps six copies of your data across three availability zones and uses a quorum: a write is durable once 4 of the 6 copies have it, and reads need 3. With two copies per zone, it keeps accepting writes after losing a whole zone, and keeps your data readable even after losing a zone plus one more copy.
network blip → automatic MySQL failover → data on both coasts
GitHub's post-incident report says a 43-second loss of connectivity between its US East Coast network hub and primary data centre made its MySQL orchestration promote primaries on the West Coast. The East Coast had accepted writes that never reached the West Coast, so the two sides diverged, and reconciling them left parts of the site degraded for just over 24 hours.
client → any replica → quorum of N replicas
Cassandra descends from Amazon's Dynamo paper: any replica can take a write, the replication factor says how many copies exist, and the consistency level (such as QUORUM) says how many must answer. Conflicting writes are resolved with timestamps, which is why clock skew matters there too.
Interview angle
Six questions, each worth 20 XP. Replication appears in almost every storage design: expect “what happens when the primary dies?”
Name the replication mode, say what can be lost on failover, and say how users still see their own writes.
With asynchronous replication the leader crashes and a follower is promoted. What can happen?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
The fix is not a faster failure detector; any detector can be wrong. It is : every new leader gets a higher epoch number, followers and storage remember the highest they have seen, and they reject any request that carries an older one. The old leader can try, but every write fails, so nothing diverges. Clients do see errors for the overlap, which is the right outcome: a refused write is better than a quietly lost one.
Leave fencing off with a 40 s step-down and watch the diverged writes pile up. Then switch it on.
For 25 s both leaders accepted writes: 12,500 writes on the old leader that the new history does not contain. Each was acknowledged to a client. On rejoin they must be discarded or hand-merged. GitHub's 2018 incident was this at scale.
Overlap = step-down time − promotion time; divergent writes = rate × overlap. Exact arithmetic; the timings are your inputs.