Scaling Data · lesson 2 of 7 · about 30 min
When replicas stop helping and the data itself must be split: how many shards, range versus hash and what each breaks, hot keys, why cross-shard queries do not scale, what happens to your data when you add a shard, and why moving data off a busy shard can knock it over. Tap any for a plain-language card.
Why shard
How many shards? Whichever limit is hit first. shards = max(data ÷ what one node holds, writes ÷ what one node can write). Each shard is then usually replicated for safety, so the machine count is shards × copies. A warning from experience: sharding is a one-way door. It complicates every query, join and transaction that crosses shards, so teams exhaust simpler options first (a bigger machine, read replicas, caching, archiving old data).
Keep 10 TB and 50,000 writes a second, with nodes of 4 TB and 10,000 writes a second: writes need 5 shards, data 3. Whichever limit is bigger decides.
The write rate is the limit: 3 shards for the data, 5 for the writes, so 5. With 3 copies each that is 15 machines, and every cross-shard query is now harder.
Exact arithmetic. Node limits are illustrative round numbers; measure your own hardware.
Range or hash?
The danger of ranges is skew. With a such as a timestamp or auto-increment ID, every insert goes to the last shard: one machine does all the work while the rest idle. Even natural keys like surnames are lumpy. Hash avoids that, at the price of range scans. MongoDB's documentation warns about exactly this: a monotonically increasing shard key sends all inserts to one chunk.
The simulation places 6,000 writes of the chosen workload on 2 to 16 shards, with real hashing for the hash strategy. “Scan” is how many shards a query for the newest 5% of the key range must visit.
Try timestamps with range partitioning, then switch to hash. Then look at what happened to the scan.
Keys are spread evenly over the whole key space.
Well balanced: the busiest shard is only 1.07× the average. And a range scan visits only 1 of 8.
6,000 seeded keys per run; hashing is a real 32-bit hash. Surname-initial weights are illustrative approximations, not census data.
Hot keys
The usual cure is : split a hot key into several sub-keys (post:42#0 to post:42#7) so it lives on several shards. Writes pick a random sub-key; reads must gather all of them. You only need to do this for the handful of keys known to be hot, so teams detect them and salt them.
Here 100,000 requests are drawn from 10,000 keys with a Zipf popularity. The skew slider is the Zipf exponent (higher = a few keys dominate).
Set skew 1.2 on 8 shards, then salt the top 5 keys 8 ways, and compare the busiest shard.
The single hottest key gets 20.8% of all traffic and lives on one shard, so that shard carries 27.0% (2.2× its fair share of 12.5%). More shards would not fix this: the key still lives on exactly one of them.
Seeded sampling of 100,000 requests from a Zipf distribution over 10,000 keys, hashed onto the shards.
Cross-shard queries
Capacity. A scatter query uses one unit of work on every shard. If a fraction f of your queries scatter, a cluster of n shards does only n × C ÷ (1 + f(n − 1)) queries a second: with f = 100% that is exactly C, the same as one machine, however many you add. Latency. The query waits for the slowest of n shards, the fan-out effect from the Latency lesson: more shards means a worse tail.
The common fixes: choose the shard key so the main queries are single-shard; keep a global secondary index (itself sharded by the lookup field); or send heavy analytic queries to a separate system.
Set 64 shards with 20% scatter queries: capacity is only about 5× one shard, not 64×.
16 shards of 5K/s each could do 80K/s, but with 20% of queries scattered the cluster manages 20K/s: 4.0× one shard, not 16×. A scatter query also waits for the slowest of 16: p99 348 ms against 27 ms for a single shard.
Capacity is exact for the formula. Latency is a seeded simulation: each shard ~10 ms, 1% of the time 20× slower. The 5,000/s per shard is illustrative.
Adding a shard
Two better designs. : hash keys into many small buckets (say 1,024 or 16,384, as Redis Cluster does with hash slots) and assign buckets to machines. A new machine just takes some whole buckets from the others, and only about 1 ÷ (n + 1) of the data moves, the theoretical minimum. (a lookup table of key or tenant to shard) lets you move exactly what you choose. The next lesson, consistent hashing, is the third answer.
Compare mod-n and logical shards at 8 → 9, then check what that means for a 40 TB dataset.
Going from 8 to 9 shards: hash mod n moves 88% of keys (35.4 TB of 40). Fixed logical shards move 11.0% (4.4 TB), close to the minimum of 11.1%.
Computed from 20,000 hashed keys (mod) and a real rebalancing of 1,024 fixed partitions onto the new node.
Pick the partitioning
The shard key is the decision you live with longest. Match each workload to the way of cutting its data. Each answer is worth 20 XP.
Ask: is the key ordered and scanned, or only looked up? and do I need to place some data by hand?
Social-app user records looked up by user_id, with an even spread of load wanted and no range queries on the id.
Tip: press 1 to 3 to answer, Enter for the next one.
Who lives by this
Four published examples of shard keys and what they cost or bought.
object id → logical shard → physical Postgres server
Instagram's engineering blog describes starting with thousands of logical shards mapped onto a small number of physical PostgreSQL servers, each logical shard a schema inside a database. Moving a busy logical shard to a new machine is a copy of one unit, and the 64-bit IDs they generate carry the shard number inside them.
workspace id → logical shard → Postgres database
Notion has written about sharding its PostgreSQL monolith by workspace id into a few hundred logical shards spread over a few dozen databases, so that everything for one workspace lives together and queries stay within a shard. Later re-sharding was done by adding databases and moving logical shards.
(channel_id, time bucket) → partition → node
Discord's engineering blog on storing trillions of messages explains that messages are partitioned by channel and a time bucket, so one very active channel does not live in a single unbounded partition. Hot partitions still hurt, and moving from Cassandra (177 nodes) to ScyllaDB (72 nodes) was partly about taming them.
shard key → chunks → shards
MongoDB's documentation offers both hashed and ranged shard keys, and warns that a monotonically increasing key (an ObjectId or a timestamp) sends all inserts to a single chunk. It recommends choosing a key with high cardinality that also appears in your common queries.
Interview angle
Six questions, each worth 20 XP. “How would you shard this?” is the standard follow-up to any storage design.
Name the shard key first, say what it makes cheap and what it makes expensive, and say how you would add a shard.
Why do read replicas not solve a write bottleneck?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
The slider is how much of each source shard's capacity the migration is allowed to take. Too little and moving 400 GB takes days; too much and, on top of the user traffic, the shard passes 100%: queues grow without limit and users see timeouts. The curve is the same queueing wall from the Latency lesson (illustrative 10 ms service time).
Ways out: throttle the copy, run it off-peak, copy from a replica instead of the primary, and use many small logical shards so each move is small and can be paused.
Leave the hot shard at 85% and give the migration 20%: it goes past 100%. Then try off-peak, or throttle to 10%.
The busiest shard needs 85% for users plus 20% for the copy: 105% of its capacity. Past 100% it can serve only 95% of requests and queues grow without limit. The rebalancing meant to fix the hot shard is what knocks it over.
Utilisation = user load + copy share. p99 uses the M/M/1 formula (illustrative 10 ms service time); 400 GB per shard at 200 MB/s full speed are illustrative.