Scaling Data · lesson 5 of 7 · about 28 min
Where files belong and why: what blobs do to a database cache, how replication and erasure coding buy durability, how multipart uploads and chunking handle big and changing files, how pre-signed URLs keep your servers out of the data path, and the orphaned objects that accumulate when two writes cannot be one. Tap any for a plain-language card.
Blobs
The standard design keeps a pointer in the row (a key such as photos/42/9f3a.jpg) and the bytes in an , which is a flat key-to-bytes service built for exactly this: cheap, enormous, durable and served straight to clients. The simulation uses one LRU cache shared by a hot set of 100,000 small rows (popularity skewed) and 50,000 large blobs, and measures how well the rows stay cached.
Raise the share of requests that read blobs from 0% to 30%: the hit rate on the small hot rows falls although nothing about them changed.
With 30% of requests reading 2.0 MB blobs, only 48% of lookups on the small hot rows are served from cache, against 70% with no blobs: the images pushed the rows out. An object store keeps the database cache for what only a database can do.
A seeded LRU simulation of 120,000 requests: 100,000 rows of 1 KB with Zipf popularity, 50,000 blobs chosen uniformly. Cache sizes and populations are scaled-down illustrations.
Durability
Loss happens only if more pieces fail while the repair is still running, so repair speed is part of durability. Amazon S3 states that it is designed for 99.999999999% (eleven nines) of object durability; this model shows how redundancy and repair time combine, using the failure rate and repair window you choose. These are modelling assumptions, not measured disk statistics.
Lengthen the repair time from 24 hours to a week: every scheme loses nines, and two copies become frightening. Erasure coding keeps its lead at less than half the storage of three copies.
With disks failing 2% a year and 24 hours to rebuild, three copies reach about 10.2 nines at 3× storage, a 10+4 erasure code 15.4 nines at 1.4×, and a 17+3 code 10.8 nines at 1.18×. Two copies manage only 6.0. Durability comes from redundancy and fast repair together.
Independent failures. P(stripe lost) = P(more than m of n pieces fail inside one repair window), repeated for each window in the year. Real systems also model correlated failures (a rack, a power feed, a bad batch), which are worse.
Uploading big objects
Part size is a genuine trade-off. Tiny parts pay a per-request overhead every time (here 80 ms of setup per part). Huge parts are expensive to repeat when a connection drops, because a failure wastes everything sent so far in that part. The curve has a sweet spot in between. The simulation assumes a connection that fails on average once per few hundred megabytes (adjust below).
Try 1 MB, 16 MB and 1 GB parts on a flaky link: the time is U-shaped.
5 GB in 16 MB parts over 4 connections takes about 115 s: 320 parts, 5.2% of which fail, 16 re-sent (5% of the work wasted). Near the sweet spot for this link.
Seeded Monte Carlo: 100 Mbit/s per connection, 80 ms per part, a part fails with probability 1 − e^(−size ÷ mean MB between failures) and is restarted. Parts are scheduled onto the least-busy connection.
Deduplication
How you cut the file matters. With fixed-size chunks, an edit that overwrites bytes in place changes one chunk, but inserting a single byte at the front shifts every later byte, changing every chunk: zero dedup. cuts where a rolling hash of the nearby bytes hits a pattern, so cut points move with the content and an insertion disturbs only the chunks around it. This is real chunking on 2 MB of seeded pseudo-random data.
Choose “Insert one byte at the start”: fixed-size chunking keeps 0%, content-defined chunking keeps about 96%.
After the edit, 0% of the new file's chunks already exist with fixed-size chunking and 96% with content-defined chunking. Inserting a byte moved every fixed boundary, so the whole file looks new, while the content-defined cut points stayed with the data.
Real chunkers on 2 MB of seeded random bytes: fixed boundaries every N KB versus a Gear rolling hash with a minimum chunk size of a quarter of the average; chunks identified by a 32-bit hash for this demo.
The upload path
The signature is the security. It covers the key, the method and the expiry, so the URL cannot be reused for another object, and a leaked one is valid only until it expires, which is why lifetimes are minutes, not days. Downloads use the same trick for private files. Afterwards the store notifies your app (an event) so it can record the metadata.
At 50 uploads a second of 5 MB each the app tier must carry 2 Gbit/s of pure pass-through. With signed URLs it carries almost nothing.
50 uploads a second of 5 MB is 2.0 Gbit/s of pass-through traffic, about 1 app server doing nothing but copying bytes.
Bandwidth = uploads per second × file size × 8. An app server is assumed to carry about 6 Gbit/s of proxied traffic (a 10 Gbit/s NIC at 60%), an illustrative figure.
Where does it live?
Small structured facts go in the database, big opaque files in an object store, hot public files at the edge, and files that need low-latency random access on a block volume. Each answer is worth 20 XP.
Ask: how big, how often changed, how often read, and who reads it?
Millions of user avatar images served to browsers.
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Four published storage systems.
object storage with eleven nines of durability
S3 stores objects in buckets under keys and states that it is designed for 99.999999999% durability by keeping redundant copies across devices and facilities. Since December 2020 it provides strong read-after-write consistency for all objects.
a photo store that cuts metadata I/O
Facebook's Haystack paper explains that serving photos from a conventional file system took several disk operations per photo, mostly to read file metadata. Haystack packs many photos into large files and keeps their offsets in memory, so a read costs about one disk operation.
its own block store
Dropbox splits files into blocks of up to 4 MB, identifies each by a SHA-256 hash so identical blocks are stored once, and in 2015–2016 moved the storage from S3 to its own infrastructure called Magic Pocket.
17 data + 3 parity
Backblaze has published that it stores each file in a vault using Reed-Solomon coding with 17 data shards and 3 parity shards across 20 drives in different servers, surviving the loss of any three at about 1.18 times the storage.
Interview angle
Six questions, each worth 20 XP. 'Where do you store the images?' opens many design interviews.
Metadata in the database, bytes in an object store, hot files on a CDN, and upload directly with signed URLs.
Why not store images as database rows?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
Safe orderings and cleanup: upload to a temporary key or tag the object as pending, write the row, then mark the object live (or move it); and run a periodic sweeper that deletes objects that are still pending after a grace period (say 7 days), or lifecycle rules on the bucket that expire unconfirmed uploads and abort stale multipart uploads. Without them garbage accumulates silently, forever.
At 100,000 uploads a day with 0.1% failing, a year leaves about 178 GB of orphaned files. Raise the volume to 5 million a day and it is almost 9 TB. A 7-day sweep keeps it to a small constant.
With 10 in 10,000 uploads losing their metadata, about 0 GB of orphans appear each day; after 365 days 178 GB of storage is paid for and unreachable.
Orphans per day = uploads × failure probability × file size. With cleanup the stock is limited to the grace period's worth.
Where to go next