Foundations · lesson 2 of 5 · about 25 min
Why one machine hits a ceiling, why more machines are not free, why stateless servers scale and stateful ones hurt, why the database does not clone like a web server, and how many servers you really need. Every number comes from a simulator or a formula you can move. Tap any for a plain-language card.
Scale up
The less obvious limit is that cores do not add up. Every program has a part that must run one step at a time (taking a lock, writing to a single log, waiting on one database row). says that if a fraction s of the work is serial, the speed-up on n cores is 1 / (s + (1 − s) / n), and it can never pass 1 / s, however many cores you add.
Pick a core count and a serial share. The price curve is illustrative (real cloud prices differ by provider), but the speed-up is computed from the formula.
With 10% serial work, 64 cores give about 8.8×, not 64×. Make the serial share smaller and watch the ceiling rise.
8 cores cost about 17× as much (illustrative) and give 4.7×. The ceiling for this workload is 10.0× however big the machine gets, so the money is better spent removing the serial part.
Amdahl's law is exact for the model it describes. The price curve (cores^1.35) is a made-up illustration, not a quote.
Scale out
The puts both in one formula: speed-up(n) = n / (1 + α(n − 1) + β·n(n − 1)). With α = β = 0 the line is straight. With α alone it flattens out. With any β above zero it eventually bends down: past a certain size, adding a machine makes the whole system slower.
That is the real reason stateless tiers scale so well: they have almost nothing to share, so both numbers are close to zero.
Drag coherency up a little. The curve peaks and falls. Then bring both to zero: that is a stateless web tier.
The peak is at about 44 machines, past the end of this chart: up to 64 machines the curve is still rising, but at only 20% efficiency.
Computed from the Universal Scalability Law. α and β are the knobs you would fit from load tests of a real system.
Stateless vs stateful
The simulation sends 200 users × 10 requests each through a balancer. A request that reaches a server holding no session for that user forces a re-login. Try each mode, then crash server 1 halfway through.
Sticky looks fine until the crash. Shared costs one extra network hop but is the only mode with zero re-logins.
Requests handled in the second half of the run (server 1 stops at the halfway mark when it crashes).
Without stickiness a user is logged in on only the server that handled their first request. Re-logins: 554 of 2000 requests. This is why in-memory sessions forced people to invent sticky sessions.
Simulated with a seeded random generator: same inputs always give the same numbers.
The data tier
Say one machine does 10K operations per second (an illustrative number). A share r of the work is reads, spread over n copies. Each copy handles T·r/n reads plus all T·(1−r) writes, so the total throughput is T = C / (r/n + (1 − r)). However many replicas you add, T can never pass C / (1 − r).
That ceiling is why a write-heavy system needs a different tool: splitting the data itself, called . You will meet it two lessons from now.
At 90% reads, ten copies give about 5.3×. At 50% reads, nothing you add gets past 2×.
With 90% reads, no number of replicas gets past 10.0×, because each copy still repeats every write. 4 copies reach 3.08× (77% of ideal). Read-heavy workloads such as catalogues and feeds are exactly where replicas pay off.
The formula is exact for the model (reads free to go anywhere, every write applied on every copy). Replica reads may be slightly stale: the Replication lesson shows how stale.
How many servers?
The response time column uses the M/M/1 queueing formula S / (1 − ρ) with a 20 ms service time (illustrative). At 50% load a request takes 40 ms; at 90%, 200 ms; at 99%, 2 seconds. That curve is the real reason for headroom, and the Latency lesson explores it in depth.
Set spares to 0 and the target to 90%, then look at what one failure does.
One server can fail at peak and the rest still run at 59% (about 49 ms per request).
Server counts and utilisation are computed; the 20 ms service time and the per-server capacity are illustrative.
Where should this state live?
Not all state is bad. The skill is knowing which state must be shared, which can stay local, and which does not belong near a server at all. Each answer is worth 20 XP.
Ask: can it be rebuilt? and must every server see the same value?
A user's login session on a web tier of 20 servers behind a load balancer.
Tip: press 1 to 3 to answer, Enter for the next one.
Who lives by this
Four systems and what they have published about it.
vertically scaled database → many small instances → AWS
Netflix has described a 2008 database corruption that stopped DVD shipments for days. It was the trigger to leave a vertically scaled data centre setup, and to rebuild on many horizontally scaled, mostly stateless services in the cloud so that no single machine could stop the business.
request → any process → backing service (state)
The twelve-factor methodology from Heroku says processes are stateless and share-nothing, any data that must persist is kept in a backing service, and sticky sessions are a violation. Session data belongs in a store such as Memcached or Redis with an expiry.
phone → Erlang servers, each holding a million+ connections
WhatsApp's engineers published that single servers held more than a million simultaneous connections, tuning the operating system and Erlang runtime so each machine does as much as possible. Scaling up has real strengths: fewer machines to run and less coordination.
request → small web tier → big SQL Server + Redis
Stack Overflow's published architecture shows a small number of powerful on-premises servers: web servers in front, SQL Server scaled up with replicas behind, and Redis as a cache. It is a reminder that vertical scaling plus caching takes a well-built system a very long way.
Interview angle
Six questions, each worth 20 XP. “How would you scale this?” opens half of all design interviews: have the vocabulary and the numbers ready.
Start with the stateless tier (easy), then go straight to the hard part: where does the state live, and what is the shared bottleneck?
What is the main reason a stateless web tier scales out easily?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
The result is perverse: the web tier's capacity grows with every server, but what the system actually serves stops growing and then drops. Two fixes. Shrink each pool, or put a pooler (PgBouncer is the well-known one) between the servers and the database so thousands of client connections share a few dozen real ones.
Here each web server could take 200 requests a second. The connection limit (100) is from the PostgreSQL docs; the database's speed-per-connection curve is illustrative (it follows the Universal Scalability Law from earlier).
Slide servers up to 40 with no pooler. Then switch the pooler on at about 40 connections.
80 connections fit under the limit of 100. Served 800/s of 800/s possible. Add servers and watch the limit arrive.
The limit (100) is PostgreSQL's documented default. Per-connection speed (100 queries/s) and the contention constants are illustrative.