Networking & Traffic · lesson 5 of 7 · about 25 min
How to stop one greedy client from hurting everyone else: the four classic algorithms (and exactly where each one cheats), how to share a limit across many servers with Redis, what to answer when you say no, and how retries can turn a spike into an outage. Tap any for a plain-language card.
Why limit requests
Limiters protect three things: availability (one noisy client cannot take the service down), fairness (each client gets a share), and cost (you pay for every call to a database, an LLM or an SMS gateway). They also blunt brute-force logins and scraping.
Here the server handles 100 requests per second. Normal users together send 60. Raise the scraper's rate, then turn the limiter on.
Without a limiter the scraper does not just hurt itself: past 100/s everybody's success rate falls together.
Under capacity everyone is served. Raise the scraper's rate and see what changes.
The simplest limiter
The flaw: the reset happens for everyone at the same instant. A client can send the full limit at the very end of one window and the full limit again at the very start of the next. The server sees twice the limit in a few hundred milliseconds, and the limiter thinks everything is fine.
Here the limit is 10 per second and the client sends 20 requests in 200 ms. Slide the burst across the 2-second boundary.
Put the burst in the middle of the boundary (around 0 ms). Then try it fully inside one window.
The burst straddles the boundary, so each side sees fewer than 10. 20 requests got through inside 200 ms, against a promised limit of 10 per second. A sliding window would have allowed only 10.
Allow bursts, cap the average
That matches how real clients behave: quiet, then a flurry (a page loading ten API calls). It is the algorithm Stripe has described using for its request rate limiter, and the one behind most API gateways. It needs just two numbers per client: the token count and the time it was last updated. Tokens are not added by a timer; the code computes how many arrived since the last request.
Below the bucket really refills. Fire single requests, or ten at once, then wait and fire again.
B decides how big a burst you tolerate. R decides the long-run speed. They are separate dials.
A full bucket: you can burst up to 10 immediately, then about 2 per second after that.
Algorithm lab
The number under each row is the most requests accepted in any single second. A strict limiter never exceeds the limit. Watch which ones do, and when that is by design (the bucket allows bursts) and when it is a bug (the fixed window at its boundary).
Try “Boundary attack” first. Then “One sudden burst”, where the token bucket is the only one that welcomes it.
10 requests just before a second ticks over, 10 just after. (zoomed in: the dashed line is the second boundary)
Fixed window accepts every request and lets 20 through inside about 300 ms. The three others hold the line near 10.
Many servers, one limit
1. The race. If the code does read the counter, check it, write it back, two requests can read the same value before either writes. Both pass. Use one step instead: INCR returns the new number in a single indivisible operation, or a small Lua script that checks and updates together.
2. Local counters. If each server counts alone (limit ÷ servers each), a lucky or unlucky spread makes one server reject a client who is still under the global limit. A shared counter fixes accuracy at the price of one extra network hop per request.
Rate limiting is where Redis's one-thread, atomic commands pay off. This is the lesson to remember.
The counter stands at 99 of 100. 4 requests arrive at the same instant on different servers.
All 4 requests read 99, all decided "under the limit", and all wrote back 100. 4 got through when only 1 should have, and the counter lost 3 updates.
What to limit by, what to answer
Choosing the algorithm is the easy half. The harder half is choosing the key, the failure mode and the response. Each answer is worth 20 XP.
Ask: who is the real caller? and what happens when the limiter itself fails?
A login endpoint is attacked: millions of password guesses against one account, sent from thousands of different IP addresses.
Tip: press 1 to 3 to answer, Enter for the next one.
Who lives by this
Four real systems and the design choices they have published.
request → rate limiter → concurrency limiter → load shedders → API
Stripe's engineering blog describes layering a token-bucket request rate limiter, a limiter on concurrent requests, and two load shedders that protect the fleet by dropping the least important traffic first. Different problems (a heavy user, a slow endpoint, a sick fleet) need different guards.
request → nearest edge server → sliding window counter → block or pass
Cloudflare has described choosing the sliding window counter for its rate limiting because it needs only two numbers per client. Measured on a sample of 400 million requests, about 0.003% were wrongly allowed or blocked.
token → REST API → rate-limit headers on every response
GitHub's REST API documents 5,000 requests per hour for an authenticated user and far fewer for anonymous callers, and returns headers (x-ratelimit-limit, remaining, reset) so clients can pace themselves instead of guessing.
client → API Gateway (token bucket per account and stage) → your backend
AWS documents API Gateway throttling as a token bucket: a steady-state rate plus a burst size. Requests over the limit get a 429, and you can set stricter limits per client through usage plans and API keys.
Interview angle
Six questions, each worth 20 XP. “Design a rate limiter” is one of the most common system design questions, so know the algorithms and the distributed catches cold.
Name the algorithm, say where the counter lives, make the update atomic, then say what happens when the counter store is down.
A fixed window limiter allows 100 requests per minute. About how many requests can one client get through in a few seconds around the window boundary?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
Two defences, both needed. On the server: refuse early. A limiter or bounded queue that answers 429 Too Many Requests straight away costs almost nothing and keeps the queue short. On the client: (wait longer after each failure) and add (randomness) so thousands of clients do not return on the same tick. A Retry-After header tells them when.
Server: 100 requests per second. Normal load: 80. For two seconds a spike sends 200. Clients time out after 1 s and try up to 5 times.
First try all four client policies with no limiter. Then switch the limiter on and try them again.
The queue never drains: 34 seconds of backlog remain at the end, 1090 answers went to clients that had already given up, and only 26% of requests were ever served. Backoff and jitter cannot save a server that has no way to say no.
Simulated with a queue, a 1-second client timeout and up to 5 attempts. Numbers come from the model, not from a real server.