Networking & Traffic · lesson 3 of 7 · about 30 min
The front door of a system: what a reverse proxy does, why every service should not repeat authentication and rate limiting, the pipeline a request passes through, a real signed token you can attack, canary routing, aggregation, and how one slow route can starve the rest. Tap any for a plain-language card.
Proxies
System design is almost always about the reverse kind. Once a reverse proxy stands in front of your servers, it is the natural place for work that every request needs: TLS, load balancing, caching, compression, rate limiting. An is exactly that: a reverse proxy that also understands APIs: who is calling (authentication), how much they may call (quotas), which service owns each URL (routing) and how to reshape requests and responses.
Flip between the two and watch what each side can and cannot see.
Examples: Corporate web proxies, Squid, many VPN and privacy tools.
The client chooses to use it (or its network forces it). The websites do not know it exists.
Why a gateway
An moves these cross-cutting concerns to one place at the edge. Services behind it can trust that a request has already been authenticated and rate limited, and get on with their real job. New services get the protections for free.
It is not free: it adds a network hop (typically a few milliseconds), and it becomes critical infrastructure that must itself be replicated. Choose the concerns and the number of services and compare.
Set 15 services and rotate a signing key. Without a gateway that is 15 deployments.
8 services × 4 concerns = 32 separate implementations that must all agree. Rotating one secret means touching 8 services, and the one that is forgotten is the weak link.
Inside the gateway
That is the real value: reject early. Bad requests (no token, expired token, over quota) are the cheapest to refuse at the front door and the most wasteful to let through, because the backend would do database lookups only to say no. Cached answers short-circuit even more.
Pick a kind of request and see where it stops. Then switch off authentication and rate limiting at the gateway, and see how much junk reaches the backend.
With the checks on, only requests that are valid and not cached reach the backend: 30% of this traffic mix.
Handled in 0.8 ms without bothering the backend. Across the traffic mix only 30% of requests reach it.
Stage times are illustrative. Traffic mix: 30% valid new, 30% cacheable repeats, 20% no token, 10% expired, 10% over quota.
Authentication at the edge
Two things to remember. The payload is only encoded, not encrypted: anyone can read it (decode it below), so never put secrets in a JWT. And the signature protects integrity, not secrecy: change one character of the payload and verification fails. Also check the exp time, because a stolen token works until it expires, which is why access tokens are short-lived.
This uses your browser's real HMAC-SHA-256. Issue a token, then try each attack.
Press “Issue a token” to create a real signed token.
Routing and releases
There is a trade-off. A smaller canary harms fewer users, but sees fewer requests, so it takes longer to collect enough evidence that something is wrong. A large canary notices quickly and hurts many. Here an alarm fires once the canary has returned 20 errors.
The new version is broken and fails some share of requests. 10,000 requests arrive every minute.
Compare releasing to 1%, 5% and 100%. At 100% there is no canary at all.
Only 5% reached the broken version, so users saw 150 errors a minute instead of 3,000. The alarm trips quickly and the gateway can route everyone back to the old version.
Fewer, fatter calls
A gateway can aggregate: the app makes one request, and the gateway fans out to the services over the fast network inside the data centre (about a millisecond away), combines the results, and replies once. This pattern is also called a when a separate gateway is built for each kind of client, so the phone gets a small payload and the web app a larger one.
The gateway can also translate protocols (a REST call from the phone becomes gRPC inside) and trim responses.
Switch on “calls depend on each other” and raise the round-trip time. The gap becomes dramatic.
Aggregating saves about 475 ms per screen load (63% faster), and the phone sets up one connection instead of 5.
A simple model: each backend call does 30 ms of work; a hop inside the data centre costs 1 ms.
Gateway or service?
A gateway is useful because it is thin. Putting business logic in it is the classic mistake. Eight jobs, 20 XP each.
Ask: does every service need this? and does it need business knowledge? Yes then no: gateway.
Checking that a request carries a valid, unexpired signed token.
Tip: press 1 to 3 to answer, Enter for the next one.
Who lives by this
Four real gateways and proxies.
devices → Zuul (filters: auth, routing, resilience) → hundreds of services
Netflix built Zuul as the front door for traffic from its devices. It is a chain of filters for authentication, dynamic routing, load shedding and monitoring, and Netflix has written about moving it to a non-blocking design to hold many more connections.
client → Kong (plugins: auth, rate limit, transform) → upstream services
Kong is an open-source API gateway built on NGINX. Features such as authentication, rate limiting, request transformation and logging are plugins you switch on per route or per consumer, which is the gateway pipeline from this lesson as configuration.
client → API Gateway (authorizer, throttling, cache) → Lambda, HTTP or AWS services
AWS's managed gateway handles authorisation (including custom authorizers), throttling with a token bucket, response caching, stages for versions and canary releases, and a usage plan with API keys per customer, with nothing to patch or scale yourself.
service A → Envoy sidecar → Envoy sidecar → service B
Envoy is a layer 7 proxy created at Lyft and now widely used, including inside service meshes such as Istio. It provides retries, timeouts, circuit breaking and rich metrics for traffic between services, which is the same job as a gateway but for east-west traffic.
Interview angle
Six questions, each worth 20 XP. Gateway questions are about what to put in it, what to keep out, and how it fails.
Draw the gateway, then say: replicated, thin, and with timeouts and bulkheads, so it is not a new single point of failure.
What is the difference between a forward proxy and a reverse proxy?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
The defences are the same ones you will meet in the reliability path: a timeout so no request holds a connection for ever, a that gives each route its own small pool so one cannot take all of it, and a that stops calling a backend that is clearly failing and returns an error at once.
Make /reports slow, watch the healthy routes fall, then try each fix.
Demand is 266 connections against a pool of 100, so only about 38% of requests get a connection in time. The healthy routes fail because of /reports, which is slow and a minor feature.
Little's law, with illustrative numbers: /search and /catalog 400 requests/s at 20 ms, /reports 50 requests/s, pool of 100 connections.