Architecture Patterns · lesson 7 of 7 · about 28 min
A proxy next to every service that handles encryption, retries, routing and metrics: what it adds to each call, how identity and certificates work, why retries at every layer multiply, how weighted routing and metrics behave, and how one bad configuration can reach every proxy at once. Tap any for a plain-language card.
A proxy beside every service
It is not free. Each call crosses two proxies (the caller's and the callee's), adding a little latency per hop, and every pod carries a proxy that uses memory and CPU. Both grow with call depth and with the number of pods. The proxy cost below is an assumption you can move; measure your own mesh before trusting any number.
A 6-call chain of 20 ms services with 1.5 ms per proxy crossing adds 18 ms (15%). With 2,000 pods at 50 MB per sidecar the proxies hold about 98 GB of memory.
A chain of 6 calls crosses 12 sidecars, adding about 18.0 ms (15% of 120 ms). 2,000 pods carrying 50 MB each hold 98 GB, and 5,000 requests a second need about 30 vCPU of proxy at an assumed 0.5 vCPU per thousand proxied requests per second.
Illustrative arithmetic: latency added = 2 × calls × proxy time; CPU = proxied calls per second × 0.5 vCPU per 1,000. Real proxy costs vary by product, features and payload size.
Who is really calling?
The control plane runs a certificate authority and issues short-lived certificates to each sidecar, renewing them automatically. Short lifetimes limit the damage of a stolen key but mean constant issuance, and a dependency on the CA: if it is down, certificates start expiring after roughly (1 − renewal point) × lifetime and everything fails closed once they all have. Choose the lifetime knowing how long you can survive a CA outage.
2,000 pods, 24 h certificates renewed at half life: about one new certificate every 22 s. If the CA goes down, the first workloads lose their identity after 12 h and all of them by 24 h.
2,000 pods with 24 h certificates renewed at 50% of their life need a new certificate every 22 s. If the certificate authority fails, the first workloads lose their identity after 12 h and every workload after 24 h.
Arithmetic: issuance rate = pods ÷ (renewal point × lifetime); earliest failure = (1 − renewal point) × lifetime; all failed = lifetime.
Helpful at one layer, harmful at four
The remedies are to retry at one layer only (usually the one closest to the caller who knows the call is safe to repeat), to cap retries with a such as “at most 20% extra requests”, and to retry only idempotent calls on transient errors. Compare the numbers.
4 layers each retrying twice can turn 1 user request into up to 81 attempts at the bottom. With a 20% budget per layer the same stack is limited to about 2.1.
4 layers each retrying 2 times can send up to 81 attempts to the bottom service for one user request when it is failing. A 20% retry budget at each layer caps that at 2.1.
Worst-case arithmetic: (1 + retries)^layers if every attempt fails, versus (1 + budget)^layers if each layer is limited to a fraction of extra requests.
Routing by weight
A small weight limits the harm but also slows the evidence. To believe a canary is bad you need to see enough errors, and the time to collect them is errors needed ÷ (traffic × weight × error rate). Interestingly the number of unlucky users is the same whatever the weight (it is just the errors you wait for); the weight changes how long the bad version stays live. Move the sliders.
At 100 requests a second, a 1% canary with a 5% error rate needs 400 seconds to show 20 errors. At 10% weight it takes 40 seconds, and the number of failed requests is the same 20.
With 1% of 100 req/s on a canary failing 5% of the time, 20 errors take 7 min to appear, during which 400 canary requests are served and 20 users see an error. A bigger weight shortens the wait but not the count.
Expected-value arithmetic: time = errors ÷ (rps × weight × error rate). Statistical confidence also needs a healthy baseline to compare with.
Golden signals without touching the code
So decide the labels deliberately: aggregate by service rather than by pod, limit routes to templates (“/orders/:id”, not each id), keep histogram buckets few, and drop what no dashboard or alert uses. The memory per series below is an assumed rough figure; check what your own metrics store really uses.
100 services with 5 dependencies each, 10 routes, 3 status classes and 20 buckets give about 330,000 series (~1 GB). Labelling per pod with 10 replicas each side multiplies that by 100: 33 million series (~99 GB).
100 services calling 5 others each make 500 edges. With 10 routes, 3 status classes and 20 buckets, that is 330K time series, roughly 1.0 GB at 3 KB each.
Series = edges × routes × status classes × (buckets + 2 for sum and count), × replicas² if labelled per pod. The 3 KB per series is an assumption for illustration.
Which tool?
A mesh solves a real problem at a real cost. Choose the lightest tool that fits. Each answer is worth 20 XP.
Ask: how many services and languages, who is calling (outside or inside), and what must be uniform?
Eight services in three languages must all use encrypted, authenticated connections with the same retry and timeout policy.
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Published origins of the service mesh.
a high-performance L7 proxy
Lyft built Envoy to give its many services uniform networking, observability and resilience regardless of language, and open-sourced it in 2016; it became a Cloud Native Computing Foundation project and the data plane for many meshes.
Google, IBM and Lyft, 2017
Istio, announced in 2017 by Google, IBM and Lyft, added a control plane to Envoy sidecars to distribute configuration, issue workload certificates and apply routing and security policy across a fleet.
a lightweight mesh
Buoyant's Linkerd helped popularise the term 'service mesh', and its second generation uses a purpose-built lightweight proxy with the aim of keeping the overhead and operational burden low.
sidecar-less mesh
Istio's ambient mode moves much of the proxying from a sidecar in every pod to shared per-node and per-namespace components, in response to the memory, CPU and upgrade cost of running a proxy beside every workload.
Interview angle
Six questions, each worth 20 XP. Good answers know what a mesh costs.
Name the benefit (uniform behaviour, identity) and the cost (latency, resources, blast radius).
What do the sidecars do in a service mesh?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
Treat mesh configuration like a release: validate and lint it before applying, push to a small slice of proxies first and soak, then widen, with automatic rollback on error-rate alarms. Know your control plane's failure mode too: sidecars should , carrying on with the last good configuration if the control plane is unreachable, instead of failing closed.
2,000 proxies, a push rate of 200 per second and a problem noticed after 60 seconds: pushing to everyone breaks all 2,000. Staged 1% then 10% then 100%, the same 60 seconds affects only 20.
Pushed to every proxy at 200/s, the bad rule had reached 2,000 of 2,000 proxies (100%) by the time it was noticed after 60 s.
Deterministic arithmetic with a 120 s soak between stages. Faster detection and smaller first stages shrink the blast radius.
Where to go next