Distributed Systems Core · lesson 6 of 11 · about 28 min
How services find each other when machines come and go: why a hard-coded list rots, what DNS, polling and pushed watches cost in stale calls, the lifecycle mistakes that fail requests during every deploy, how a registry avoids evicting a healthy fleet, and what to do when the registry itself is down or an address has been reused. Tap any for a plain-language card.
Instances come and go
Say 50% of the fleet is replaced every hour (a busy autoscaling or deploy cadence). The graph shows the share of requests a client with a fixed list sends to dead instances as the day goes on, and the small constant error of a registry whose entries lag by the heartbeat TTL plus half the clients' refresh interval. Move the sliders.
At 50% churn per hour, a fixed list sends most traffic to dead instances within a few hours. A registry stays under 1%.
With 50% of the fleet replaced an hour, a list that is 6 h old points 95.0% of requests at instances that are gone. A registry that expires dead entries after 30 s and is re-read every 10 s has a lag of about 35 s, so only 0.49% of requests are affected. Retries and health-aware load balancing shave that further.
Fixed list: share of original instances still alive = e^(−churn × hours). Registry: churn rate × lag. Both are expectations, ignoring bursts.
How clients find out
While the window is open, a share 1 ÷ N of requests picks the dead instance. If the client on a different instance, the chance every attempt fails shrinks to (1 ÷ N)^(retries + 1). So the cheap partner of discovery is a client that retries idempotent calls and ejects instances that fail. Compare the three over one instance death.
Set 10 instances and zero retries, then turn retries up to 2.
After an instance dies out of 10, clients keep calling it for about 60 s with DNS, 35 s with a polled registry and 30.3 s with a pushed watch. With 0 retries, 10.0% of requests in the window fail: about 30,000 failed requests with DNS against 15,150 with a watch.
Window = detection time plus staleness (half the DNS TTL, half the poll interval, or about 0.3 s for push). Failed share = (1 ÷ N)^(retries + 1). Assumes retried calls are safe to repeat.
The lifecycle
Toggle the three mistakes and count the failed requests across a full rolling deploy of the fleet. The fixes are all cheap: become ready only when you can serve, deregister first and wait a moment, then finish in-flight requests before exiting.
With all three mistakes on, a 10-instance deploy at 2,000 requests a second loses thousands of requests. Turn them off one by one.
Across 10 instances this deploy fails about 100,600 requests: 40,000 sent to instances still starting, 60,000 sent to dead instances that were still listed, and 600 cut off mid-flight. None of it is a bug in the service's logic.
Per instance: share of traffic (rps ÷ N) × the length of each failure window; a mid-flight request lasts 300 ms. The deploy replaces every instance once.
When the registry is wrong
The trade is deliberate. In that mode, a few instances that really died stay listed (a small, bounded error), but the fleet that is fine stays reachable. Availability of the list is chosen over its accuracy: Eureka is an AP system by design, where ZooKeeper-based discovery chooses the opposite.
Set 40% of heartbeats lost with self-preservation off: 80 healthy instances evicted. Then switch it on.
40% of heartbeats were lost to the network, and the registry evicted 80 healthy instances: clients can reach only 58% of the fleet, although most of it never failed.
Renewals expected = N. Self-preservation applies when renewals received fall below threshold × N. Arithmetic on your inputs; Eureka's default renewal threshold is 0.85.
Who discovers the discoverer?
Compare the two behaviours across a registry outage. With no cache, every call fails for the whole outage. With a cache, calls fail only when they hit an instance that was replaced since the last good refresh, and clients retry around those. This is also why a registry should be a deliberately: serving a slightly old answer during a partition (AP) is usually the better trade for discovery.
Set a 10-minute outage with 20% of the fleet churning an hour, first with no cache, then with.
Looking up on every call, a 10-minute registry outage fails 100% of requests for 10 minutes: the registry is a single point of failure for the whole system.
Cached list: failing share = 1 − e^(−churn rate × time since the last good refresh). No cache: every request needs the registry.
Which discovery?
Each style puts the lookup in a different place. Pick the one that fits. Each answer is worth 20 XP.
Ask: who can run the lookup logic? and how fast must changes spread?
A public website reachable at a stable name, with a few servers that change rarely.
Tip: press 1 to 3 to answer, Enter for the next one.
Who lives by this
Four real discovery systems and what they have published.
instances renew every 30 s · evict after 90 s · self-preservation
Netflix built Eureka as an AP registry for client-side discovery on AWS: instances renew their lease (every 30 seconds by default), unrenewed ones expire, and if the share of renewals drops below a threshold (85% by default) the server enters self-preservation and stops evicting, preferring a stale list to losing healthy instances in a network blip.
agents · Raft servers · DNS and HTTP interface
Consul keeps a catalogue of services and runs health checks through agents on each node, with a small Raft-based server cluster as the source of truth. Clients can look services up over HTTP or through a DNS interface, and unhealthy instances are removed from the answers automatically.
Pod readiness → EndpointSlice → kube-proxy / CoreDNS
In Kubernetes, a Service has a stable virtual IP and DNS name, and the control plane tracks the pods behind it as EndpointSlices, including only pods that pass their readiness probe. kube-proxy (or a mesh) updates its routing rules when the endpoints change, watching the API server rather than polling.
control plane streams endpoints to proxies
Envoy, created at Lyft, popularised xDS: a family of APIs by which a control plane streams clusters, endpoints and routes to every proxy over gRPC. Changes in the fleet reach every sidecar in moments without clients doing any lookups, which is the pushed-watch model in this lesson at large scale.
Interview angle
Six questions, each worth 20 XP. “How do your microservices find each other?” is a standard follow-up in any microservice design.
Name the registry, how instances register and expire, how clients learn changes, and what happens if the registry is down.
Why is a hard-coded list of server addresses a poor approach in the cloud?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
The risk is the product of two things: how likely a freed address is reused within the staleness window, and how long clients keep it. The window is exactly what the previous scenes measured. Defences: shrink the window (push-based discovery, short TTLs), authenticate the peer (mutual TLS with a service identity, so a call to the wrong service fails the handshake instead of being answered), and delay address reuse.
With a 60 s stale window, 30% hourly reuse and 40 deaths a day, a few misrouted requests a day. Then cut the window to 5 seconds.
A freed address is reused within a 60 s window with probability 39.35%. At 500 requests a second, an unlucky death misroutes about 5,902 requests, and across 40 deaths a day that is roughly 236,082 requests reaching the wrong service. Halve the window and the damage falls in proportion.
P(reuse) = 1 − e^(−rate × window); on average the reuse happens halfway through the window. Expected values; real reuse is bursty during deploys.
Where to go next