Architecture Patterns · lesson 4 of 7 · about 28 min
Services that react to events instead of being called: what an event should carry, who runs a multi-step process, why users sometimes read stale state right after writing, how skewed keys create hot partitions, how to follow one event through the system, and what a single poison event does to a partition. Tap any for a plain-language card.
Events as the backbone
Neither is right everywhere. Notifications suit rare, large or sensitive data; state transfer suits data many consumers want and tolerate being slightly old. A third form, event sourcing, makes the events themselves the record of truth (see the CQRS lesson). Compare the load of the first two for your numbers.
With 8 consumers, notifications create 8,000 callbacks a second back to the publisher; state transfer creates none.
1,000 events a second to 8 consumers: notifications make 8,000 callback requests a second to the publisher (41.8 MB/s in all); state-carrying events make none (41.6 MB/s). The price of state transfer is that each consumer holds its own copy, which can be a little out of date.
Notification: small event + one callback per consumer. State transfer: the event carries the data. The two move similar bytes; the difference is the callback load and the runtime dependency on the publisher.
Who runs the workflow
Choreography is elegant for short, simple reactions, and becomes opaque as flows grow: the process exists nowhere in the code, so answering “where is order 42?” means reading the logs of each service in turn. Orchestration costs a central component and more messages, and makes the flow visible, testable and recoverable in one place. A good rule: choreograph simple fan-out notifications; orchestrate multi-step business processes with failure handling.
Make the process 8 steps with a failure at step 6: choreography needs six services' traces to find it, orchestration one status page.
A 6-step process stuck at step 4: with choreography an engineer follows the event trail through 4 services; with orchestration the coordinator's state names the stuck step at once. Orchestration sends 12 messages instead of 6 and adds a central component, the price of visibility and control.
Counts of messages and inspection points, not times. In practice choreographed flows are made traceable with correlation ids.
Eventual consistency, felt by users
Whether it happens is a race between the read model's lag and how soon the user looks again. Remedies, from simplest: return the new state in the write's own response and render it optimistically without re-reading; route the next reads to the write side for a short window; pass a version token so the reader can wait until the read model reaches it; or show an honest “saving…” state. Set the lag and the user's typical reload time.
A read model that is 500 ms behind shows stale data on about one reload in eight at a 1.5 s reload. At 3 s of lag it is most of the time.
With a read-model lag of about 500 ms and users reloading after about 1.5 s, 11.7% of reloads land before the update arrives. Rare enough to tolerate, but never zero.
30,000 seeded saves: lag and reload delay are both log-normal around the medians; a reload is stale when it happens before the lag ends.
Ordering has a price
The system is as fast as its busiest partition, not its average one. Adding more consumers does not help; adding partitions only helps if the hot key moves, and it cannot split, because splitting would break its ordering. Remedies: choose a key that spreads load (and only as fine-grained as the ordering you really need), add a random suffix to split a hot key when order across the pieces does not matter, and handle the largest tenants in dedicated partitions. Set the skew of your keys.
With Zipf-skewed keys over 16 partitions, the busiest partition carries about 15% of the traffic against an ideal 6%, so the consumers are used at only about 40% of their capacity.
With 5,000 keys of skew 1.0 over 16 partitions, the busiest partition carries 15.1% of events against an ideal 6.3%, so the whole pipeline runs at about 41% of the throughput its 16 consumers could deliver.
Keys have Zipf popularity and are hashed at random to partitions; efficiency = average partition load ÷ busiest partition load.
Debuggability
Without them an engineer greps by timestamps and customer ids across services and guesses. The cost of adding the ids is trivial at the start and enormous to retrofit across forty services. Make correlation ids part of the event envelope standard from day one, propagate them in every message header, and log them in every line. Set the shape of the flow.
With fan-out of 3 over 8 hops there are almost 3,300 events per original. With a correlation id you read 8; without, you search them all.
One original request caused 3,280 events over 8 hops with a fan-out of 3. Without a correlation id, finding the one chain that failed means sifting through them: about 1.8 hours at 2 seconds each.
Events = (fan^hops − 1) ÷ (fan − 1), a geometric series. Inspection assumes 2 seconds per event; with an id you read only the path.
Which design?
Notifications keep events small and consumers coupled by callbacks; state-carrying events decouple at the cost of copies; choreography is simple reactions; orchestration is a visible multi-step process. Each answer is worth 20 XP.
Ask: how big is the data, how many consumers, and does the process need compensation and visibility?
A sensitive medical record changed; interested systems should know but must fetch it with their own access checks.
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Four published sources on event-driven design.
'What do you mean by Event-Driven?'
Fowler separated four things often called event-driven: event notification, event-carried state transfer, event sourcing and CQRS, pointing out that people discussing 'events' often mean different ones and that each has distinct trade-offs.
durable workflow engines
Cadence, created at Uber, and its successor Temporal let developers write multi-step business processes as ordinary code whose progress is persisted so it survives crashes, with retries and timers handled by the engine: orchestration made durable.
a workflow orchestration engine
Netflix open-sourced Conductor, an orchestration engine that coordinates workflows across microservices from a central definition, after finding that purely event-driven choreography made processes hard to follow and manage as they grew.
correlation across services
The W3C Trace Context standard defines the traceparent header that carries a trace id from service to service, and OpenTelemetry propagates it through HTTP calls and message headers so that the events caused by one request can be joined into a single trace.
Interview angle
Six questions, each worth 20 XP. Event-driven answers are judged on whether you know the downsides: stale reads, ordering and debugging.
Name the kind of event, who coordinates, and how you debug and recover.
What is the drawback of a tiny event notification?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
The cure is a policy for failing events: retry a bounded number of times with backoff (many failures are transient), and then move the event to a with its error so a person can inspect and replay it, letting the partition continue. Alert on DLQ growth and on per-partition lag. The cost is losing strict order for that one event, which is almost always better than stopping everything.
With retry-forever, 500 events a second pile up for as long as the problem is unnoticed: 30 minutes is 900,000 delayed events. With 5 tries and a dead-letter queue the stall is 10 seconds.
The consumer retries one bad event until someone notices after 30 minutes: the partition is stalled for 30 min and 900,000 events wait behind it, none of them at fault.
Stall = attempts × backoff (dead-letter) or the time until someone notices (retry forever). Events delayed = arrival rate × stall.
Where to go next