Queues & Streaming · lesson 4 of 5 · about 28 min
How services talk through events instead of calls: what it buys in latency and availability, how fan-out and consumer lag behave, why saving to a database and publishing are a dual-write trap and how the outbox fixes it, how to evolve event schemas, why a log can be replayed, and what happens when events form a cycle. Tap any for a plain-language card.
Publish and subscribe
In the order service records one fact, “OrderPlaced”, to a topic and returns. Any number of subscribers receive it independently, each at its own pace; a new one subscribes without the publisher knowing or caring. The price is that the work becomes asynchronous: the e-mail goes out a moment later, and you must design for events being delayed, repeated or reordered. Set how many services are involved.
With six downstream services at 30 ms each and 99.9% availability, the synchronous checkout takes 210 ms and is down 5 hours a month. The event-based one takes 32 ms.
Calling 6 services in turn takes 210 ms and works only when all of them are up (99.30%). Publishing one event takes about 32 ms and depends only on the broker's own availability; the 6 services consume it independently, and a 7th needs no change to the publisher.
Synchronous latency = (services + 1) × call time; availability = a^(services + 1). Event path = one publish plus 2 ms. The broker is assumed highly available.
Fan-out
A burst (a sale, a retry storm, a backfill) makes lag jump even for a healthy consumer, and the headroom decides how long it takes to recover. A consumer that is slower than the average rate never catches up: its lag grows without bound, its messages get older, and eventually retention deletes them before they are read. Watch lag, not just errors. Chart: the backlog of one subscriber (black) over two minutes with a burst in the middle.
Set the consumer to 1,200/s against 1,000/s arrivals and a 3× burst: it recovers slowly. Drop it below 1,000/s and the backlog never drains.
The burst left 4,800 messages waiting; with 200/s of spare capacity they drain in about 24 s.
Backlog(t+1) = max(0, backlog(t) + arrivals − processed), one-second steps; the dashed line is a subscriber with three times the average arrival rate.
The outbox
The pattern avoids it. In the same database transaction as the order, insert a row into an outbox table describing the event. A separate relay process reads the table (or the database's change log) and publishes to the broker, marking rows sent. The order and its event now exist together or not at all. The relay may publish a row twice if it crashes before marking it, so delivery is at-least-once and consumers must be idempotent, which they should be anyway.
With 1 crash in 1,000 updates, dual writes lose or invent about a hundred events per 100,000 updates. The outbox produces only harmless duplicates.
Across 10,000,000 updates with a 10 in 10,000 crash rate, about 4,300 changes are never announced and 5,200 announcements describe changes that did not happen: the services drift out of agreement with no error anywhere.
100,000 seeded updates scaled to the chosen volume. In a dual write a crash in the gap loses the event half the time and creates a ghost half the time; the outbox relay may repeat a row but never drops one.
Event schemas
A schema registry (Confluent's, for Avro and Protobuf) checks every new schema against the previous versions and rejects incompatible ones before they ship. The reliable habits are: add optional fields with defaults, never rename or retype a field (add a new one and retire the old over several releases), and treat required fields as a promise you can never take back. Pick a change and a deployment order.
“Rename a field” breaks in both orders. “Add an optional field” is safe either way.
Rename a field: A rename is a remove plus an add: both directions break. Deploying the publisher first requires forward compatibility, so 12 subscribers would start failing.
Rule-based, following the compatibility definitions of Avro and Protobuf schema evolution.
Replay
Replay takes time: events divided by consumption speed. A month of a busy topic can take hours to read through, which is why logs are compacted (keep only the latest event per key) or paired with periodic snapshots, so a new consumer starts from a snapshot and replays only the tail. Retention is a decision: keep too little and a slow or new consumer cannot catch up; keep it forever and storage grows without end.
A billion events replayed at 200,000 per second take 83 minutes. At 10,000 per second they take more than a day.
Replaying 1,000,000,000 events at 200,000 a second takes 1.4 h. Fast enough to bootstrap a new consumer from scratch.
Replay time = events ÷ consumption rate. Arrival rate assumes the log holds one retention period of events.
Which style?
A call needs an answer now, a queue shares work, a topic tells many, and a log remembers for later. Each answer is worth 20 XP.
Ask: do I need the answer now? one worker or many readers? must late readers see history?
The checkout page needs the current price of an item before it can render.
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Four published sources on event-driven design.
'The Log: What every software engineer should know about real-time data's unifying abstraction'
Kreps argued that an ordered, append-only log of changes is the natural backbone for data integration: every system subscribes to the log and derives its own view. It describes the thinking behind Kafka at LinkedIn.
topics with independent subscriptions
Cloud messaging services let one published message fan out to many subscribers, each with its own durable queue or subscription so that a slow subscriber does not hold up the others; AWS's common pattern is an SNS topic delivering into one SQS queue per consumer.
transactional outbox
The transactional outbox pattern writes events into an outbox table in the same local transaction as the state change and has a relay publish them, avoiding dual writes; Debezium's outbox event router supports it by reading the table through change data capture.
compatibility checks for Avro and Protobuf
A schema registry stores every version of an event schema and refuses a new version that violates the configured compatibility mode (backward, forward or full), so incompatible changes are caught before they reach producers.
Interview angle
Six questions, each worth 20 XP. Event-driven design is a favourite way to decouple services in system-design answers.
Name the trade-off: you gain decoupling and resilience, you lose immediacy and simple reasoning about flow.
What is the main benefit of publishing an event instead of calling each service?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
Defences: draw the event flow so cycles are visible; make handlers idempotent and state-based (if the new value equals the stored one, emit nothing: a no-op update must not publish); put a hop count or causation id in each event and drop events beyond a depth; use on consumers; and alert on consumer lag and topic growth so the loop shows up in minutes, not at disk-full.
At g = 1.1 each original event becomes 57 after 20 hops and keeps climbing. At g = 0.9 it settles near 8.
Each event causes 1.10 more, so one original becomes 64 events within 20 hops and the count never levels off: 100 originals a second grows to about 673 a second at the last hop. This is how a topic fills a disk.
Geometric series: total = (1 − g^(hops+1)) ÷ (1 − g), and 1 ÷ (1 − g) in the limit for g below 1.
Where to go next