Queues & Streaming · lesson 3 of 5 · about 28 min
How to make messaging safe when networks fail and consumers crash: at-most-once, at-least-once and the honest meaning of exactly-once, idempotency keys and their lifetime, operations that don't mind repeats, transactional pipelines, retries that reorder, and why the key and the effect must be written together. Tap any for a plain-language card.
Delivery guarantees
“Exactly-once delivery” over an unreliable network is impossible to promise without help (it is a version of the Two Generals problem). What systems offer is exactly-once effect: at-least-once delivery plus a consumer that makes duplicates harmless, with a dedup table or by being idempotent. Set a crash rate and compare the three ways of acknowledging.
At 1% crashes, half of which land in the dangerous window, ack-first loses about 5,000 per million, ack-after duplicates the same number, and the dedup table fixes it.
Acknowledging after the work never loses a message, but a crash after the effect and before the ack redelivers it: about 4,680 per million is done twice. For a payment that is a double charge.
100,000 seeded messages per strategy; a crash lands in the dangerous gap between work and ack half of the time. The dedup table is assumed to be updated atomically with the effect (see the last scene for what happens when it is not).
Idempotency keys
Remembering every key forever is expensive, so keys expire, and the lifetime must exceed how long a client can plausibly keep retrying (including mobile clients that come back online hours later). Stripe says keys may be removed after at least 24 hours; SQS FIFO deduplicates within a 5-minute window. A retry that arrives after the key has expired looks like a new request and is executed again. Below, retry delays are heavy-tailed: most arrive quickly, a few much later.
With a 5-minute key lifetime and a one-minute typical retry delay, a few percent of retries arrive too late. Stretch the lifetime to a day.
With keys remembered for 5 min and a typical retry delay of 1 min, 15.66% of retries arrive after the key has expired and run a second time. For payments, remember keys for at least a day.
Retry delays are log-normal (σ = 1.6) around the median; 60,000 seeded retries.
Designing for repeats
So a good design converts accumulating operations into set-like ones: record the fact with a natural or client-supplied id (payment_id), make the insert on that id being new, store the absolute value rather than a delta, or use a version number so an old message cannot overwrite a newer one. Choose an operation, a message count and a duplicate rate.
Pick the increment, 1,000,000 messages and a 1% duplicate rate: the balance is wrong by 100,000. Pick SET and nothing changes.
balance = balance + 10: Every repeat adds again. With 1% of 1,000,000 messages delivered twice, 10,000 duplicate effects occur, leaving the balance off by 100,000. Add an idempotency key, or restate the operation as a set.
Duplicate effects = messages × duplicate rate, for operations that accumulate; zero for operations that set state.
Exactly-once, honestly
Notice the scope. This works because both the output and the offset live inside the same system. The moment the effect is outside (a call to a payment API, an e-mail, a different database), the platform cannot make it atomic with the offset, and you are back to at-least-once delivery plus idempotent effects. “Exactly-once” is a guarantee about a pipeline, not about the world. Choose a design and read every crash point.
Compare the first two designs: each has exactly one dangerous crash point. The transactional design has none, but only inside the platform.
Write the output, then commit the offset: The effect happened, but the offset did not move, so the message is processed again. Every crash at that point produces a duplicate result.
Enumerated crash points of a read-process-write loop; the transactional design follows Kafka's exactly-once semantics (KIP-98).
Ordering
Order is only guaranteed within a partition: messages for one key must share a partition so they are consumed in order. Set the failure rate and how many requests are in flight, and measure how often the stream is out of order.
With 5 requests in flight and 5% failures, nearly 5% of messages arrive out of order. One in flight, or sequence numbers, brings it to zero.
With 5 requests in flight and 5% failing, 4.8% of messages overtake an earlier one that was being retried. Fix it with one request in flight or with sequence numbers.
10,000 seeded messages; a failed request is retried after a short delay (up to 5 times); a message is out of order when it is consumed before one that was sent earlier.
Which guarantee?
Choose the weakest guarantee that is actually safe, and make the effect repeatable. Each answer is worth 20 XP.
Ask: what is worse, a lost message or a repeated one? is the effect inside the messaging system?
Sampling click events for a rough dashboard, where losing a few is fine.
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Four published sources on delivery guarantees.
idempotent producer and transactions
Kafka added an idempotent producer (each producer has an id and sequence numbers per partition so retries are not duplicated) and transactions that atomically write output records and consumer offsets, which gives exactly-once processing for read-process-write pipelines inside Kafka.
standard and FIFO queues
SQS standard queues deliver at least once and may deliver messages out of order; FIFO queues preserve order within a message group and deduplicate messages with the same deduplication id within a five-minute window.
idempotency keys
Stripe's API accepts an Idempotency-Key header on POST requests: the first request's result is saved against the key and replayed for repeats, and keys can be removed from the system after they are at least 24 hours old.
a classic impossibility result
The Two Generals problem shows that two parties communicating over an unreliable channel cannot be certain they agree, which is why delivery over a network cannot be guaranteed exactly once, and why practical systems combine at-least-once delivery with idempotent processing.
Interview angle
Six questions, each worth 20 XP. 'What happens if the consumer crashes?' is a stock follow-up to any queue design.
Say at-least-once plus idempotent consumers, explain keys, and mention what exactly-once really covers.
What does a consumer that acknowledges before processing risk?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
The remedy is to put the key and the effect in one transaction in one database: insert the idempotency record with a unique constraint and perform the effect, committing together. If the request is repeated the unique constraint fails and the stored result is returned. When the effect is an external call, record the intent first, make the call with the provider's own idempotency key, and reconcile. Set the numbers.
With 5 million requests and 1 crash in 10,000 in the gap, key-first loses 500 payments, effect-first double-charges 500. A single transaction does neither.
Writing the key first means a crash before the effect leaves a key with no effect: the retry is told it is done. About 500 of 5,000,000 requests are silently lost.
Affected requests = requests × crash probability in the gap. Which failure you get depends on the order of the two writes.
Where to go next