Distributed Systems Core · lesson 11 of 11 · about 28 min
What to do when one business action spans several databases: why uncoordinated writes end half-done, how two-phase commit gets atomicity and what it costs in latency and availability, why a crashed coordinator blocks everyone, how sagas trade isolation for availability with compensating actions, and what happens when the undo itself fails. Tap any for a plain-language card.
Half-done work
The result is a half-done state: money taken but no order, stock reserved for nobody. The simulation below runs the steps in order with no coordination at all; each step fails independently with the chance you set. The rest of this lesson covers the two classic answers, and , and the cost of each.
Even at 1% failure per step, thousands per million actions end half-done. Add more steps and it grows.
3 services each failing 1% of the time: 1.99% of actions, 19,915 in every million, leave some steps applied and others not. Each one is a customer with a charge and no order, or stock held for nothing, and somebody has to clean it up.
200,000 seeded runs. A run is half-done when step k (not the first) fails and steps 1..k−1 stay applied.
Two-phase commit
That makes the whole action atomic, but it costs two round trips, each as slow as the slowest participant, and the action succeeds only if every participant votes yes. With 5 participants that each refuse 1% of the time, about 5% of transactions abort. Spanner and many distributed SQL databases use 2PC between the shards a transaction touches.
Go from 1 to 20 participants and watch the median and tail latency climb and the commit rate fall.
With 5 participants, a transaction takes 24 ms at the median (41 ms at the 99th percentile) against 17 ms for one participant, because both phases wait for the slowest. 95.2% commit; the rest abort because at least one of them voted no.
20,000 seeded transactions. Each participant response = round trip (log-normal) + 3 ms for a log flush; each phase waits for the slowest. Commit rate follows (1 − p)^n.
The blocking problem
The damage is not the in-doubt transaction itself but everything queued behind its locks: any other transaction that wants the same rows waits. A hot row (a popular product, an account) stalls the system. Three-phase commit and consensus-replicated coordinators reduce the problem (Spanner replicates each coordinator with Paxos), but they add complexity and cost. In PostgreSQL, prepared transactions that are never resolved are a well-known way to hold locks and block cleanup indefinitely.
At 500 transactions a second with 5% touching the locked rows, a one-minute coordinator outage queues thousands of transactions.
One incident of 1.0 min with 500 transactions a second, 5% of them touching the locked rows, queues about 1,500 transactions behind the in-doubt ones. 2 such incidents a month blocks about 3,000. Shorter recovery, or fewer hot rows, is the lever.
A simple model: blocked = transactions per second × share touching locked rows × outage seconds. It ignores retries and timeouts, which make real incidents worse.
Sagas
The price is that the intermediate states are visible: while the saga runs, another customer can see the seat as taken, and a refund is a new event, not a vanished one (the customer sees the charge and the refund). Sagas give up isolation to get availability. Choose which step fails and read the timeline.
Fail step 3 (“Issue ticket”): the card is charged, then refunded, and the seat is released. Fail step 5 and everything is undone in reverse.
Step 3 (Issue ticket) failed after 2 steps had committed. The saga undid refund card, then release seat and ended after 2000 ms. For that time, other users could see the intermediate states.
Step durations are illustrative (the card charge is the slow one). The order of undo is the reverse of the order of doing.
What it costs
That is the trade in a sentence: 2PC buys atomicity and isolation with availability and latency; sagas buy availability with weaker isolation and the burden of writing compensations. A third option is to avoid the problem: keep data that must change together in one database or one shard, and use a single local transaction.
Set 5 services at 99.9% and compare the two numbers, then add services.
5 services plus a coordinator, each 99.900% available, give a 2PC action that is up 99.401% of the time: 4.3 h of downtime a month. A saga accepts requests 99.900% of the time (43.2 min) and finishes the rest when the slower services return.
Series availability = a^(n+1). The saga figure is the first service's availability; completion is eventual and depends on retries and compensations.
Which approach?
The best distributed transaction is the one you avoid. When you cannot, choose between locking everyone (2PC) and doing then undoing (sagas). Each answer is worth 20 XP.
Ask: can the data live in one database? is isolation essential? are the participants separate companies or services?
Move money between two accounts that live in the same PostgreSQL database.
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Four published systems and papers on distributed transactions.
'Sagas', for long-lived transactions
The SIGMOD 1987 paper 'Sagas' proposed breaking a long-lived transaction into a sequence of sub-transactions, each with a compensating transaction that semantically undoes it. If the saga cannot finish, the compensations run in reverse order. It was aimed at long database transactions that would otherwise hold locks for hours.
two-phase commit across Paxos groups
Spanner's paper describes transactions that span multiple shards using two-phase commit, with each participant and the coordinator being a Paxos-replicated group. Replicating the coordinator removes the classic blocking problem of a single crashed coordinator, at the price of extra replication latency.
PREPARE TRANSACTION
PostgreSQL supports two-phase commit through PREPARE TRANSACTION and COMMIT PREPARED, and its documentation warns that it is meant for transaction managers rather than applications, and that prepared transactions left unresolved keep holding locks and block cleanup of old row versions.
orchestrating sagas with retries
Cadence (from Uber) and its successor Temporal persist the state of long-running workflows so that code written as a normal sequence of steps survives crashes, retries activities with configurable policies, and makes the saga pattern a documented use case with compensations run on failure.
Interview angle
Six questions, each worth 20 XP. Distributed transactions come up whenever a design touches money, inventory or orders across services.
Offer avoiding the transaction first (one database), then 2PC for atomicity, saga for availability, and idempotency everywhere.
What does 'in doubt' mean in two-phase commit?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
The two rules that make sagas work are the same ones as for any retried call: make every step and every compensation (the same request id applied twice has the effect of once), and retry compensations until they succeed rather than give up. Production workflow engines such as Temporal and Cadence exist largely to persist saga state and retry for you. Set the failure rates below.
Compensation failing 5% of the time with 2 retries and no idempotency double-applies thousands per million. Turn idempotency on and raise the retries.
Of every million 5-step sagas, 70 end stuck because a compensation never succeeded and 10,940 apply an undo twice (a double refund). Idempotent compensations remove the duplicates; more retries remove the stuck ones.
100,000 seeded sagas of 5 steps. Half of failed compensation attempts are modelled as lost replies (the undo had actually happened).
Where to go next