double-entry bookkeeping
The system of recording each transaction as equal debits and credits, described in print by Luca Pacioli in 1494, is still the foundation of ledgers; software ledgers simply enforce that the entries always sum to zero.
Design Interviews · lesson 13 of 15 · about 30 min
Design a payment system
A payment system end to end: what makes money different from other data, why the ledger writes both sides at once, how a retry is made harmless, how disagreements with the bank are found, how late events are handled, and how a crash between charging and recording is survived. Tap any for a plain-language card.
0 / 9 sections done
Money is not like other data
A payment system moves other people's money. Speed matters, but being exactly right matters far more.
A card payment is a small workflow with several parties: the merchant's checkout, the payment service, the processor or acquirer, the card network and the issuing bank. It has distinct steps: authorize (reserve the amount), capture (claim it), settle (money actually moves, a day or more later) and possibly refund. Design for the slowest, most failure-prone party being outside your control. Size the volume first.
100 million orders a day is 1,160 payments a second, 5,800 at a 5× peak, about $4 billion of volume a day and 2,300 ledger entries a second (two per payment).
Payments per second (average / peak)1.16K / 5.79K
Value moved per day$4.0 B
Ledger entries per second2.31K
Fees per day$116 M
100 million payments a day of $40 is 1.16K a second (5.79K at peak) and $4.0 billion of value, with 2.31K ledger entries a second. The volume is moderate; the difficulty is that each one must be exactly right and recoverable.
Arithmetic. Settlement happens in batches with the banks, usually one or more days after the authorization.
Every debit has a credit
A customer pays 40. Where did the 40 go? In a ledger, every movement of money is two entries that cancel out.
The invariant only holds if the two entries are written atomically, in one database transaction. Writing the debit, then the credit, as separate steps leaves a window where a crash makes money vanish. At a hundred thousand transfers and a small crash chance the books drift by thousands, and nobody notices until an audit. Try the two modes.
100,000 transfers with a 0.1% chance of a crash between the two writes: separate writes leave the books out by about 5,000 across 99 broken transfers. In one transaction the drift is exactly 0.
Transfers completed99,901
Broken transfers (half-written)99
Money that vanished (the books should sum to 0)-5,035
99 of 100,000 transfers crashed between the debit and the credit, so the ledger no longer balances: 5,035 of money has disappeared.
Seeded simulation: transfers of 1 to 100; a crash after the debit leaves the credit unwritten unless both writes share a transaction.
The spinner spun, so the customer pressed Pay again
The charge succeeded, but the response never reached the browser. The customer sees an error and tries again.
Details matter. The key check and the charge must be one atomic step (otherwise two simultaneous retries both pass the check); a key reused with different parameters is an error, not a replay; keys expire after a long window (a day or more); and the server must still make the call to the processor idempotent in turn, passing along its own key. Without all of this, every flaky mobile connection is a refund request. Compare.
100,000 charges with 1% of responses lost: without an idempotency key about 1,045 customers are charged twice (1.0%), roughly $41,800 at $40 each. With the key, none.
Customers charged twice1,045
Share of all charges1.04%
Money wrongly taken$41,800
With 1.0% of responses lost and a retry that charges again, 1,045 of 100,000 customers (1.04%) were charged twice, $41,800 wrongly taken, each one a support ticket and a refund.
Seeded simulation of n charges; a lost response triggers one retry, which charges again unless a stored result for the key exists.
Trust, but check with the bank
Your ledger says one million payments succeeded. The bank's settlement file says 999,000. Who is right?
Each exception is then resolved: query the processor for the true status, fix your ledger with a correcting entry, refund or re-run as needed. Without reconciliation, small mismatch rates are invisible and compound: a 0.15% disagreement on a million payments a day is 1,500 unexplained payments every day. It is also your main defence against bugs you do not know you have. Compare a month with and without it.
1 million payments a day with 0.1% lost webhooks and 0.05% amount differences produce 1,500 mismatches a day. Without a reconciliation job, 45,000 pile up in a month, about $1.8 million of unexplained money at $40 each; with a daily job, at most one day's 1,500 are ever open.
New mismatches per day1,500
Mismatches open and unexplained45,000
Money in unexplained limbo$1.8M
With no reconciliation, 1,500 mismatches a day pile up to 45,000 in 30 days: about $1.8M that nobody can explain.
Arithmetic: mismatches per day = payments × (lost-webhook rate + amount-difference rate); without a job they accumulate for the chosen days.
“Captured” arrives before “authorised”
The processor tells you about each step of a payment by webhook. They are sent in order. They do not always arrive in order.
The fix is to make state transitions monotonic and checked: a payment only moves forward along allowed edges, an event that would move it backwards (or repeat the current state) is ignored, and an event that skips a step is accepted when the processor is the source of truth, or held until the missing one arrives. Store event ids to drop duplicates, and when in doubt, fetch the current status from the processor rather than trusting the message. Try the reordering rate.
If each neighbouring pair of events is swapped 5% of the time, a naive last-event-wins consumer leaves about 5% of payments in the wrong final state; at 20% it is 20%. A forward-only state machine gets all of them right.
Payments in the wrong final state (naive)5.1%
Per million payments50,500
With 5% of neighbouring events swapped, a consumer that trusts the last event leaves 5.1% of payments in the wrong state: 50,500 per million. A forward-only state machine ignores late events and gets every one right.
Seeded simulation of four events (created, authorised, captured, settled) where each adjacent pair is swapped with the chosen probability.
Pick the safeguard
Six problems, four safeguards.
Each problem has one safeguard that fits best. Each answer is worth 20 XP.
Match the symptom: double charges, unbalanced books, unexplained differences, or charged-but-unrecorded?
Question 1 / 6
A customer's request timed out, they retried and were charged twice.
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Money in software.
Public facts behind payment engineering.
payments data store
Uber has described building LedgerStore, an immutable, append-only datastore for payments and financial data, because correctness, auditability and the ability to replay history mattered more than flexibility.
authorization and settlement
A card payment is first authorised, which reserves the amount, and later captured and settled in batches through the card network and banks, often a day or more afterwards, which is why reconciliation against settlement files is necessary.
reducing scope
The PCI DSS standard governs systems that handle card numbers; most merchants avoid that burden by sending card details straight from the browser to a payment provider and handling only a token.
Interview angle
What interviewers actually ask.
Six questions, each worth 20 XP. Strong answers insist on exactly-once effects, auditability and recovery.
Say how a retry is made safe, what the ledger guarantees and how disagreements are found.
Question 1 / 6
How is a retried payment request made harmless?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
The code calls the card processor, the charge succeeds, and then the server is killed before it writes “paid” to the database.
The remedy is to record intent first. Write a pending payment row (with its idempotency key) in your database before calling the processor, call the processor with that key, then update the row to paid or failed. If a crash leaves a row stuck in pending, a asks the processor for the status of that key and finishes the job, either confirming the order or releasing it. No payment can be silently lost; the worst case is a delay of one reconciler cycle. Compare.
1 million payments with a 0.02% chance of a crash between charge and record: the naive order leaves 200 customers charged with nothing on record. Writing pending first and reconciling every 10 minutes resolves all of them within 5 minutes on average.
Payments interrupted mid-way200
Customers charged with no trace in your system200
Average time to resolvenever (someone must notice)
200 of 1,000,000 payments crashed between the charge and the record: 200 customers were charged and your database has no idea. Only a complaint will reveal it.
Expected-value arithmetic: interrupted = payments × crash chance; a reconciler cycle resolves a stuck pending row after half its period on average.