Design a Payment System
Money must move exactly once. Idempotency keys, a double-entry ledger, provider webhooks and reconciliation keep every cent accounted for.
- ~1,160/speak payments, 10x the 116/s average
- $500M/daymoney moved; 0.01% errors is 1,000 payments
- 24 hoursminimum time idempotency keys are kept
Show what to say
“This is correctness under failure, not scale. Every checkout carries an idempotency key claimed under a unique constraint, and a PSP timeout means unknown, never failed; a deduplicated webhook or a status query settles it. Money moves as append-only double-entry ledger transactions written with an outbox event, and daily reconciliation catches whatever slips through.”
Trade-offs
- Ledger vs a balance column: full history, audits and rebuilds, but more rows and balances need aggregating or caching.
- Idempotency in the DB vs Redis: atomic with the payment, but more load on the primary.
- Sync charge vs async: instant answer and simple, but latency and uptime are tied to the PSP.
Bring it up before they ask
Money moving twice: a lost reply, a double tap or a duplicate webhook must each produce exactly one charge and one ledger posting, which unique constraints guarantee.
New to the words? Read the one-minute primer
Label each checkout with an , so a retry never charges twice. A timeout means unknown, not failed. Record every cent once in a ledger, and check it daily.
Functional
- Charge the card for an order through a PSP
- Payment status for the client (poll or push)
- Full and partial refunds
- An auditable ledger of every money movement
- Payouts to merchants (a follow-up)
Non-functional
- Never double-charge, never lose a payment
- Exactly-once effect on at-least-once delivery
- Strong consistency, in integer minor units
- Degrade gracefully when the PSP is slow
- Minimal PCI scope: cards go straight to the PSP
Out of scope
- Being a card network or acquiring bank
- Fraud models (a follow-up)
- Tax, invoicing and currency hedging
This question is about things going wrong, not scale. Replies get lost, arrive twice, customers double-tap. The rule: every cent moves once and is recorded once. The PSP is the payment provider, like Stripe.
Scope and scale
Ask:
Read more · 190 words
- Who pays whom? A marketplace: customers pay, the platform keeps a cut, merchants get paid later.
- Which methods? Cards answer in a second. Bank debits confirm hours later, by .
- Charge now, or hold first and capture on shipment?
- Refunds, currencies, peak load?
| Quantity | Calculation | Result |
|---|---|---|
| Payments | 10M/day ÷ 86,400 s | ≈ 116/s average |
| Peak | ~10× on Black Friday | ≈ 1,160/s |
| PSP calls in flight | 1,160/s × ~1 s latency | ≈ 1,160 concurrent |
| Ledger entries | 4 per payment × 10M | 40M rows/day |
| Storage | payments + ledger | ≈ 11 GB/day, 4 TB/year |
| Money moved | 10M × $50 average | $500M/day |
One Postgres copes. Two numbers matter:
- 1,160 calls are always waiting on the PSP, so never hold a lock across one.
- At $500M a day, a 0.01% error rate is 1,000 wrong payments daily. Constraints and reconciliation beat luck.
API and data model
Read more · 170 words
| Method | Path | Body | Result |
|---|---|---|---|
| POST | /payments | Idempotency-Key header; orderId, amount, currency, paymentMethod (PSP token) | 201 done, 202 still unknown, 402 declined |
| GET | /payments/:id | id, status, amount, refunded | |
| POST | /refunds | Idempotency-Key; paymentId, amount | 201, or 409 above the refundable amount |
| POST | /webhooks/psp | The PSP's signed event | 200 once stored, duplicates included |
The client makes the key once per checkout and reuses it on retries. The server keeps keys 24 hours. Same key and body: replay the stored response. Still running: 409. Different body: 422.
The data model:
The payment is a state machine:
created → processing, committed before the PSP call.processing → succeeded | failed | requires_action, orunknownon timeout.unknown → succeeded | failed, settled by a or status query, never a guess.succeeded → partially_refunded → refunded.
Money lives in the ledger, not a balance column. A balance is the sum of entries.
-- Money is bigint minor units plus an ISO 4217 currency: 5000 USD = $50.00. Never float, never numeric-as-text.
CREATE TABLE payments (
id uuid PRIMARY KEY,
order_id text NOT NULL,
customer_id text NOT NULL,
merchant_id text NOT NULL,
amount bigint NOT NULL CHECK (amount > 0),
currency char(3) NOT NULL,
status text NOT NULL CHECK (status IN ('created', 'processing', 'requires_action', 'unknown',
'succeeded', 'failed', 'canceled', 'partially_refunded', 'refunded')),
refunded bigint NOT NULL DEFAULT 0 CHECK (refunded >= 0 AND refunded <= amount),
psp_payment_id text UNIQUE,
version int NOT NULL DEFAULT 0,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now()
);
-- At most one live payment per order, even if a client forgets its key. A decline frees the order for a new attempt.
CREATE UNIQUE INDEX payments_one_live_per_order ON payments (order_id) WHERE status NOT IN ('failed', 'canceled');
CREATE INDEX payments_stuck ON payments (updated_at) WHERE status IN ('processing', 'unknown'); -- the recovery job's queue
CREATE TABLE idempotency_keys (
customer_id text NOT NULL,
key text NOT NULL, -- client-generated, one per checkout attempt
request_hash text NOT NULL, -- same key + different body → 422
status text NOT NULL DEFAULT 'started' CHECK (status IN ('started', 'done')),
payment_id uuid REFERENCES payments (id),
response_code int,
response_body jsonb, -- replayed byte for byte on a retry
created_at timestamptz NOT NULL DEFAULT now(), -- purge after 24 hours or more
PRIMARY KEY (customer_id, key)
);
CREATE TABLE refunds (
id uuid PRIMARY KEY, -- also the PSP idempotency key for the refund
payment_id uuid NOT NULL REFERENCES payments (id),
amount bigint NOT NULL CHECK (amount > 0),
status text NOT NULL,
psp_refund_id text UNIQUE,
created_at timestamptz NOT NULL DEFAULT now()
);
-- Double-entry ledger: append-only. Mistakes are fixed with new, reversing transactions.
CREATE TABLE accounts (
id text PRIMARY KEY, -- 'cash_at_psp', 'merchant_payable:m_42'
type text NOT NULL CHECK (type IN ('asset', 'liability', 'revenue', 'expense')),
currency char(3) NOT NULL
);
CREATE TABLE ledger_transactions (
id bigserial PRIMARY KEY,
source text NOT NULL UNIQUE, -- 'pay_…:charge', 'rf_…:refund': each business event posts once
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE ledger_entries (
id bigserial PRIMARY KEY,
transaction_id bigint NOT NULL REFERENCES ledger_transactions (id),
account_id text NOT NULL REFERENCES accounts (id),
amount bigint NOT NULL CHECK (amount <> 0), -- + debit, − credit; each transaction sums to 0
currency char(3) NOT NULL,
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX ledger_entries_by_account ON ledger_entries (account_id, id);
-- The posting code rejects Σ ≠ 0; a deferred constraint trigger can enforce it in the database as well.
REVOKE UPDATE, DELETE ON ledger_transactions, ledger_entries FROM payments_app;
CREATE TABLE webhook_events (
event_id text PRIMARY KEY, -- the PSP's id: a redelivery inserts nothing
type text NOT NULL,
payload jsonb NOT NULL,
received_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE outbox (
id bigserial PRIMARY KEY,
topic text NOT NULL, -- 'payment.succeeded'
key text NOT NULL, -- partition key: the payment id
payload jsonb NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
published_at timestamptz
);
CREATE INDEX outbox_unpublished ON outbox (id) WHERE published_at IS NULL;Architecture
A payment is always in one state, and only some moves are legal.
Read more · 164 words
Say the PSP call times out after 10 seconds. The card may or may not be charged. Mark it failed and a retry can charge twice. Mark it unknown, retry with the same key, and let the decide. Only requests that move money need a key; a read can just repeat. Play each scenario below.
Press Pay $50, then Play the bad day. Then switch or Webhook off and replay.
- Keys off: two charges. The books balance, but the money really moved twice.
- Dedupe off: one charge, posted twice. Only reconciliation notices.
Tap any box to see what it is and what it talks to.
- Client
- Service
- External API
- Worker
- Database
- Queue
Text description
- Alice taps Pay. The app sends POST /payments with Idempotency-Key k1 and a card token from the PSP's SDK: no card number touches our servers.
- One transaction claims k1 (INSERT … ON CONFLICT DO NOTHING) and inserts pay_1 as processing. It commits before anything leaves the building.
- Charge $50.00, passing our own idempotency key to the PSP. No database transaction stays open during the call.
- The PSP answers: succeeded.
- A second transaction moves pay_1 to succeeded, posts four balanced ledger entries, writes an outbox row, and stores the 201 response under k1.
- Alice sees the order confirmed.
- Seconds later the PSP's signed webhook evt_9 arrives: payment succeeded.
- The handler records evt_9 in webhook_events (new), finds pay_1 already succeeded, and does nothing else. It answers 200.
- The outbox relay picks up the payment.succeeded row…
- …and publishes it. If the relay crashes after publishing, it publishes again: consumers dedupe by event id.
- The order service marks the order paid and starts fulfilment.
Step-by-step answer
Keep card data out
The PSP's SDK swaps the card for a token in the browser.
Claim the idempotency key in the same transaction
One claims the key and creates the payment. The unique key lets only one request win:
BEGIN; INSERT INTO idempotency_keys (customer_id, key, request_hash) VALUES ($1, $2, $3) ON CONFLICT (customer_id, key) DO NOTHING; -- 0 rows: someone owns the key -- if 0 rows: SELECT request_hash, status, response_code, response_body ... -- hash differs → ROLLBACK, 422 -- status 'done' → ROLLBACK, replay the stored response -- status 'started' → ROLLBACK, 409 INSERT INTO payments (id, ..., status) VALUES ($4, ..., 'processing'); COMMIT;Never hold a transaction across the PSP call
Commit the claim, call the PSP, then record the outcome in a second transaction. After a crash, the recovery job asks the PSP.
A timeout means unknown
Mark it
unknownand return202. Retry with your own key, never a new one. The or a status query decides. Why not retry with a fresh key? If the first call did charge, the customer pays twice. Why not mark it failed? The customer taps Pay again, with the same result.Webhooks: verify, dedupe, move forward
Check the signature.
ON CONFLICT DO NOTHINGon the event id drops repeats. Ignore late events that go backward. Answer200fast.One writer for each move
The reply, the webhook and the recovery job share one function: lock the row, check the move is legal, then update everything in one transaction. The first signal wins.
Double-entry ledger
Each transaction's entries sum to zero, in integer cents. $50.00 with a PSP fee and a 10% commission:
Account Type Debit Credit Cash at PSP asset $48.25 PSP fees expense $1.75 Merchant payable (m_42) liability $45.00 Platform revenue revenue $5.00 Total $50.00 $50.00 Append-only: fixes are new entries. One leg is the remainder, so rounding loses no cent.
Outbox for events
Write
payment.succeededto anoutboxtable in the same transaction. A relay publishes it, maybe twice, so . See the dual-write problem.Reconcile every day
Match the PSP's settlement report to the ledger, then payouts to the bank statement. Fix mismatches with new entries, never by editing history.
Refunds
One conditional UPDATE reserves a refund, so concurrent refunds can't exceed the charge:
UPDATE payments SET refunded = refunded + $2, status = CASE WHEN refunded + $2 = amount THEN 'refunded' ELSE 'partially_refunded' END WHERE id = $1 AND status IN ('succeeded', 'partially_refunded') AND refunded + $2 <= amount;Zero rows means
409. Multi-step flows (authorize, reserve stock, capture) run as a : every step has an undo.
Implementation
Read more · 37 words
chargeEntriesmakes the merchant shareamount - commission, so every amount from 1¢ to $100 balances.Ledger.postis safe to repeat. Reusing an id for different entries throws.applyEventreturnsapplied: falsefor duplicates and late events.
// A tiny double-entry ledger. Amounts are integers in minor units (cents for USD):
// positive is a debit, negative is a credit, so a balanced transaction sums to exactly 0.
// Floats never touch money: 0.1 + 0.2 !== 0.3, and a ledger can't be off by a fraction of a cent.
/** Throws unless the entries form one valid transaction. */
export function validate(entries) {
if (!Array.isArray(entries) || entries.length < 2) throw new Error('A transaction needs at least two entries');
const currency = entries[0].currency;
let sum = 0;
for (const e of entries) {
if (!e.account) throw new Error('Every entry needs an account');
if (!Number.isSafeInteger(e.amount) || e.amount === 0) throw new Error(`Amount must be a non-zero integer in minor units, got ${e.amount}`);
// One currency per transaction: an FX conversion is two transactions through an FX account.
if (e.currency !== currency) throw new Error(`Mixed currencies in one transaction: ${currency} and ${e.currency}`);
sum += e.amount;
}
if (sum !== 0) throw new Error(`Unbalanced transaction: entries sum to ${sum}`);
}
export class Ledger {
#txs = new Map(); // id → transaction, e.g. 'pay_1:charge'
#entries = []; // append-only: nothing is ever updated or deleted
/** Post a transaction once. Re-posting the same id is a no-op; reusing an id for other entries is an error. */
post({ id, entries, memo = '' }) {
validate(entries);
const seen = this.#txs.get(id);
if (seen) {
if (JSON.stringify(seen.entries) !== JSON.stringify(entries)) throw new Error(`Transaction ${id} was already posted with different entries`);
return { ...seen, duplicate: true };
}
const tx = Object.freeze({ id, memo, entries: Object.freeze(entries.map((e) => Object.freeze({ ...e }))) });
this.#txs.set(id, tx);
for (const e of tx.entries) this.#entries.push({ txId: id, ...e });
return { ...tx, duplicate: false };
}
/** Balances are derived from the entries, never stored as the source of truth. */
balance(account, currency = 'USD') {
let sum = 0;
for (const e of this.#entries) if (e.account === account && e.currency === currency) sum += e.amount;
return sum;
}
/** Every account's balance per currency: the trial balance, which must sum to 0 in each currency. */
trialBalance() {
const out = {};
for (const e of this.#entries) {
out[e.currency] ??= {};
out[e.currency][e.account] = (out[e.currency][e.account] ?? 0) + e.amount;
}
return out;
}
get entries() {
return this.#entries.map((e) => ({ ...e }));
}
}
/**
* The posting for one successful card payment on a marketplace. The PSP keeps its fee
* (2.9% + 30¢ here) and the platform keeps a commission; the rest is owed to the merchant.
* The merchant's share is the remainder, so rounding can never create or lose a cent.
*/
export function chargeEntries({ amount, currency = 'USD', merchantId, feeBps = 290, feeFixed = 30, commissionBps = 1000 }) {
const fee = Math.round((amount * feeBps) / 10_000) + feeFixed;
const commission = Math.round((amount * commissionBps) / 10_000);
return [
{ account: 'cash_at_psp', amount: amount - fee, currency }, // asset: the PSP holds it until settlement
{ account: 'psp_fees', amount: fee, currency }, // expense
{ account: `merchant_payable:${merchantId}`, amount: -(amount - commission), currency }, // liability: we owe the merchant
{ account: 'platform_revenue', amount: -commission, currency }, // revenue
];
}
/** A refund the merchant pays for: the money goes back out of what we hold at the PSP. */
export function refundEntries({ amount, currency = 'USD', merchantId }) {
return [
{ account: `merchant_payable:${merchantId}`, amount, currency },
{ account: 'cash_at_psp', amount: -amount, currency },
];
}Failure modes and trade-offs
Read more · 198 words
| What breaks | Effect | Mitigation |
|---|---|---|
| PSP times out | Charged or not: unknown | unknown; same-key retry; webhook, then status query |
| DB down after the PSP charged | Money moved, no record | Intent committed first; webhook retries; reconciliation |
| Webhook storm | Slow handler, more retries | Store, answer 200, process from a queue |
| Double submit | Two charges | Unique key; 409 while running |
| Two refunds at once | Too much refunded | refunded + $2 <= amount in the UPDATE |
| Currency rounding | Cents vanish | Integer minor units; remainder leg |
| Outbox relay down | Orders not marked paid | Rows wait; alert on the oldest |
Choices to defend:
| Choice | Gain | Cost |
|---|---|---|
| Ledger, not a balance column | Full history, auditable | More rows to sum |
| Key in the DB, not Redis | Atomic with the payment | More load on the primary |
| Charge now, not async | Instant, simple | Tied to the PSP's speed |
| Hold then capture | Free voids | Holds expire in about 7 days |
Follow-up questions
How do you handle multiple currencies?
Tag every entry with its currency. Never mix two in one . Convert through an FX account and store the rate.
How do payouts to merchants work?
A daily job pays each merchant's balance minus a refund reserve. Failed payouts reverse with new entries.
Where do fraud checks go?
Before the PSP call. Low risk passes, medium asks for extra checks, high is blocked. If the risk service is down, allow small amounts only.
Can you fail over to a second PSP?
Only if the first certainly never charged. After a timeout, a second charge can mean two. Settle the first through its .
Reconciliation found a mismatch. Now what?
Fix forward, never by editing entries. Replay a missed webhook or reverse a bad posting. Send the unexplained to a human.
Check yourself
Question 1 of 3Score 2 out of 3 or better to mark this page complete.