BackendHard45 min

Design a Payment System

Money must move exactly once. Idempotency keys, a double-entry ledger, provider webhooks and reconciliation keep every cent accounted for.

Say it in 30 seconds
  • ~1,160/speak payments, 10x the 116/s average
  • $500M/daymoney moved; 0.01% errors is 1,000 payments
  • 24 hoursminimum time idempotency keys are kept
Show what to say

“This is correctness under failure, not scale. Every checkout carries an idempotency key claimed under a unique constraint, and a PSP timeout means unknown, never failed; a deduplicated webhook or a status query settles it. Money moves as append-only double-entry ledger transactions written with an outbox event, and daily reconciliation catches whatever slips through.”

Trade-offs

  • Ledger vs a balance column: full history, audits and rebuilds, but more rows and balances need aggregating or caching.
  • Idempotency in the DB vs Redis: atomic with the payment, but more load on the primary.
  • Sync charge vs async: instant answer and simple, but latency and uptime are tied to the PSP.

Bring it up before they ask

Money moving twice: a lost reply, a double tap or a duplicate webhook must each produce exactly one charge and one ledger posting, which unique constraints guarantee.

New to the words? Read the one-minute primer

0 of 0 sections ticked
30-second answer

Label each checkout with an , so a retry never charges twice. A timeout means unknown, not failed. Record every cent once in a ledger, and check it daily.

Functional

  • Charge the card for an order through a PSP
  • Payment status for the client (poll or push)
  • Full and partial refunds
  • An auditable ledger of every money movement
  • Payouts to merchants (a follow-up)

Non-functional

  • Never double-charge, never lose a payment
  • Exactly-once effect on at-least-once delivery
  • Strong consistency, in integer minor units
  • Degrade gracefully when the PSP is slow
  • Minimal PCI scope: cards go straight to the PSP

Out of scope

  • Being a card network or acquiring bank
  • Fraud models (a follow-up)
  • Tax, invoicing and currency hedging

This question is about things going wrong, not scale. Replies get lost, arrive twice, customers double-tap. The rule: every cent moves once and is recorded once. The PSP is the payment provider, like Stripe.

Scope and scale

Ask:

Read more · 190 words
  • Who pays whom? A marketplace: customers pay, the platform keeps a cut, merchants get paid later.
  • Which methods? Cards answer in a second. Bank debits confirm hours later, by .
  • Charge now, or hold first and capture on shipment?
  • Refunds, currencies, peak load?
QuantityCalculationResult
Payments10M/day ÷ 86,400 s≈ 116/s average
Peak~10× on Black Friday≈ 1,160/s
PSP calls in flight1,160/s × ~1 s latency≈ 1,160 concurrent
Ledger entries4 per payment × 10M40M rows/day
Storagepayments + ledger≈ 11 GB/day, 4 TB/year
Money moved10M × $50 average$500M/day

One Postgres copes. Two numbers matter:

  • 1,160 calls are always waiting on the PSP, so never hold a lock across one.
  • At $500M a day, a 0.01% error rate is 1,000 wrong payments daily. Constraints and reconciliation beat luck.

API and data model

Read more · 170 words
MethodPathBodyResult
POST/paymentsIdempotency-Key header; orderId, amount, currency, paymentMethod (PSP token)201 done, 202 still unknown, 402 declined
GET/payments/:idid, status, amount, refunded
POST/refundsIdempotency-Key; paymentId, amount201, or 409 above the refundable amount
POST/webhooks/pspThe PSP's signed event200 once stored, duplicates included

The client makes the key once per checkout and reuses it on retries. The server keeps keys 24 hours. Same key and body: replay the stored response. Still running: 409. Different body: 422.

The data model:

The payment is a state machine:

  • created → processing, committed before the PSP call.
  • processing → succeeded | failed | requires_action, or unknown on timeout.
  • unknown → succeeded | failed, settled by a or status query, never a guess.
  • succeeded → partially_refunded → refunded.

Money lives in the ledger, not a balance column. A balance is the sum of entries.

schema.sql
-- Money is bigint minor units plus an ISO 4217 currency: 5000 USD = $50.00. Never float, never numeric-as-text.
CREATE TABLE payments (
  id             uuid        PRIMARY KEY,
  order_id       text        NOT NULL,
  customer_id    text        NOT NULL,
  merchant_id    text        NOT NULL,
  amount         bigint      NOT NULL CHECK (amount > 0),
  currency       char(3)     NOT NULL,
  status         text        NOT NULL CHECK (status IN ('created', 'processing', 'requires_action', 'unknown',
                               'succeeded', 'failed', 'canceled', 'partially_refunded', 'refunded')),
  refunded       bigint      NOT NULL DEFAULT 0 CHECK (refunded >= 0 AND refunded <= amount),
  psp_payment_id text        UNIQUE,
  version        int         NOT NULL DEFAULT 0,
  created_at     timestamptz NOT NULL DEFAULT now(),
  updated_at     timestamptz NOT NULL DEFAULT now()
);
-- At most one live payment per order, even if a client forgets its key. A decline frees the order for a new attempt.
CREATE UNIQUE INDEX payments_one_live_per_order ON payments (order_id) WHERE status NOT IN ('failed', 'canceled');
CREATE INDEX payments_stuck ON payments (updated_at) WHERE status IN ('processing', 'unknown');   -- the recovery job's queue

CREATE TABLE idempotency_keys (
  customer_id   text        NOT NULL,
  key           text        NOT NULL,                  -- client-generated, one per checkout attempt
  request_hash  text        NOT NULL,                  -- same key + different body → 422
  status        text        NOT NULL DEFAULT 'started' CHECK (status IN ('started', 'done')),
  payment_id    uuid        REFERENCES payments (id),
  response_code int,
  response_body jsonb,                                 -- replayed byte for byte on a retry
  created_at    timestamptz NOT NULL DEFAULT now(),    -- purge after 24 hours or more
  PRIMARY KEY (customer_id, key)
);

CREATE TABLE refunds (
  id            uuid        PRIMARY KEY,               -- also the PSP idempotency key for the refund
  payment_id    uuid        NOT NULL REFERENCES payments (id),
  amount        bigint      NOT NULL CHECK (amount > 0),
  status        text        NOT NULL,
  psp_refund_id text        UNIQUE,
  created_at    timestamptz NOT NULL DEFAULT now()
);

-- Double-entry ledger: append-only. Mistakes are fixed with new, reversing transactions.
CREATE TABLE accounts (
  id       text    PRIMARY KEY,                        -- 'cash_at_psp', 'merchant_payable:m_42'
  type     text    NOT NULL CHECK (type IN ('asset', 'liability', 'revenue', 'expense')),
  currency char(3) NOT NULL
);
CREATE TABLE ledger_transactions (
  id         bigserial   PRIMARY KEY,
  source     text        NOT NULL UNIQUE,              -- 'pay_…:charge', 'rf_…:refund': each business event posts once
  created_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE ledger_entries (
  id             bigserial   PRIMARY KEY,
  transaction_id bigint      NOT NULL REFERENCES ledger_transactions (id),
  account_id     text        NOT NULL REFERENCES accounts (id),
  amount         bigint      NOT NULL CHECK (amount <> 0),   -- + debit, − credit; each transaction sums to 0
  currency       char(3)     NOT NULL,
  created_at     timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX ledger_entries_by_account ON ledger_entries (account_id, id);
-- The posting code rejects Σ ≠ 0; a deferred constraint trigger can enforce it in the database as well.
REVOKE UPDATE, DELETE ON ledger_transactions, ledger_entries FROM payments_app;

CREATE TABLE webhook_events (
  event_id    text        PRIMARY KEY,                 -- the PSP's id: a redelivery inserts nothing
  type        text        NOT NULL,
  payload     jsonb       NOT NULL,
  received_at timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE outbox (
  id           bigserial   PRIMARY KEY,
  topic        text        NOT NULL,                   -- 'payment.succeeded'
  key          text        NOT NULL,                   -- partition key: the payment id
  payload      jsonb       NOT NULL,
  created_at   timestamptz NOT NULL DEFAULT now(),
  published_at timestamptz
);
CREATE INDEX outbox_unpublished ON outbox (id) WHERE published_at IS NULL;

Architecture

A payment is always in one state, and only some moves are legal.

Read more · 164 words

Say the PSP call times out after 10 seconds. The card may or may not be charged. Mark it failed and a retry can charge twice. Mark it unknown, retry with the same key, and let the decide. Only requests that move money need a key; a read can just repeat. Play each scenario below.

Press Pay $50, then Play the bad day. Then switch or Webhook off and replay.

  • Keys off: two charges. The books balance, but the money really moved twice.
  • Dedupe off: one charge, posted twice. Only reconciliation notices.
A state machine is a vending machine: it is always in one state, and each event moves it to the next.

Press Play to watch it run · 11 steps

The Payments API claims an idempotency key and writes the payment in Postgres before calling the PSP. Signed webhooks settle unknown outcomes, an outbox relay announces payments, and a daily job reconciles the ledger with the PSP's settlement report.

Payment system architectureThe Payments API claims an idempotency key and writes the payment in Postgres before calling the PSP. Signed webhooks settle unknown outcomes, an outbox relay announces payments, and a daily job reconciles the ledger with the PSP's settlement report.payclaim keychargewebhookspost onceoutboxpublishorder paidreportcompareThe PSP's SDK turns the card into a token in the browser, so card numbers never reach our servers.Checkout appPayments APIstateless ×NTakes our idempotency key, charges the card, and sends signed webhooks for every state change.PSPStripe / AdyenReconciliationdailyIdempotency keys, payments, the ledger, webhook events and the outbox, so one transaction covers them all.Payments DBPostgres primaryWebhook handlerverify, dedupeOutbox relayEvent buspayment.succeededOrder service

Tap any box to see what it is and what it talks to.

  • Client
  • Service
  • External API
  • Worker
  • Database
  • Queue
Open in playground
Text description
  1. Alice taps Pay. The app sends POST /payments with Idempotency-Key k1 and a card token from the PSP's SDK: no card number touches our servers.
  2. One transaction claims k1 (INSERT … ON CONFLICT DO NOTHING) and inserts pay_1 as processing. It commits before anything leaves the building.
  3. Charge $50.00, passing our own idempotency key to the PSP. No database transaction stays open during the call.
  4. The PSP answers: succeeded.
  5. A second transaction moves pay_1 to succeeded, posts four balanced ledger entries, writes an outbox row, and stores the 201 response under k1.
  6. Alice sees the order confirmed.
  7. Seconds later the PSP's signed webhook evt_9 arrives: payment succeeded.
  8. The handler records evt_9 in webhook_events (new), finds pay_1 already succeeded, and does nothing else. It answers 200.
  9. The outbox relay picks up the payment.succeeded row…
  10. …and publishes it. If the relay crashes after publishing, it publishes again: consumers dedupe by event id.
  11. The order service marks the order paid and starts fulfilment.
Interactive

Payment flow: exactly-once on a bad day

One $50 checkout: a timeout, a second tap, a duplicate webhook.

webhooksposts$Customer$200.00Order #1$50.00Pay $50Merchant+$0.00APIPOST /paymentskeysemptyidempotencyPayment serviceidleledger0 postingsPSP0 calls, 0 charges
Charged 0× $0.00 Ledger: 0 postings, ✓ balanced

Press Pay $50 for the happy path, or Play the bad day: a PSP timeout, an impatient second tap and a duplicate webhook.

Inject one failure at a time
Ledger and history
Ledger (append-only)Σ every transaction = $0.00 ✓ balanced

No entries yet. A capture is posted when a payment succeeds.

    Merchant payable: $0.00 · PSP calls: 0Reconciliation runs once the traffic settles.
    What happened, newest first
    1. Nothing yet.

    Step-by-step answer

    1. Keep card data out

      The PSP's SDK swaps the card for a token in the browser.

    2. Claim the idempotency key in the same transaction

      One claims the key and creates the payment. The unique key lets only one request win:

      BEGIN;
      INSERT INTO idempotency_keys (customer_id, key, request_hash)
      VALUES ($1, $2, $3)
      ON CONFLICT (customer_id, key) DO NOTHING;      -- 0 rows: someone owns the key
      -- if 0 rows: SELECT request_hash, status, response_code, response_body ...
      --   hash differs     → ROLLBACK, 422
      --   status 'done'    → ROLLBACK, replay the stored response
      --   status 'started' → ROLLBACK, 409
      INSERT INTO payments (id, ..., status) VALUES ($4, ..., 'processing');
      COMMIT;
    3. Never hold a transaction across the PSP call

      Commit the claim, call the PSP, then record the outcome in a second transaction. After a crash, the recovery job asks the PSP.

    4. A timeout means unknown

      Mark it unknown and return 202. Retry with your own key, never a new one. The or a status query decides. Why not retry with a fresh key? If the first call did charge, the customer pays twice. Why not mark it failed? The customer taps Pay again, with the same result.

    5. Webhooks: verify, dedupe, move forward

      Check the signature. ON CONFLICT DO NOTHING on the event id drops repeats. Ignore late events that go backward. Answer 200 fast.

    6. One writer for each move

      The reply, the webhook and the recovery job share one function: lock the row, check the move is legal, then update everything in one transaction. The first signal wins.

    7. Double-entry ledger

      Each transaction's entries sum to zero, in integer cents. $50.00 with a PSP fee and a 10% commission:

      AccountTypeDebitCredit
      Cash at PSPasset$48.25
      PSP feesexpense$1.75
      Merchant payable (m_42)liability$45.00
      Platform revenuerevenue$5.00
      Total$50.00$50.00

      Append-only: fixes are new entries. One leg is the remainder, so rounding loses no cent.

    8. Outbox for events

      Write payment.succeeded to an outbox table in the same transaction. A relay publishes it, maybe twice, so . See the dual-write problem.

    9. Reconcile every day

      Match the PSP's settlement report to the ledger, then payouts to the bank statement. Fix mismatches with new entries, never by editing history.

    10. Refunds

      One conditional UPDATE reserves a refund, so concurrent refunds can't exceed the charge:

      UPDATE payments
      SET refunded = refunded + $2,
          status = CASE WHEN refunded + $2 = amount THEN 'refunded' ELSE 'partially_refunded' END
      WHERE id = $1 AND status IN ('succeeded', 'partially_refunded') AND refunded + $2 <= amount;

      Zero rows means 409. Multi-step flows (authorize, reserve stock, capture) run as a : every step has an undo.

    Implementation

    Read more · 37 words
    • chargeEntries makes the merchant share amount - commission, so every amount from 1¢ to $100 balances.
    • Ledger.post is safe to repeat. Reusing an id for different entries throws.
    • applyEvent returns applied: false for duplicates and late events.
    // A tiny double-entry ledger. Amounts are integers in minor units (cents for USD):
    // positive is a debit, negative is a credit, so a balanced transaction sums to exactly 0.
    // Floats never touch money: 0.1 + 0.2 !== 0.3, and a ledger can't be off by a fraction of a cent.
    
    /** Throws unless the entries form one valid transaction. */
    export function validate(entries) {
      if (!Array.isArray(entries) || entries.length < 2) throw new Error('A transaction needs at least two entries');
      const currency = entries[0].currency;
      let sum = 0;
      for (const e of entries) {
        if (!e.account) throw new Error('Every entry needs an account');
        if (!Number.isSafeInteger(e.amount) || e.amount === 0) throw new Error(`Amount must be a non-zero integer in minor units, got ${e.amount}`);
        // One currency per transaction: an FX conversion is two transactions through an FX account.
        if (e.currency !== currency) throw new Error(`Mixed currencies in one transaction: ${currency} and ${e.currency}`);
        sum += e.amount;
      }
      if (sum !== 0) throw new Error(`Unbalanced transaction: entries sum to ${sum}`);
    }
    
    export class Ledger {
      #txs = new Map(); // id → transaction, e.g. 'pay_1:charge'
      #entries = []; // append-only: nothing is ever updated or deleted
    
      /** Post a transaction once. Re-posting the same id is a no-op; reusing an id for other entries is an error. */
      post({ id, entries, memo = '' }) {
        validate(entries);
        const seen = this.#txs.get(id);
        if (seen) {
          if (JSON.stringify(seen.entries) !== JSON.stringify(entries)) throw new Error(`Transaction ${id} was already posted with different entries`);
          return { ...seen, duplicate: true };
        }
        const tx = Object.freeze({ id, memo, entries: Object.freeze(entries.map((e) => Object.freeze({ ...e }))) });
        this.#txs.set(id, tx);
        for (const e of tx.entries) this.#entries.push({ txId: id, ...e });
        return { ...tx, duplicate: false };
      }
    
      /** Balances are derived from the entries, never stored as the source of truth. */
      balance(account, currency = 'USD') {
        let sum = 0;
        for (const e of this.#entries) if (e.account === account && e.currency === currency) sum += e.amount;
        return sum;
      }
    
      /** Every account's balance per currency: the trial balance, which must sum to 0 in each currency. */
      trialBalance() {
        const out = {};
        for (const e of this.#entries) {
          out[e.currency] ??= {};
          out[e.currency][e.account] = (out[e.currency][e.account] ?? 0) + e.amount;
        }
        return out;
      }
    
      get entries() {
        return this.#entries.map((e) => ({ ...e }));
      }
    }
    
    /**
     * The posting for one successful card payment on a marketplace. The PSP keeps its fee
     * (2.9% + 30¢ here) and the platform keeps a commission; the rest is owed to the merchant.
     * The merchant's share is the remainder, so rounding can never create or lose a cent.
     */
    export function chargeEntries({ amount, currency = 'USD', merchantId, feeBps = 290, feeFixed = 30, commissionBps = 1000 }) {
      const fee = Math.round((amount * feeBps) / 10_000) + feeFixed;
      const commission = Math.round((amount * commissionBps) / 10_000);
      return [
        { account: 'cash_at_psp', amount: amount - fee, currency }, // asset: the PSP holds it until settlement
        { account: 'psp_fees', amount: fee, currency }, // expense
        { account: `merchant_payable:${merchantId}`, amount: -(amount - commission), currency }, // liability: we owe the merchant
        { account: 'platform_revenue', amount: -commission, currency }, // revenue
      ];
    }
    
    /** A refund the merchant pays for: the money goes back out of what we hold at the PSP. */
    export function refundEntries({ amount, currency = 'USD', merchantId }) {
      return [
        { account: `merchant_payable:${merchantId}`, amount, currency },
        { account: 'cash_at_psp', amount: -amount, currency },
      ];
    }

    Failure modes and trade-offs

    Read more · 198 words
    What breaksEffectMitigation
    PSP times outCharged or not: unknownunknown; same-key retry; webhook, then status query
    DB down after the PSP chargedMoney moved, no recordIntent committed first; webhook retries; reconciliation
    Webhook stormSlow handler, more retriesStore, answer 200, process from a queue
    Double submitTwo chargesUnique key; 409 while running
    Two refunds at onceToo much refundedrefunded + $2 <= amount in the UPDATE
    Currency roundingCents vanishInteger minor units; remainder leg
    Outbox relay downOrders not marked paidRows wait; alert on the oldest

    Choices to defend:

    ChoiceGainCost
    Ledger, not a balance columnFull history, auditableMore rows to sum
    Key in the DB, not RedisAtomic with the paymentMore load on the primary
    Charge now, not asyncInstant, simpleTied to the PSP's speed
    Hold then captureFree voidsHolds expire in about 7 days

    Follow-up questions

    How do you handle multiple currencies?

    Tag every entry with its currency. Never mix two in one . Convert through an FX account and store the rate.

    How do payouts to merchants work?

    A daily job pays each merchant's balance minus a refund reserve. Failed payouts reverse with new entries.

    Where do fraud checks go?

    Before the PSP call. Low risk passes, medium asks for extra checks, high is blocked. If the risk service is down, allow small amounts only.

    Can you fail over to a second PSP?

    Only if the first certainly never charged. After a timeout, a second charge can mean two. Settle the first through its .

    Reconciliation found a mismatch. Now what?

    Fix forward, never by editing entries. Replay a missed webhook or reverse a bad posting. Send the unexplained to a human.

    Check yourself

    Question 1 of 3

    Score 2 out of 3 or better to mark this page complete.

    1. The PSP call times out after 10 s. What should happen to the payment?