Design a Payment System
Design the payment backend for a commerce platform: accept a customer's payment through a PSP like Stripe, record money movement in a double-entry ledger, pay merchants out, and guarantee that retries, crashes, and PSP flakiness never charge anyone twice or lose a cent. Correctness dominates every other concern.
1Requirements
Functional
- • Pay-in: charge a buyer for an order via a PSP (Stripe/Adyen); the platform never touches raw card numbers (PCI scope stays at the PSP).
- • Record every money movement in an immutable double-entry ledger that is the internal source of truth.
- • Pay-out: transfer accumulated funds to merchants on a schedule, net of platform fees.
- • Support refunds (full and partial) and surface PSP-initiated events like chargebacks and disputes.
- • Reconcile internal ledger state against PSP settlement reports daily and flag every discrepancy.
- • Expose payment status to order systems via API and webhooks/events.
Non-functional
- • Correctness over availability: it is better to fail a payment visibly than to double-charge or record wrong amounts; core writes are strongly consistent and ACID.
- • Effectively exactly-once money movement built from at-least-once delivery plus idempotency everywhere.
- • Durability and auditability: ledger entries are append-only, immutable, and retained for 7+ years.
- • Latency: authorization round-trip dominated by the PSP (~1-2s); internal overhead budget under ~100ms.
- • Availability target 99.99% on the pay-in path; degraded mode queues payments rather than dropping them.
2Back-of-envelope estimation
| Payment volume | 10M payments/day | ~116/s average; Black Friday peak ~10x = ~1.2K/s. Modest QPS: payments are a correctness problem, not a throughput problem. |
| Ledger write rate | ~60M entries/day | Each payment yields ~3 transactions (auth/capture, fee, payable) x 2 entries each (double-entry); still under 1K writes/s average, comfortably single-primary Postgres territory per shard. |
| Ledger growth | ~30 GB/day, ~11 TB/year | 60M entries x ~500 bytes; 7-year retention ≈ 77 TB, pushing old partitions to cheap append-only archive storage. |
| Reconciliation batch | 10M PSP settlement records/day | A daily batch join of PSP reports against internal ledger; even a 0.01% mismatch rate = 1,000 cases/day, so auto-classification of discrepancies is mandatory. |
| PSP fees at stake | ~$870K/day | 10M x $50 avg x ~1.75% blended PSP fee (roughly 2.9% + $0.30 on cards): fee accounting itself is material money and must be in the ledger. |
3API design
POST /v1/payments (header: Idempotency-Key)Initiates a pay-in: {order_id, amount_minor, currency, payment_method_token}. Amounts are integer minor units, never floats. Returns the same result for the same key no matter how many times it is called.
GET /v1/payments/{paymentId}Returns payment state (created, pending, succeeded, failed, refunded) and associated ledger transaction ids; polling fallback for consumers of the async status events.
POST /v1/payments/{paymentId}/refunds (header: Idempotency-Key)Full or partial refund; validated against remaining refundable amount, executed at the PSP, and recorded as a compensating ledger transaction.
POST /v1/psp-webhooks/{provider}Receives PSP events (payment_intent.succeeded, charge.dispute.created, payout.paid). Signature-verified, persisted, deduped by event id, then processed async; webhooks are treated as hints, with polling as the fallback truth.
4High-level design
Checkout tokenizes card details directly against the PSP (Stripe Elements), so raw PANs never touch platform servers and PCI scope collapses to SAQ-A. The order service then calls the payment service with an idempotency key derived from the order attempt.
The payment service is the orchestrator and keeps a state machine per payment. On a new request it writes the payment row in 'created', then calls the PSP to create/confirm a PaymentIntent, passing the same idempotency key to Stripe so PSP-side retries are also safe. State transitions are recorded before and after each external call so a crash at any point leaves a resumable record rather than mystery money.
Confirmed outcomes are written to the ledger service: every transaction is a balanced set of double-entry postings (debit PSP receivable, credit merchant payable and platform fee revenue) inside one ACID transaction in Postgres. The ledger is append-only; corrections are new reversing entries, never updates, which is what makes it auditable.
Asynchrony rides on a transactional outbox: the ledger commit and an outbox event are one DB transaction, and a relay publishes to Kafka, so downstream consumers (order fulfillment, notifications, analytics, payout scheduling) see exactly the committed truth. PSP webhooks flow into the same event backbone after signature verification and dedup, advancing payment state machines for async outcomes like disputes.
A scheduled payout service aggregates each merchant's payable balance from the ledger, initiates transfers via the PSP's payout API (again idempotently), and records the corresponding ledger entries. Nightly reconciliation jobs pull PSP settlement files, match them against ledger transactions three ways (internal ledger vs. PSP records vs. bank statement), auto-classify known mismatch patterns (timing, fees, currency rounding), and open cases for humans on the rest.
5Data model
payment
id BIGINT PK, idempotency_key VARCHAR UNIQUE, order_id BIGINT, buyer_id BIGINT, merchant_id BIGINT, amount_minor BIGINT, currency CHAR(3), status VARCHAR, psp VARCHAR, psp_payment_intent_id VARCHAR UNIQUE, created_at TIMESTAMP, updated_at TIMESTAMPThe state machine row; unique constraints on both the idempotency key and the PSP intent id are the double-charge backstops.
ledger_transaction
id BIGINT PK, type VARCHAR, payment_id BIGINT, external_ref VARCHAR, posted_at TIMESTAMP, description VARCHARGroups a balanced set of entries; immutable once posted.
ledger_entry
id BIGINT PK, transaction_id BIGINT FK, account_id BIGINT, direction CHAR(1), amount_minor BIGINT, currency CHAR(3), CHECK (amount_minor > 0)Per transaction, sum(debits) must equal sum(credits) per currency; enforced in the service and by invariant checks. Balances are derived (materialized per account) rather than stored as mutable truth.
psp_event
id BIGINT PK, provider VARCHAR, event_id VARCHAR UNIQUE, type VARCHAR, payload JSONB, signature_valid BOOLEAN, processed_at TIMESTAMP, received_at TIMESTAMPRaw webhook archive; unique event_id gives webhook dedup, and retaining payloads makes disputes and incident forensics tractable.
6Deep dives
Idempotency: the backbone of not charging twice
Retries are unavoidable: clients time out, networks drop responses, queues redeliver. The contract that makes retries safe is an idempotency key per logical operation. On first sight of a key the service atomically inserts a record (unique constraint) and proceeds; on any later sight it returns the stored outcome of the original attempt. The key must be scoped to the operation (payment attempt for order X), persisted with the response, and honored across every state the original attempt might be in, including 'still in progress', where the correct response is the in-progress state, not a second execution.
The subtle failure mode is the crash between committing your intent and calling the PSP, or between the PSP call and recording its result. The fix is to persist state transitions around the external call ('psp_call_pending' before, outcome after) and pass the same idempotency key to Stripe, which supports idempotency keys natively for exactly this reason. On recovery, a sweeper finds payments stuck in pending, queries the PSP for the intent's actual status, and resumes the state machine. You never guess; you ask the PSP what happened.
Exactly-once is thus an end-to-end illusion assembled from at-least-once retries plus idempotent effects at every hop: client to API (idempotency key), API to PSP (Stripe idempotency key + unique intent id), PSP to platform (webhook event-id dedup), platform to consumers (outbox + consumer-side dedup). Any single hop lacking idempotency reintroduces double-charging, which is why the interviewer will probe each hop.
Why a double-entry ledger, and how to build one
Single-entry records ('merchant balance += $48.25') destroy information: when a balance is wrong you cannot tell why. Double-entry records every movement as balanced debits and credits between accounts: a $50 sale posts debit psp_receivable $50, credit merchant_payable $48.25, credit platform_fees $1.75 minus PSP cost. The invariant that all entries in a transaction sum to zero per currency is checkable at write time and continuously afterward, so entire classes of bugs (money created or destroyed) become detectable instantly rather than at month-end.
Implementation rules that matter: append-only (corrections are reversing entries, so history is never rewritten and audits can replay everything), integer minor units with explicit currency (floating point is disqualifying in a payments interview), and both legs plus the transaction row written in one ACID transaction. Account balances are derived state: computed from entries and materialized with snapshots (balance as of entry N) so reads are fast without making a mutable balance the truth.
Scaling the ledger is deliberately boring: partition by account or merchant, keep hot partitions on a strongly consistent primary, archive old partitions. At ~1K writes/s average you do not need a distributed database; you need the transaction and the invariants. Reaching for eventual consistency in the ledger is the trap answer.
PSP integration: webhooks, state machines, and never trusting one channel
The Stripe-style flow: create a PaymentIntent server-side, confirm it from the client (which handles 3-D Secure challenges), then learn the outcome. Outcomes arrive on two channels: the synchronous API response and asynchronous webhooks, and neither is sufficient alone. The sync response can be lost to a timeout after the charge succeeded; webhooks are delivered at-least-once, out of order, and occasionally late. The robust pattern is to treat webhooks as triggers, dedupe them by event id, verify signatures, and let the state machine advance only along legal transitions, with a reconciling poller that queries the PSP directly for any payment stuck in a non-terminal state past a deadline.
Out-of-order handling falls out of the state-machine design: if 'payment_intent.succeeded' arrives after you already processed it via polling, the transition is a no-op; if a 'charge.refunded' arrives for a payment you have not marked succeeded, park the event and re-drive it after fetching current PSP state. Idempotent transitions plus a poll-based source of truth make webhook ordering irrelevant to correctness.
Plan for PSP failure as a first-class scenario. Short outages: queue payment intents internally and drain when the PSP recovers, telling the buyer 'processing' rather than failing checkout. Sustained degradation: multi-PSP routing with health-based failover, though this multiplies integration and reconciliation surface (different fee schedules, settlement formats, refund semantics), so it is a deliberate business decision, not a free redundancy trick.
Reconciliation: the safety net that catches everything else
Every mechanism above can still leak: a webhook lost past retry windows, a bug that posts a fee wrong, a PSP settling an amount that differs from the auth. Reconciliation is the periodic proof that internal records match external reality. Daily, the PSP's settlement report (every charge, refund, fee, chargeback, and the net payout) is matched against the ledger, and the bank statement is matched against expected payouts: a three-way check between what we recorded, what the PSP says happened, and what actually hit the bank.
Most mismatches are benign and auto-classifiable: timing differences (charge on day N, settlement on N+1), FX rounding, fee-schedule drift. The pipeline should auto-clear these categories and emit metrics on their rates, escalating only genuine breaks (missing transaction, amount mismatch, unknown charge) to a human-worked case queue with links to the ledger transaction, PSP objects, and raw webhook archive. A discrepancy rate creeping upward is often the first observable symptom of a code bug in fee logic or a stuck consumer.
Design choices upstream determine whether reconciliation is tractable: stable external references on every ledger transaction (PSP charge id), immutable raw webhook storage, and append-only ledger history are what make 'investigate this $0.30 break from last Tuesday' a ten-minute job instead of an archaeology project. Interviewers rate candidates who volunteer reconciliation unprompted, because it signals operational experience with money systems.
7Rapid implementation: build the MVP
Theory is table stakes. Here is how you would stand up a working version fast, on a budget, with the core algorithm in real code.
Stack
Node.js + Express, Postgres for payments and the ledger, Stripe in test mode (no real money), deployed on a Render free-tier instance.
- 01Create Postgres tables payments, idempotency_keys, ledger_transactions, and ledger_entries; every amount is an integer in minor units (cents).
- 02Write the idempotency middleware and mount it on POST /payments and POST /refunds; require the Idempotency-Key header.
- 03Integrate Stripe test mode: create and confirm a PaymentIntent server-side, passing the same idempotency key through to Stripe.
- 04On confirmed success, post the balanced double-entry ledger transaction (receivable debit, merchant payable and fee credits) in one DB transaction, then store the response against the idempotency key.
- 05Add the webhook endpoint: verify the Stripe signature, insert the event with a unique event_id (duplicates no-op), and advance the payment state machine only along legal transitions.
- 06Add a sweeper cron that finds payments stuck in pending for over 10 minutes, queries Stripe for the intent's real status, and resumes the state machine; never guess.
- 07Write a nightly reconciliation script that pulls Stripe balance transactions and diffs them against the ledger, printing every mismatch.
- 08Test with Stripe CLI webhook replays and duplicate POSTs to prove the same key never charges twice.
Idempotency-key middleware
typescriptexport function idempotency(pool: Pool) {
return async (req: Request, res: Response, next: NextFunction) => {
const key = req.header("Idempotency-Key");
if (!key) {
return res.status(400).json({ error: "Idempotency-Key required" });
}
// atomic claim: unique constraint decides who runs the handler
const claim = await pool.query(
"INSERT INTO idempotency_keys (key, status) " +
"VALUES ($1, 'in_progress') ON CONFLICT (key) DO NOTHING " +
"RETURNING key",
[key]
);
if (claim.rowCount === 1) {
res.locals.idemKey = key; // handler stores its response under this
return next();
}
const prior = await pool.query(
"SELECT status, response FROM idempotency_keys WHERE key = $1",
[key]
);
if (prior.rows[0].status === "in_progress") {
// original attempt still running: report it, never re-execute
return res.status(409).json({ error: "request already in progress" });
}
return res.status(200).json(prior.rows[0].response); // replay outcome
};
}Double-entry ledger posting in one transaction
sql-- A $50.00 capture: all rows commit together or not at all.
BEGIN;
INSERT INTO ledger_transactions (id, type, payment_id, external_ref)
VALUES (11, 'capture', 42, 'pi_3XyzStripeIntentId');
-- balanced postings in integer cents: debits equal credits
INSERT INTO ledger_entries
(transaction_id, account_id, direction, amount_minor) VALUES
(11, 1001, 'D', 5000), -- psp_receivable
(11, 2042, 'C', 4825), -- merchant_payable (merchant 42)
(11, 3001, 'C', 175); -- platform_fee_revenue
-- invariant check: divide by zero aborts the whole transaction
-- if debits and credits do not net to zero
SELECT 1 / CASE WHEN COALESCE(SUM(
CASE direction WHEN 'D' THEN amount_minor ELSE -amount_minor END
), -1) = 0 THEN 1 ELSE 0 END
FROM ledger_entries WHERE transaction_id = 11;
COMMIT;
-- corrections are new reversing entries; rows are never updatedBottlenecks & failure modes
- ⚠The PSP itself: 1-2s auth latency and third-party availability dominate the user experience; internal queuing and multi-PSP failover are the levers.
- ⚠Hot ledger accounts (the platform fee account is credited on every payment) serialize writes; mitigate with sub-account sharding summed at read time.
- ⚠Idempotency-key storage on the hot path must be strongly consistent; a cache-only implementation reintroduces double-charge risk on cache loss.
- ⚠Webhook bursts after a PSP incident (hours of queued events delivered at once) can flood consumers; dedup plus state-machine no-ops make the flood safe, rate limiting makes it survivable.
- ⚠Reconciliation case volume scales with payment volume; without auto-classification, ops headcount becomes the system's real bottleneck.
Key takeaways
- ▸Payments is a correctness problem at modest QPS: choose ACID, strong consistency, and visible failure over availability tricks.
- ▸Exactly-once money movement is assembled from at-least-once delivery plus idempotency at every hop: client key, PSP key, webhook event-id dedup, outbox consumers.
- ▸The double-entry ledger with append-only balanced entries in integer minor units is the internal source of truth; balances are derived, never mutated.
- ▸Drive everything through explicit per-payment state machines; on ambiguity (crash, timeout, weird webhook order) query the PSP and resume, never guess.
- ▸Reconciliation against PSP and bank records is the mandatory safety net; design ledger references and webhook archives so breaks are cheap to investigate.