Hard

Design a Digital Wallet

A digital wallet service where users hold balances and transfer money to each other instantly. Money must never be created, destroyed, or double-spent, so the design centers on a double-entry ledger, idempotent transfer operations, distributed transactions when wallets live on different shards, and the ability to audit and rebuild any balance by replaying the event log.

1Requirements

Functional

  • Users hold a wallet balance and can top up from an external payment method and withdraw to a bank account.
  • Users transfer funds to other users; transfers are atomic: either both balances change or neither does.
  • Every transfer is recorded as immutable double-entry ledger entries; balances are derivable from the ledger alone.
  • Clients can safely retry any transfer without risk of double execution (idempotency keys).
  • Users see their transaction history with running balance, and support staff can audit any account's full lineage.
  • Balances can never go negative; insufficient funds fails the transfer atomically.

Non-functional

  • Correctness over availability: reject transfers during partitions rather than risk double-spending (CP system).
  • Transfer commit latency under 500 ms p99 including durable replication.
  • Scale to 100 million wallets and 20K transfers per second peak, which forces sharding and cross-shard transfers.
  • The ledger is append-only and tamper-evident; no UPDATE or DELETE ever touches posted entries.
  • Full recoverability: any wallet balance must be reconstructible by replaying its ledger entries from genesis.
  • Regulatory-grade auditability: every state change traces to a request id, actor, and timestamp.

2Back-of-envelope estimation

Transfer TPS20K peak, ~5K average
Ledger entries per transfer2 (plus 2 for fees if charged)
Ledger growth~29 GB/day
Cross-shard transfer fraction~99% with random sharding
Idempotency key cache~35 GB for 7 days

3API design

POST /v1/transfers {idempotencyKey, fromWalletId, toWalletId, amount, currency}

Executes an atomic transfer. Retries with the same idempotencyKey return the original result, never a second execution.

GET /v1/transfers/{transferId}

Returns transfer state: pending, completed, or failed, with the ledger entry ids it produced.

GET /v1/wallets/{walletId}/balance

Current available balance plus any holds; consistent read served from the wallet's home shard.

GET /v1/wallets/{walletId}/entries?cursor=...

Paginated immutable ledger entries with running balance for statements and audit.

POST /v1/topups {idempotencyKey, walletId, amount, paymentMethodId}

Credits a wallet from an external processor; double-entry against a corporate clearing account.

4High-level design

The system's source of truth is an append-only ledger. Every money movement produces balanced entries: a transfer of 50 from Alice to Bob writes a debit entry on Alice's account and a credit entry on Bob's account inside one transaction, and the invariant sum(debits) = sum(credits) holds over the whole ledger at all times. External money entering or leaving the system is balanced against internal clearing accounts, so even top-ups obey double entry. Balances are a materialized view: a cached aggregate per wallet, always rebuildable by replaying entries.

The transfer service is the only writer to the ledger. Each request carries a client-generated idempotency key; the service records (key, result) atomically with the transfer itself, so a retried request short-circuits to the stored result. This makes at-least-once delivery from clients and queues safe.

Wallets are sharded by wallet id across Postgres (or Spanner-style) shards. A same-shard transfer is one local ACID transaction: lock both wallet rows in a deterministic order (lower id first, preventing deadlock), check funds, write both entries, update both cached balances, commit. Cross-shard transfers use a saga with reserved funds: phase 1 debits and holds the money on the source shard (writing a pending ledger entry), phase 2 credits the destination shard, and a completion step marks the transfer done. A recovery worker scans for stuck pending transfers and either completes or compensates (releases the hold) based on the recorded state, giving effective atomicity without holding cross-shard locks.

Every committed transfer also emits an event to a durable log (Kafka or the ledger table itself streamed via CDC). Downstream consumers build read models: transaction history, analytics, fraud scoring, and reconciliation jobs that continuously verify that cached balances equal replayed ledger sums and that the global ledger balances to zero. Any discrepancy pages a human; the ledger, not the cache, wins every dispute.

For audit and disaster recovery, the event log enables replay: a corrupted balance table or a new read model is rebuilt by replaying entries in order. Periodic snapshots (balance as of entry N) bound replay time, exactly like snapshots in event sourcing.

5Data model

wallets

wallet_id, user_id, currency, cached_balance, held_amount, version, updated_at

cached_balance is a derived value; version supports optimistic checks. Sharded by wallet_id.

ledger_entries

entry_id, transfer_id, wallet_id, direction (debit|credit), amount, currency, balance_after, created_at

Append-only, no updates or deletes ever. amount stored as BIGINT minor units, never floating point.

transfers

transfer_id, idempotency_key, from_wallet_id, to_wallet_id, amount, currency, state (pending|completed|failed|compensated), created_at, completed_at

Unique index on idempotency_key enforces exactly-once effect; state machine drives cross-shard recovery.

idempotency_results

idempotency_key, transfer_id, response_body, expires_at

Lets retries return the exact original response; TTL 7 days.

6Deep dives

Why double-entry, not a balance column

A single balance column updated in place is how money silently disappears. A crashed process between two UPDATEs, a retried message, or a bug leaves no trail: you know the balance is wrong but not why. Double-entry bookkeeping, unchanged since the 15th century, fixes this structurally: every movement writes a debit in one account and an equal credit in another, both in one transaction. The global invariant sum(all debits) = sum(all credits) means money is conserved by construction; any bug that violates it is detectable by a reconciliation query.

Balances become derived data: balance(wallet) = sum(credits) - sum(debits) over its entries. You cache this for reads, and you store balance_after on each entry so statements show running balances without aggregation, but the entries are the truth. When cache and ledger disagree, the ledger wins and the cache is rebuilt.

Two implementation rules that interviewers probe: store amounts as integer minor units (cents), never floats, because 0.1 + 0.2 problems are unacceptable in money; and make entries strictly append-only, with corrections done by reversing entries, never by mutation, so the audit trail is complete.

Idempotent transfers end to end

Money movement over a network faces the classic uncertainty: the client sends a transfer, the response times out, and the client cannot know whether it executed. Without protection, the natural response (retry) double-spends. The fix is an idempotency key: the client generates a UUID per logical transfer and sends it on every retry of that transfer.

Server side, the key must be recorded in the same atomic transaction as the transfer's effects. Insert the transfer row with a unique constraint on idempotency_key; if the insert conflicts, the transfer already happened (or is in flight), so read and return its stored result. Checking the key in a separate step before the transaction (check-then-act) reintroduces the race: two concurrent retries both pass the check and both execute. The unique constraint is the serialization point.

Idempotency must extend through the whole pipeline. Queue consumers processing transfer events must be idempotent too (the ledger insert keyed by transfer_id conflicts on redelivery), and calls out to external payment processors must forward an idempotency key so the processor also will not double-charge. Exactly-once effect is achieved by at-least-once delivery plus idempotent handlers at every hop.

Cross-shard transfers: saga with reserved funds

Once wallets are sharded, most transfers touch two shards and a single ACID transaction is off the table. Two-phase commit (2PC) is the textbook answer but couples availability of every transfer to a coordinator and holds locks across a network round trip; a wounded coordinator leaves participants blocked. Most production wallets instead use a saga: a sequence of local transactions with recorded state and compensations.

The flow: (1) On the source shard, atomically check funds, move the amount from available balance to held, write a pending debit entry, and set transfer state to source_debited. (2) On the destination shard, write the credit entry and update the balance; set state to completed and convert the hold into a posted debit. If step 2 fails permanently, a compensation releases the hold and reverses the pending debit, state becomes compensated. Every step is idempotent (keyed by transfer_id) so the recovery worker can re-drive any step after a crash.

The user-visible semantics are: money leaves the sender immediately (held), arrives at the receiver within milliseconds normally, and in the failure case returns to the sender. Money is never in both places and never in neither, from any observer's ledger view. This is the pattern to articulate in interviews: not distributed locks, but a state machine plus idempotent steps plus a recovery sweeper.

Audit, reconciliation, and replay

The ledger doubles as an event log, which buys three capabilities. First, audit: every entry references its transfer, which references an idempotency key, actor, and timestamp, so any balance change traces to a cause. Regulators and support staff query lineage, not logs.

Second, reconciliation as a continuous process, not an incident response. A background job per shard recomputes sum-of-entries per wallet and compares to cached_balance; a global job verifies debits equal credits across shards for each time window; an external job matches processor settlement files against top-up and withdrawal entries. Discrepancies halt related accounts and page a human. Mature wallet systems treat reconciliation findings as sev-1 by default.

Third, replay: because entries are ordered and immutable, any derived state (balance cache, history view, fraud features, a new analytics model) is rebuilt by replaying entries from genesis or from a periodic snapshot. Snapshots (wallet balance as of entry N, taken daily) bound replay time to one day of entries. This is event sourcing applied where it genuinely pays for itself: the events are legally required anyway, so deriving state from them is nearly free architecture.

7Rapid implementation: build the MVP

Theory is table stakes. Here is how you would stand up a working version fast, on a budget, with the core algorithm in real code.

Stack

Node.js/TypeScript + Postgres (single instance is genuinely correct for an MVP: real ACID) + a nightly reconciliation script; Neon or Supabase free tier.

  1. 01Create the schema: wallets, transfers (unique idempotency_key), ledger_entries (append-only, BIGINT minor units), plus a REVOKE UPDATE, DELETE ON ledger_entries for everyone including the app role.
  2. 02Seed a corporate clearing wallet so top-ups and withdrawals are double-entry from day one.
  3. 03Implement POST /transfers as one Postgres transaction: insert transfer row (conflict on idempotency_key returns stored result), lock both wallets in id order with SELECT FOR UPDATE, check funds, insert debit and credit entries with balance_after, update cached balances, commit.
  4. 04Implement top-up as the same transfer primitive from the clearing wallet, gated by a fake payment-processor stub.
  5. 05Build the statement endpoint: ledger entries with running balance, paginated by entry_id.
  6. 06Write the reconciliation script: for every wallet assert cached_balance equals the entry sum, and assert the global ledger sums to zero; run it in CI and nightly.
  7. 07Torture-test idempotency: fire the same transfer 50 times in parallel (Promise.all) and assert exactly one execution and identical responses.
  8. 08Add a replay command that truncates cached balances and rebuilds them purely from ledger_entries, proving the ledger is sufficient.

Idempotent double-entry transfer in one transaction

sql
BEGIN;

-- Serialization point: a retry conflicts here and we return the stored result.
INSERT INTO transfers (transfer_id, idempotency_key, from_wallet_id, to_wallet_id, amount, state)
VALUES (:tid, :ikey, :from, :to, :amount, 'completed')
ON CONFLICT (idempotency_key) DO NOTHING;
-- If 0 rows inserted: SELECT * FROM transfers WHERE idempotency_key = :ikey; return it. Done.

-- Lock both wallets in deterministic order to avoid deadlock.
SELECT wallet_id, cached_balance FROM wallets
WHERE wallet_id IN (:from, :to)
ORDER BY wallet_id
FOR UPDATE;

-- Insufficient funds aborts everything, including the transfer row.
UPDATE wallets SET cached_balance = cached_balance - :amount
WHERE wallet_id = :from AND cached_balance >= :amount;
-- If 0 rows updated: ROLLBACK and fail with INSUFFICIENT_FUNDS.

UPDATE wallets SET cached_balance = cached_balance + :amount
WHERE wallet_id = :to;

INSERT INTO ledger_entries (transfer_id, wallet_id, direction, amount, balance_after)
VALUES
  (:tid, :from, 'debit',  :amount, (SELECT cached_balance FROM wallets WHERE wallet_id = :from)),
  (:tid, :to,   'credit', :amount, (SELECT cached_balance FROM wallets WHERE wallet_id = :to));

COMMIT;

Transfer endpoint with idempotent retry handling

typescript
import { pool } from "./db";

export async function transfer(ikey: string, from: string, to: string, amount: bigint) {
  if (amount <= 0n) throw new Error("INVALID_AMOUNT");
  const client = await pool.connect();
  try {
    await client.query("BEGIN");
    const ins = await client.query(
      "INSERT INTO transfers (idempotency_key, from_wallet_id, to_wallet_id, amount, state) " +
      "VALUES ($1, $2, $3, $4, 'completed') ON CONFLICT (idempotency_key) DO NOTHING RETURNING transfer_id",
      [ikey, from, to, amount]
    );
    if (ins.rowCount === 0) {
      await client.query("ROLLBACK");
      const prev = await pool.query("SELECT * FROM transfers WHERE idempotency_key = $1", [ikey]);
      return { replayed: true, ...prev.rows[0] }; // exact original outcome
    }
    const tid = ins.rows[0].transfer_id;
    const [a, b] = [from, to].sort(); // deterministic lock order
    await client.query("SELECT 1 FROM wallets WHERE wallet_id = ANY($1) ORDER BY wallet_id FOR UPDATE", [[a, b]]);
    const deb = await client.query(
      "UPDATE wallets SET cached_balance = cached_balance - $1 WHERE wallet_id = $2 AND cached_balance >= $1",
      [amount, from]
    );
    if (deb.rowCount === 0) { await client.query("ROLLBACK"); throw new Error("INSUFFICIENT_FUNDS"); }
    await client.query("UPDATE wallets SET cached_balance = cached_balance + $1 WHERE wallet_id = $2", [amount, to]);
    await client.query(
      "INSERT INTO ledger_entries (transfer_id, wallet_id, direction, amount, balance_after) VALUES " +
      "($1, $2, 'debit', $3, (SELECT cached_balance FROM wallets WHERE wallet_id = $2)), " +
      "($1, $4, 'credit', $3, (SELECT cached_balance FROM wallets WHERE wallet_id = $4))",
      [tid, from, amount, to]
    );
    await client.query("COMMIT");
    return { replayed: false, transfer_id: tid };
  } catch (e) {
    await client.query("ROLLBACK").catch(() => {});
    throw e;
  } finally {
    client.release();
  }
}

Reconciliation: prove the ledger balances

sql
-- 1. Global conservation law: all debits equal all credits.
SELECT
  COALESCE(SUM(amount) FILTER (WHERE direction = 'debit'), 0)  AS total_debits,
  COALESCE(SUM(amount) FILTER (WHERE direction = 'credit'), 0) AS total_credits
FROM ledger_entries;
-- Assert total_debits = total_credits; anything else is corruption.

-- 2. Every cached balance equals its replayed ledger sum.
SELECT w.wallet_id, w.cached_balance, l.ledger_balance
FROM wallets w
JOIN LATERAL (
  SELECT COALESCE(SUM(CASE direction WHEN 'credit' THEN amount ELSE -amount END), 0)
         AS ledger_balance
  FROM ledger_entries e WHERE e.wallet_id = w.wallet_id
) l ON true
WHERE w.cached_balance <> l.ledger_balance;
-- Assert zero rows; any row is a wallet to freeze and investigate.

Bottlenecks & failure modes

  • Hot wallets (merchant accounts, promo accounts) serialize on their row lock; mitigate with sub-accounts that shard one logical balance into N rows summed on read.
  • Cross-shard transfers dominate with random sharding; the saga path must be the optimized common case, and the recovery sweeper must keep pending-transfer count near zero.
  • The idempotency-results store is on the critical path of every transfer; it must be as available and durable as the ledger itself.
  • Reconciliation jobs scanning the full ledger contend with live traffic; run them on read replicas or CDC-fed copies.
  • Ledger growth is unbounded by design; partition entries by time and archive cold partitions to cheap storage while keeping them queryable for audit.

Key takeaways

  • Make the append-only double-entry ledger the source of truth and treat balances as rebuildable caches; sum(debits) = sum(credits) is a machine-checkable conservation law.
  • Idempotency keys must be persisted atomically with the transfer via a unique constraint; check-then-act patterns reintroduce the double-spend race.
  • Prefer a saga with reserved funds and an idempotent recovery sweeper over 2PC for cross-shard transfers; design the state machine, not a distributed lock.
  • Store money as integer minor units, lock accounts in deterministic id order to avoid deadlocks, and never mutate posted entries; correct with reversing entries.
  • Continuous reconciliation (cache vs ledger, ledger vs external processor) converts silent corruption into paged alerts.

Brush up on the underlying topics