Distributed Transactions and Sagas

Two-Phase Commit and Why It Is Rarely Used

Two-phase commit (2PC) makes multiple resource managers commit atomically. In phase one (prepare), a coordinator asks every participant to get the transaction durable and locked, and each votes yes or no; in phase two, if all voted yes the coordinator writes a commit record and tells everyone to commit, otherwise it broadcasts abort. The protocol guarantees atomicity: either all participants commit or none do.

The fatal flaw is blocking on coordinator failure. A participant that voted yes is in the in-doubt state: it holds locks and cannot unilaterally commit or abort, because it does not know the global decision. If the coordinator crashes after prepare, participants can hold locks for the entire coordinator recovery time, stalling every other transaction that touches those rows. 2PC also requires all participants to speak the protocol (XA support), and its multiple synchronous round trips make it slow: throughput drops by an order of magnitude versus local transactions in typical measurements. This is why 2PC across heterogeneous services (your Postgres, someone's REST API, a Kafka topic) is effectively a non-starter, and why modern microservice architectures avoid it.

Where 2PC-family protocols do live on is inside tightly controlled infrastructure: Google Spanner runs 2PC across Paxos groups, using Paxos replication to make both the coordinator and participants highly available, which removes the classic single-coordinator blocking problem at the cost of significant engineering and TrueTime infrastructure. The interview takeaway: 2PC gives atomicity but sacrifices availability and latency, and is viable only when one team controls all participants.

Sagas: Choreography vs Orchestration

A saga replaces one distributed transaction with a sequence of local transactions, each committed independently, plus a compensating transaction for each step to semantically undo it if a later step fails. Booking a trip becomes: reserve flight, reserve hotel, charge card; if the charge fails, run the compensations cancel hotel then cancel flight. Compensations are semantic, not rollbacks: you cannot un-send an email, so you send a correction; you refund a charge rather than erasing it. Steps must therefore be designed so that compensation is possible, which sometimes means introducing a pending state (reserve, then confirm) rather than acting irreversibly, and some steps are pivot points after which the saga must run forward to completion because compensation is no longer possible.

Choreography implements the saga through events with no central controller: order service publishes OrderPlaced, payment service reacts and publishes PaymentCaptured, inventory reacts to that, and failure events trigger compensating reactions. It is loosely coupled and has no single point of failure, but the workflow exists nowhere as an artifact: to answer where is order 123 stuck, you grep event streams across five services, and adding a step means changing multiple services' subscriptions. It fits sagas of 2 to 4 steps.

Orchestration puts a saga orchestrator in charge: it sends commands (ReservePayment, ReserveInventory), receives replies, persists the saga's state machine, and drives compensations on failure. The workflow is explicit, queryable, and testable in one place, at the cost of the orchestrator being a component that must itself be highly available and at some risk of accumulating business logic that belongs in the services. Tools like Temporal, AWS Step Functions, and Camunda exist precisely to make orchestrator state durable and recoverable. For sagas of 5 or more steps, or anything with timeouts, retries, and human approval steps, orchestration is generally the right call, and Uber, DoorDash, and Netflix all run large workflow orchestration platforms (Temporal came out of Uber's Cadence) for this reason. Note the isolation caveat: sagas have no I in ACID, so other transactions can observe intermediate states (order placed but not yet paid), which must be acceptable or masked with status fields.

Idempotency: The Load-Bearing Wall

Every retry-based mechanism in distributed systems (at-least-once queues, HTTP retries, saga step retries) rests on idempotency: performing the same operation twice has the same effect as once. Without it, a timeout plus retry double-charges a card. The canonical implementation is the idempotency key: the client generates a unique key per logical operation (a UUID per checkout attempt), sends it with the request, and the server atomically checks-and-records the key alongside the side effect; on a duplicate it returns the stored result of the first execution without re-executing. Stripe's API works exactly this way via the Idempotency-Key header, storing responses for 24 hours, and it is the reference example to cite.

Implementation details that interviewers probe: the key check and the business write must be in the same transaction (or use a unique constraint on the key column and treat the violation as a duplicate), otherwise a race between two concurrent retries executes twice. Keys need a TTL and a scope (per operation type). And you must decide what a duplicate with a different payload means, usually a 422 error rather than silent acceptance.

Alternatives and complements: natural idempotency by design (UPSERT with a deterministic ID, set-status operations rather than increments), deduplication tables keyed by message ID on consumers, and conditional writes (compare-and-set with a version number) that make replays harmless. A useful framing: exactly-once processing is always implemented as at-least-once delivery plus idempotent handling; there is no other trick.

Distributed Locks and Fencing Tokens

A distributed lock ensures at most one process acts on a resource at a time across machines, used for leader election, cron singletons, and guarding non-idempotent external actions. Implementations include a lease in Redis (SET key value NX PX 30000), or a session-based lock in ZooKeeper or etcd. Every practical lock is a lease with an expiry, because a lock without expiry plus a crashed holder equals a permanent deadlock.

Leases create the classic safety bug that Martin Kleppmann's critique of Redlock made famous: process A acquires the lease, hits a 40-second GC pause or network partition, the lease expires, process B acquires it and starts writing, then A wakes up still believing it holds the lock and writes too, corrupting data. The lock service was correct; the client's belief was stale. Timeouts alone cannot fix this because you can never distinguish a slow process from a dead one.

The fix is fencing tokens: the lock service hands out a strictly monotonically increasing number with each lease grant (ZooKeeper's zxid or a version counter in etcd works naturally), the client includes the token with every write, and the protected resource (the storage service) rejects any write bearing a token lower than the highest it has seen. When zombie A writes with token 33 after B wrote with token 34, storage rejects A. The crucial architectural implication is that the resource itself must participate by checking tokens, which is also why fencing sometimes degenerates into just use conditional writes or transactions in the storage layer directly, and why the best interview answer often is: prefer making the operation idempotent or using the database's own concurrency control, and reach for distributed locks only when coordinating an external, non-transactional side effect.

Tradeoffs