Replication

Leader-follower replication

In leader-follower (primary-replica) replication, one node accepts all writes and streams its change log to followers, which apply the same changes in order. Postgres streaming replication ships WAL records; MySQL ships binlog events. Reads can go to any replica, which is how most read-heavy systems scale: one primary handling writes with 5 replicas can serve roughly 6x the read throughput.

The key choice is synchronous versus asynchronous propagation. Asynchronous replication acknowledges the write once the leader commits, giving low latency but risking loss of the last few writes if the leader dies before followers catch up. Synchronous replication waits for at least one follower to confirm, guaranteeing durability at the cost of latency and availability (a slow follower stalls writes). Semi-synchronous setups, one sync follower plus several async ones, are a common production compromise.

This topology dominates practice: Postgres, MySQL, MongoDB replica sets, Redis, and Kafka partitions (each partition has a leader and in-sync replicas) all use it. Its fundamental limits are that write throughput is capped by one node and that failover is a hard problem.

Replication lag and read consistency

Asynchronous followers lag the leader, typically by milliseconds but sometimes by seconds or minutes under load, during network hiccups, or while replaying a large migration. Any read served by a lagging replica can return stale data, which produces user-visible anomalies that interviewers love to probe.

The canonical anomaly is violating read-your-writes: a user updates their profile (write hits the leader), the confirmation page reads from a replica that has not applied the change, and the user sees their old profile and files a bug. Standard fixes: route a user's reads to the leader for a short window after they write (for example 10 seconds, or until the replica's replay position passes the write's log position), pin each session to a replica at least as fresh as its last write, or have clients send a minimum log sequence number with reads.

Two related anomalies are worth naming. Monotonic reads: a user refreshing a page must not see data go backward in time, which happens if consecutive reads hit replicas with different lag; pinning a session to one replica fixes it. Consistent prefix: comments must not appear before the post they reply to, an issue mainly in partitioned systems where different partitions replicate at different speeds.

Failover and its failure modes

When the leader dies, a follower must be promoted. Automatic failover involves detecting the failure (usually a heartbeat timeout of 10 to 30 seconds), electing the most up-to-date follower, and repointing clients. Every step can go wrong, which is why systems like Patroni for Postgres, MySQL group replication with a consensus layer, and MongoDB's Raft-based elections exist.

The two classic hazards are lost writes and split brain. With asynchronous replication, the promoted follower may be missing the dead leader's last writes; when the old leader returns, its divergent writes are typically discarded, and GitHub's 2018 incident (a 43 second network partition led to writes on two masters and hours of reconciliation) is the standard cautionary tale. Split brain, two nodes both believing they are leader, corrupts data fast; the defenses are quorum-based elections (a leader needs majority acknowledgment) and fencing, forcibly isolating the old leader (the grimly named STONITH: shoot the other node in the head).

A practical interview detail: failover time is part of your availability budget. If detection takes 15 seconds and promotion 15 more, every unplanned leader failure costs 30 seconds of write downtime, which alone nearly exhausts a 99.99 percent monthly budget of about 4.3 minutes.

Multi-leader and leaderless replication

Multi-leader replication lets several nodes accept writes, typically one leader per region, with leaders replicating to each other asynchronously. It gives each region low write latency and tolerates region-level partitions, but concurrent writes to the same record in different regions conflict. Resolution strategies include last-writer-wins (simple but silently drops data, and clock skew makes 'last' unreliable), application-level merge logic, and CRDTs (conflict-free replicated data types) that merge mathematically, used by Redis Enterprise CRDBs and Riak.

Leaderless replication, from Amazon's Dynamo paper and implemented by Cassandra and Riak, has no leader at all: clients (or coordinators) write to N replicas and consider the write successful after W acknowledgments; reads query R replicas and take the newest value. If R + W > N (commonly N=3, W=2, R=2), read and write sets overlap and reads see the latest acknowledged write, giving tunable consistency per request. Lower W or R buys latency and availability at the price of staleness.

Leaderless systems repair divergence continuously: read repair updates stale replicas noticed during reads, hinted handoff stores writes destined for a down node on a neighbor until it recovers, and anti-entropy processes compare replicas in the background using Merkle trees. There is no failover event because there is no leader to fail, which is precisely why Cassandra targets always-writable workloads across regions.

Tradeoffs