Fault Tolerance Patterns

Redundancy and Failover

The foundation of fault tolerance is eliminating single points of failure through redundancy: N+1 instances behind a load balancer, replicas of every database, multiple availability zones, sometimes multiple regions. The arithmetic that motivates it: a single machine with 99 percent availability gives 87.6 hours of downtime a year, but two independent redundant machines give 99.99 percent, under the (often violated) assumption that failures are independent. Correlated failures (same rack, same AZ, same bad deploy, same certificate expiry) are why cloud architectures spread across at least three availability zones and why a single global config push is the most common cause of large outages.

Failover comes in two main shapes. Active-passive keeps a standby that takes over when the primary fails, promoted either manually or by automated health checks; it is simpler and avoids split-brain by construction, but the standby is idle cost, failover takes seconds to minutes (DNS TTLs, promotion time, connection draining), and the passive path is chronically undertested, so failovers fail exactly when needed. Active-active serves traffic from all nodes simultaneously, so failover is just the load balancer removing a dead node from rotation: near-zero recovery time and no wasted capacity, but every node must handle concurrent writes or the system must partition traffic, which is where conflict resolution and sticky routing complexity lives.

Two metrics frame every failover discussion: RTO (recovery time objective, how long until service is restored) and RPO (recovery point objective, how much data you may lose). Synchronous replication gives RPO of zero at a latency cost; async replication is faster but a failover can lose the last seconds of writes. A senior answer states the RTO/RPO target first and derives the replication and failover design from it.

Retries, Exponential Backoff, and Jitter

Retries are the first response to transient failure and the fastest way to turn a partial outage into a total one. A naive immediate retry against a struggling service multiplies its load exactly when it can least afford it: if every client retries 3 times, a service at 100 percent capacity suddenly faces 300 percent load, a retry storm. The standard discipline is exponential backoff: wait 100ms, then 200, 400, 800, capped at some maximum, so pressure decays instead of spiking.

Backoff alone is not enough because a mass failure synchronizes clients: everything that failed at time T retries at T+100ms, then T+300ms, in coordinated waves (the thundering herd). The fix is jitter, randomizing each delay. AWS's Architecture Blog analysis found full jitter (sleep a uniform random amount between 0 and the exponential cap) close to optimal: it smears the herd across time, dramatically reducing peak contention for the same total work.

Retries also need a budget and placement discipline. Retry only idempotent operations or use idempotency keys; cap total attempts (typically 2 to 3) and total elapsed time against the caller's own timeout; and retry at one layer, not every layer, because 3 retries at each of 4 layers in a call chain is 81 attempts hitting the bottom service. Mature systems use retry budgets (for example, retries may add at most 10 percent extra load, the approach used in Google SRE practice and in service meshes like Linkerd) and treat timeout choice as part of the same design: a downstream timeout must be shorter than the upstream deadline it lives inside, ideally propagating deadlines through the call chain.

Circuit Breakers and Bulkheads

A circuit breaker stops calling a dependency that is failing, converting slow cascading failures into fast local ones. It is a state machine: closed (normal operation, counting failures), open (failure rate exceeded the threshold, for example 50 percent of calls in a 10-second window, so all calls fail immediately without touching the dependency for a cooldown like 30 seconds), and half-open (after the cooldown, allow a few probe requests; success closes the circuit, failure reopens it). The point is twofold: the caller stops burning threads and latency on a dead dependency, and the dependency gets breathing room to recover instead of being hammered while down. Netflix's Hystrix popularized the pattern (since retired in favor of Resilience4j and adaptive concurrency limits, worth mentioning to show currency), and service meshes like Envoy implement it as outlier detection, ejecting bad hosts from the pool.

Every circuit breaker needs a fallback answer for what to return while open: a cached value, a default (empty recommendations row), a queued write to process later, or an explicit error to the user. The fallback is a product decision, not just an engineering one.

Bulkheads isolate resources so one failing dependency cannot exhaust shared capacity, named after ship compartments that contain flooding. The classic incident: service X calls dependencies A and B from one thread pool of 200; B starts timing out at 30 seconds, every thread piles up waiting on B, and now calls to perfectly healthy A fail too because no threads remain. Bulkheading gives each dependency its own pool or semaphore (say 10 concurrent calls max to B), so B's failure saturates only B's compartment. The same principle applies at every level: separate connection pools per downstream, separate instance groups per customer tier or workload class, cell-based architecture where customers are partitioned into independent cells so an incident hits one cell's customers, not everyone. AWS builds heavily on cells for exactly this blast-radius argument.

Graceful Degradation and Chaos Engineering

Graceful degradation is deciding in advance which parts of the product are load-bearing and which are shed first under stress. Netflix's canonical example: if the personalization service is down, serve a popularity-based generic row rather than an error, because playing video matters and perfect recommendations do not. Other standard degradations: serve stale cache when the database is unhealthy (an old homepage beats a 500), disable expensive features under load (search suggestions, real-time counters), switch to read-only mode during a primary failover, and load-shed the lowest-priority traffic first (drop batch and crawler traffic before user requests). This requires explicitly ranking functionality by criticality and wiring feature flags or kill switches so operators can shed load in seconds during an incident, not ship a change.

Chaos engineering verifies that all of the above actually works by injecting failures on purpose, in production, in a controlled way. Netflix's Chaos Monkey (2011) randomly terminated production instances during business hours, forcing every team to build instance-death tolerance as table stakes; the practice grew into terminating whole AZ and region dependencies (Chaos Kong) and a discipline of formal experiments: define steady state (business metric like stream starts per second), form a hypothesis (killing one Cassandra node does not move it), inject the failure with a small blast radius and an abort button, and compare. The finding that matters is always the surprise: the retry storm nobody predicted, the hard dependency that was supposed to be soft.

The cultural point to land in an interview: failover paths, fallbacks, and breakers that are never exercised are broken by default (untested backups famously do not restore). Regular game days, automated fault injection in CI or staging, and periodic real failovers of production databases are what turn a fault-tolerance diagram into actual fault tolerance.

Tradeoffs