Load Balancing

Layer 4 vs Layer 7

A Layer 4 (transport-layer) load balancer routes based on IP address and TCP/UDP port. It does not inspect payloads, so it is extremely fast and cheap per connection, AWS Network Load Balancer operates at L4 and handles millions of requests per second with sub-millisecond added latency, preserving the client's source IP. L4 is the right choice for raw TCP or UDP workloads, very high throughput, and cases where you terminate TLS on the backend.

A Layer 7 (application-layer) load balancer terminates the connection, parses HTTP, and routes on content: path, host header, cookies, or query parameters. This enables path-based routing (/api to one service, /static to another), TLS termination, HTTP/2 and gRPC multiplexing, request rewriting, and WAF integration. AWS Application Load Balancer, Nginx, HAProxy, and Envoy all operate at L7. The cost is more CPU per request and slightly higher latency because the proxy fully processes each request.

In practice many architectures layer them: an L4 balancer for fast, resilient traffic distribution in front of a fleet of L7 proxies that do smart routing. Kubernetes commonly pairs a cloud L4 balancer with an Envoy or Nginx ingress at L7.

Balancing Algorithms

Round robin cycles through backends in order and is the default nearly everywhere. It is fair when requests are uniform and servers are identical, but a few slow requests can pile onto an unlucky server. Weighted round robin assigns proportionally more traffic to bigger machines, useful during canary deploys, for example sending 5 percent of traffic to a new version.

Least connections routes each new request to the backend with the fewest active connections, which naturally adapts when request durations vary wildly, common with APIs where one endpoint takes 10 ms and another 2 seconds. Least response time and least loaded variants use latency or reported load instead. Power-of-two-choices, used by Envoy and Nginx, picks two random backends and sends to the less loaded one, capturing most of the benefit of least-connections without global state.

Consistent hashing routes based on a hash of a key (client IP, user ID, cache key) so that the same key almost always lands on the same backend. This matters for cache locality: if each backend caches data for its users, hashing keeps hit rates high, and when a node is added or removed only about 1/N of keys move rather than nearly all of them. This is the same technique behind distributed caches and is worth knowing deeply for senior interviews.

Health Checks and Failure Handling

Load balancers continuously probe backends and stop sending traffic to unhealthy ones. Active checks hit an endpoint like /healthz every 5-30 seconds and mark a node down after, say, 3 consecutive failures and up again after 2 successes, thresholds that prevent flapping. Passive checks watch real traffic and eject backends that return errors or time out, Envoy calls this outlier detection.

Health checks should be shallow enough to be cheap but deep enough to be meaningful. A check that only confirms the process is alive misses a wedged database connection pool; a check that queries the database can cause cascading failures where a brief database blip marks every app server unhealthy at once. Many teams use a liveness check (is the process running) separately from a readiness check (can it serve traffic), which is exactly the Kubernetes model.

Also consider connection draining: when removing a backend for deploys, the balancer stops new connections but lets in-flight requests finish, typically with a 30-300 second drain timeout. Without it, every deploy causes a burst of user-visible errors.

Global vs Local Load Balancing

Local load balancing distributes traffic among servers within one data center or region. Global load balancing (GSLB) distributes users across regions, usually via DNS (Route 53 latency-based or geolocation routing) or anycast IPs (Cloudflare and Google's front ends advertise the same IP from hundreds of locations, and BGP routes each user to the nearest one).

Global balancing serves three goals: latency (send a user in Frankfurt to eu-central rather than us-east, saving roughly 90 ms of round trip), disaster recovery (fail an entire region out by changing routing), and data sovereignty or capacity placement. DNS-based approaches are simple but blunted by TTL caching and resolvers that ignore TTLs; anycast fails over in seconds because BGP reconverges without waiting for client caches.

A complete picture for an interview: GeoDNS or anycast picks the region, an L4 balancer distributes across L7 proxies in that region, and the L7 layer routes to service instances. Being able to sketch that three-tier funnel quickly is a strong senior signal.

Tradeoffs