Scalability
Scalability is a system's ability to handle growing load by adding resources, either by making individual machines bigger (vertical) or by adding more machines (horizontal). Nearly every system design interview hinges on how you scale past a single box.
Vertical vs Horizontal Scaling
Vertical scaling (scaling up) means adding more CPU, RAM, or faster disks to a single machine. It is the simplest path: no code changes, no distributed-systems complexity, and strong consistency comes for free because there is one node. Modern cloud instances go remarkably far, an AWS u-24tb1.metal offers 24 TB of RAM, and a single well-tuned Postgres box can serve tens of thousands of queries per second. Many companies run profitably on one large primary database for years.
The limits are hard, though. There is a ceiling on how big one machine can get, price grows super-linearly (a machine with 2x the specs often costs 3-4x), and a single machine is a single point of failure. Upgrades usually require downtime or a failover.
Horizontal scaling (scaling out) adds more commodity machines behind a load balancer. Capacity grows roughly linearly with node count, failures of individual nodes are survivable, and you can scale incrementally. The cost is architectural: you now need load balancing, service discovery, and a strategy for data that no longer fits on one node (sharding, replication). Google, Amazon, and Netflix all built their platforms on the assumption that any individual commodity server can and will die.
Stateless Services
The key enabler of horizontal scaling is statelessness: any request can be served by any instance because instances hold no client-specific state between requests. Session data moves out of process memory into a shared store such as Redis or Memcached, or into a signed token like a JWT that the client carries with each request.
With stateless app servers, the load balancer can spray traffic freely, autoscalers can add or remove instances at will, and deploys become trivial rolling replacements. Contrast this with sticky sessions, where a user is pinned to one server: if that server dies, the session is lost, and hot users create hot servers.
State does not disappear, it gets pushed to the edges of the architecture: databases, caches, object stores like S3, and message queues. A common interview framing is that the stateless tier scales easily and the stateful tier (the database) is where scaling gets hard, which is why so much of system design is really about scaling data.
Elasticity and Autoscaling
Elasticity is the ability to add and remove capacity automatically in response to demand. AWS Auto Scaling Groups, Kubernetes Horizontal Pod Autoscaler, and GCP managed instance groups watch signals like CPU utilization, request count per target, or queue depth and adjust instance counts. A typical policy might target 60 percent average CPU, scaling out fast and scaling in slowly to avoid flapping.
Elasticity matters because real traffic is bursty. A retail site might see 10x normal load on Black Friday, and a news site can spike 50x in minutes. Provisioning statically for peak wastes money the other 99 percent of the time; provisioning for average means falling over at peak. Autoscaling has lag, though, booting a VM can take 1-3 minutes, so systems still need headroom, and sudden spikes are often absorbed first by caches, CDNs, and load shedding.
Mention the difference between predictive scaling (scale up before a scheduled event, like a product launch) and reactive scaling (respond to metrics). Mature systems use both.
Scaling Reads and Writes Differently
Most systems are read-heavy, ratios of 100:1 reads to writes are common for social or content products. Reads scale with caching (CDN, Redis, application-level caches) and read replicas, both of which are far cheaper than sharding. A single cache layer with a 90 percent hit rate cuts database read load by 10x.
Writes are harder. Replicas do not help write throughput because every write must reach the primary. Options include sharding (partition data across many primaries), write batching, and absorbing bursts with a message queue so the database consumes at a steady rate. Each adds complexity, which is why interviewers respect answers that delay sharding until simpler levers are exhausted.
A strong senior answer sequences the levers: optimize queries and indexes, add caching, add read replicas, scale vertically while it is cheap, then shard when write volume or data size truly demands it.
Key points
- ▸Vertical scaling is simpler but has a hard ceiling, super-linear cost, and a single point of failure; horizontal scaling is near-limitless but forces distributed-systems complexity.
- ▸Stateless services are the prerequisite for horizontal scaling; push state into Redis, databases, object storage, or client-held tokens.
- ▸Elasticity (autoscaling) matches capacity to bursty demand, but scaling lag means you still need headroom and caches for sudden spikes.
- ▸Scale reads with caching and replicas before touching writes; scale writes with sharding and queue-based buffering only when needed.
- ▸Sequence your levers in an interview: indexes and query tuning, caching, replicas, vertical scaling, then sharding last.
- ▸Design for failure at scale: with hundreds of commodity nodes, individual failures are routine, not exceptional.
Tradeoffs
Vertical scaling (scale up)
Pros
- + No application changes or distributed-systems complexity
- + Strong consistency is trivial on a single node
- + Fast to execute; often just an instance resize
Cons
- − Hard ceiling on maximum machine size
- − Cost grows super-linearly with specs
- − Single point of failure; upgrades often need downtime
Horizontal scaling (scale out)
Pros
- + Near-linear capacity growth with commodity hardware
- + Fault tolerance: losing one node is survivable
- + Enables elasticity and zero-downtime rolling deploys
Cons
- − Requires load balancing, service discovery, and stateless design
- − Data layer becomes hard: sharding, replication, consistency tradeoffs
- − Operational complexity and observability burden grow
Sticky sessions instead of externalized state
Pros
- + Simple to implement; in-memory session access is fast
- + No shared session store to operate
Cons
- − Server loss destroys sessions
- − Uneven load distribution and hot instances
- − Blocks clean autoscaling and rolling deploys
In the interview
- ★Do not jump straight to microservices and sharding; interviewers want to see you scale incrementally and justify each step with numbers.
- ★State the read:write ratio early, it determines whether caching and replicas solve the problem or whether you need to shard writes.
- ★Explicitly call out what is stateful in your design and where that state lives; it shows you understand why horizontal scaling works.
- ★Use concrete capacity math: for example, if one app server handles 1,000 RPS and you expect 50,000 RPS peak, you need roughly 50 instances plus headroom.