Performance Metrics

Latency vs Throughput

Latency is how long one operation takes, measured in milliseconds; throughput is how many operations complete per unit time, measured in requests per second or MB/s. They are related but not interchangeable, and the classic illustration is a truck full of hard drives driving across the country: enormous throughput, terrible latency. AWS Snowball is literally this, petabyte-scale transfer with days of latency.

Optimizing one can hurt the other. Batching improves throughput (amortize per-request overhead across 100 items) while raising the latency of the first item in the batch; Kafka producers exploit exactly this with linger.ms. Conversely, minimizing latency (send each item immediately) sacrifices throughput. Pipelining, parallelism, and asynchrony raise throughput without improving, and sometimes while worsening, single-request latency.

The third variable is concurrency, tied together by Little's Law: concurrency = throughput x latency. A service handling 1,000 RPS at 50 ms average latency has 50 requests in flight, sized comfortably by a small connection pool; the same throughput at 2 s latency means 2,000 in flight and exhausted thread pools. This is also why latency degrades under load: as utilization approaches saturation, queueing theory takes over and wait times grow non-linearly, latency at 90 percent utilization is dramatically worse than at 70 percent, which is why services are capacity-planned to run around 50-70 percent.

Percentiles: p50, p95, p99

Averages lie about latency because latency distributions are heavily right-skewed: many fast requests and a long tail of slow ones. A service can average 40 ms while 1 percent of requests take 2 seconds; the mean hides the 2-second experiences entirely. Percentiles expose the distribution: p50 (median) is the typical experience, p95 and p99 characterize the tail, and p99.9 matters at large scale. SLOs are therefore written against percentiles, for example p99 latency under 300 ms, never against means.

The tail matters more than its percentage suggests. First, heavy users make many requests: a user issuing 100 requests in a session has a 63 percent chance of hitting at least one p99-tail request. Second, tail amplification through fan-out: if one page load calls 100 backend services and each has a 1 percent chance of being slow, the page is slow 63 percent of the time, the page's latency is governed by the slowest of its fan-out calls. This is the central argument of Google's Tail at Scale paper, and the mitigations are worth knowing: hedged requests (send a duplicate to a second replica after the first exceeds the p95 mark and take whichever answers first), tight timeouts with retries against different replicas, and cutting fan-out.

Two practical notes: percentiles cannot be averaged across hosts or windows, aggregating requires histograms (the reason Prometheus uses histogram buckets), and always state which percentile you mean; p50 of 20 ms with p99 of 800 ms and p50 of 60 ms with p99 of 90 ms are very different services, and the second is often the better one.

Availability and the Nines

Availability is the fraction of time (or of requests) a service works, expressed in nines. The downtime math to memorize per year: 99 percent (two nines) is 3.65 days; 99.9 percent is 8.76 hours; 99.99 percent is 52.6 minutes; 99.999 percent (five nines) is 5.26 minutes. Per 30-day month, 99.9 is about 43 minutes and 99.99 about 4.3 minutes, tighter than most teams' incident response time, which is the honest reason few services truly deliver four nines: at that level, recovery must be automatic because a human cannot even be paged and oriented in 4 minutes.

Composition rules drive architecture. Serial dependencies multiply: a request touching five 99.9 percent services is at best 99.5 percent available, dependencies drag you down. Redundant parallel paths multiply failure probabilities instead: two independent 99 percent instances where either suffices give 99.99 percent. This one calculation is the mathematical core of why we deploy replicas, multi-AZ databases, and multi-region failover, and also why reducing hard dependencies (graceful degradation, serving cached data when a dependency is down) directly buys availability.

Each additional nine costs disproportionately more, roughly an order of magnitude in engineering and infrastructure, so the right target is a product decision: an internal batch tool is fine at 99.5, a payments API is not. Also distinguish availability from durability: S3 offers 99.99 percent availability but eleven nines of durability, meaning it may occasionally be unreachable but essentially never loses your data.

SLA, SLO, and SLI

The three terms form a hierarchy. An SLI (indicator) is the measurement itself: the fraction of requests returning success in under 300 ms, measured at the load balancer. An SLO (objective) is the internal target on that indicator: 99.9 percent of requests succeed within 300 ms over a rolling 30 days. An SLA (agreement) is the external contract with customers, with financial penalties: AWS EC2 credits 10 percent of the bill below 99.99 percent monthly uptime and 30 percent below 99.0. SLAs are deliberately looser than SLOs, you want to breach your internal target well before you owe customers money.

The operationally powerful concept is the error budget, popularized by Google SRE: a 99.9 percent SLO means 0.1 percent of requests may fail, about 43 minutes per month. That budget is spent on incidents, risky deploys, and experiments; while budget remains, teams ship fast, and when it is exhausted, feature work yields to reliability work. This converts the eternal velocity-versus-stability argument into a number both sides accept, and it explicitly rejects chasing 100 percent, which is unattainable and wastes the last increment of effort on diminishing returns.

Good SLIs measure user experience, not machine vitals: success rate and latency percentiles at the edge, not CPU utilization. Define them precisely (measured where, over what window, excluding what) because every ambiguity becomes an argument during an incident review.

Back-of-Envelope Numbers

Interviewers expect fluency with the latency ladder, descended from Jeff Dean's numbers every engineer should know: L1 cache about 1 ns; main memory reference about 100 ns; reading 1 MB sequentially from RAM about 10 microseconds; SSD random read about 100 microseconds; reading 1 MB from SSD about 1 ms; disk seek about 10 ms; same-datacenter round trip about 0.5 ms; same-region cloud RTT 1-2 ms; cross-continent (US East to Europe) about 80 ms; US to Asia 150-250 ms. The takeaways encoded in the ladder: memory is about 1,000x faster than SSD, SSD about 100x faster than disk for random access, an in-region network hop is cheaper than an SSD read, and crossing an ocean costs more than almost anything your code does.

Capacity math starts with time constants: a day is 86,400 seconds, call it 10^5 for estimation. Ten million DAU making 10 requests each is 10^8 requests per day, about 1,200 RPS average, and peak is typically 2-5x average, say 5,000 RPS. Storage: 10^8 tweets per day at 500 bytes is 50 GB per day of text, about 18 TB per year, trivially small, but if 10 percent attach a 1 MB image, that is 10 TB per day, and suddenly the design is about blob storage and CDNs, not the database. Ballpark single-node throughputs for sanity checks: a tuned Postgres does thousands to low tens of thousands of transactions per second, Redis about 100k ops per second per node, Kafka hundreds of MB per second per broker, a stateless app server 1,000-10,000 RPS.

The purpose is decision-making, not precision: round aggressively to powers of ten, state assumptions out loud, and use the result to pick an architecture, 1,200 RPS average means a modest service where a single primary database with replicas is plausible; 500k RPS means sharding, heavy caching, and CDN offload are mandatory. An answer within 3x that drives the right design beats a precise answer that drives nothing.

Tradeoffs