Observability: Metrics, Logs, Traces

The Three Pillars and Their Cost Profiles

Metrics are numeric time series (request count, latency histogram, queue depth), aggregated at the source, cheap to store and query, and ideal for dashboards and alerts. Their cost scales with cardinality, not traffic: a counter labeled by endpoint and status code is nearly free at any request volume, but adding a user_id label with a million values multiplies the series count and is the classic way teams blow up their Prometheus. Metrics tell you that something is wrong and roughly where, but not why for a specific request.

Logs are discrete, timestamped events with arbitrary detail, the ground truth for individual occurrences. Modern practice is structured logging (JSON with consistent fields like request_id, user_id, latency_ms) so logs are queryable rather than grepped prose. Cost scales linearly with traffic, which is why log volume at scale forces sampling or tiered retention (hot searchable for 7 to 30 days, cold object storage after), and why a service logging 1KB per request at 10,000 RPS produces roughly 850GB per day before replication.

Traces follow one request across service boundaries: a trace is a tree of spans, each span one operation with start time, duration, and attributes, glued together by a trace ID propagated in headers (W3C traceparent). Google's Dapper paper (2010) established the model; Zipkin, Jaeger, and now OpenTelemetry, the CNCF standard that unifies instrumentation APIs and wire formats for all three pillars, descend from it. Traces answer where did these 3 seconds go across 12 services, the question neither metrics nor logs can. Because full tracing is expensive, systems sample: head-based sampling (decide at the front, say 1 percent) is cheap but misses rare errors; tail-based sampling (buffer, then keep the slow and failed traces) keeps the interesting ones at the cost of buffering infrastructure. The three pillars converge in practice: a metric alert fires, you pivot to exemplar traces from the bad window, then to the logs of the failing span, all joined by trace and request IDs.

RED and USE: Knowing What to Measure

The RED method, articulated by Tom Wilkie, defines the three signals for every request-driven service: Rate (requests per second), Errors (failed requests per second), and Duration (latency distribution, not averages). Every service dashboard should lead with these three, uniformly, so an on-call engineer can walk down the architecture during an incident reading identical panels for every service. RED is essentially the user's view of a service: how often it is asked, how often it lies, how long it takes.

The USE method, from Brendan Gregg, covers resources rather than services: for every resource (CPU, memory, disk I/O, network, connection pools, thread pools), check Utilization (fraction of time busy), Saturation (queued work that cannot be served yet, like run-queue length or a full connection pool), and Errors. Saturation is the leading indicator; a disk at 90 percent utilization with no queue is fine, while a growing queue means latency is about to explode. USE catches the causes (a saturated pool) whose symptoms RED displays (rising duration).

Latency must be handled as distributions and percentiles: means lie. If p50 is 20ms and p99 is 2 seconds, one customer in a hundred has a terrible experience, and at 100 requests per page load, most page loads contain a p99 request. Track p50, p95, p99 from histograms, and never average percentiles across hosts (aggregate the histograms instead). Google's SRE book frames the same territory as the four golden signals: latency, traffic, errors, saturation, which is RED plus saturation, and citing both shows range.

Alerting That Pages on Pain, Not Noise

The cardinal rule is to alert on symptoms (user-visible pain: error rate, latency, correctness) and not on causes (CPU is high, a host is down). Cause-based alerts generate pages for conditions users never notice (one dead instance behind a load balancer is Tuesday), and each false page erodes on-call trust until real pages get ignored, the alert fatigue that post-mortems repeatedly identify as a contributing factor. Causes belong on dashboards for diagnosis and in tickets for follow-up, not on pagers at 3am.

Every page must be actionable, urgent, and novel: a human must need to do something now that automation cannot. Anything else is a ticket or a dashboard. Good hygiene includes runbook links in every alert, multi-window checks to suppress flapping, and severity tiers where only the top tier pages.

The modern refinement is SLO-based alerting on burn rate: instead of paging when error rate exceeds a static 1 percent for 5 minutes, page when the error budget is being consumed too fast. A burn rate of 1 means you will spend exactly your monthly budget in a month; the SRE Workbook's standard policy is to page at 14.4x burn over 1 hour (consuming 2 percent of the monthly budget in an hour) and 6x over 6 hours, and only ticket slower burns. Multi-window multi-burn-rate alerts catch both fast outages and slow bleeds while staying quiet for noise that will not threaten the SLO.

SLOs and Error Budgets

An SLI is a measured indicator (the fraction of requests under 300ms that returned non-5xx, measured at the load balancer); an SLO is the target on it (99.9 percent over a rolling 30 days); an SLA is the external contract with financial penalties, always looser than the internal SLO. Choosing the SLI carefully matters more than the number: measure as close to the user as possible, define what counts as good explicitly, and exclude only what you can defend.

The error budget is the SLO's complement made spendable: 99.9 percent over 30 days allows 43.2 minutes of full downtime, or 0.1 percent of requests failing continuously. This reframes reliability from a virtue into a resource. Budget remaining means teams ship fast, run chaos experiments, and take risks; budget exhausted triggers the agreed policy, classically a feature freeze with engineering redirected to reliability until the budget recovers. The genius of the mechanism, as Google's SRE book presents it, is political: it replaces the eternal dev-versus-ops argument about whether the system is reliable enough with a number both sides agreed to in advance, and it makes 100 percent explicitly the wrong target, since each added nine costs roughly 10x and users on flaky wifi cannot tell 99.99 from 99.999.

Practical failure modes worth naming: SLOs set aspirationally rather than from measured baselines (instant permanent violation, policy ignored), too many SLOs (nobody can attend to 40 of them; pick 2 or 3 user journeys), and error budget policies without teeth (a freeze that leadership overrides the first time it binds is theater). A senior candidate ties the loop together: SLIs feed SLOs, SLOs define budgets, budgets drive burn-rate alerting and the ship-versus-stabilize decision.

Tradeoffs