Caching

Why caching matters

The core motivation is the latency gap between storage tiers. Reading from L1 cache takes about 1 nanosecond, main memory about 100 nanoseconds, an SSD around 100 microseconds, and a cross-region network round trip 100 milliseconds or more. A Redis GET served from RAM in the same datacenter typically returns in under 1 millisecond, while the equivalent SQL query with joins might take 10 to 50 milliseconds under load. Caching converts expensive repeated work into cheap lookups.

Caching also protects the database. If 90 percent of reads hit the cache, the database sees only 10 percent of traffic, which often means the difference between one Postgres primary and a fleet of read replicas. Real systems lean on this heavily: Facebook's memcache tier famously serves billions of requests per second so that MySQL only handles the misses.

The tradeoff is staleness. Any cache is a copy of data that can drift from the source of truth, so every caching discussion in an interview should address invalidation and acceptable staleness for the specific use case.

Caching strategies: cache-aside, read-through, write-through, write-back

Cache-aside (lazy loading) is the most common pattern. The application checks the cache first; on a miss it reads the database, then writes the value into the cache with a TTL. It is simple and only caches data that is actually requested, but the first request after expiry always pays the miss penalty, and application code owns the consistency logic.

Read-through moves the loading logic into the cache layer itself: the application always talks to the cache, and the cache fetches from the database on a miss. Write-through writes to the cache and the database synchronously on every write, keeping the cache fresh at the cost of higher write latency and caching data that may never be read.

Write-back (write-behind) acknowledges the write once it lands in the cache and flushes to the database asynchronously, often batched. This gives excellent write throughput and absorbs spikes, but risks data loss if the cache node dies before the flush. It suits metrics, counters, and like counts, where losing a few seconds of writes is tolerable, and is the same idea a database uses internally with its buffer pool.

Eviction and invalidation

Caches are smaller than the datasets they front, so something must be evicted. LRU (least recently used) evicts the entry idle the longest and works well when recent access predicts future access. LFU (least frequently used) keeps hot keys even if they were not touched in the last few seconds, which handles scan-heavy workloads better but is costlier to track. Redis implements approximated LRU and LFU by sampling keys rather than maintaining exact ordering, and also supports TTL-based expiry and random eviction.

Invalidation is the harder problem. TTLs bound staleness cheaply: a 60 second TTL means at most 60 seconds of stale reads. Explicit invalidation (delete the key on write) gives fresher data but must handle race conditions, for example a read that fetches an old value from the database and writes it to the cache just after an invalidation. A common mitigation is to delete rather than update the cache on writes, and rely on the next read to repopulate.

A subtle failure mode is the thundering herd (cache stampede): a hot key expires and thousands of concurrent requests all miss and hammer the database simultaneously. Mitigations include per-key locking so only one request recomputes while others wait, probabilistic early refresh before expiry, jittered TTLs so keys do not expire in unison, and serving slightly stale data while a background refresh runs.

Multi-tier caching and technology choices

Real systems cache at several layers. The browser caches static assets via Cache-Control headers, a CDN like CloudFront or Cloudflare caches at edge locations near users, the application tier caches in-process (a Guava or Caffeine map) and in a shared store like Redis, and the database caches pages in its buffer pool. Each layer trades freshness for latency: a CDN can serve an image in 20 milliseconds from an edge POP versus 200 milliseconds from origin across an ocean.

Redis versus Memcached is a classic comparison. Memcached is a simple, multi-threaded, in-memory key-value store that excels at raw string caching. Redis is single-threaded per core for command execution but offers rich data structures (sorted sets for leaderboards, hashes, streams), persistence via RDB snapshots and AOF logs, replication, Lua scripting, and clustering. Most teams today default to Redis for its versatility; Memcached still wins for simple, huge, multi-threaded object caches.

In-process caches are the fastest (no network hop, sub-microsecond) but each app instance holds its own copy, so a fleet of 100 servers has 100 potentially inconsistent caches and a cold cache after every deploy. A shared Redis tier adds roughly 0.5 to 1 millisecond per lookup but gives one consistent view. Many systems layer both: a small in-process cache with a very short TTL in front of Redis.

Tradeoffs