Storage and Search

Block, file, and object storage

Block storage exposes raw fixed-size blocks, like a virtual disk: AWS EBS, or a SAN. It offers the lowest latency (sub-millisecond) and supports in-place random writes, which is why databases run on it, but a volume attaches to essentially one server and capacity is provisioned, not elastic. File storage (NFS, AWS EFS) adds a POSIX hierarchy shared across many clients, convenient for legacy apps and shared workspaces, but metadata operations make it hard to scale to extreme sizes.

Object storage (Amazon S3, Google Cloud Storage, Azure Blob) stores immutable blobs by key in a flat namespace, accessed over HTTP. You cannot edit a byte in place; you replace the whole object. In exchange you get practically unlimited capacity, 11 nines of durability (S3 stores redundantly across at least 3 availability zones, using replication and erasure coding), and pennies per GB-month, with tiers from S3 Standard down to Glacier Deep Archive at roughly one twentieth the cost for archival data.

The standard interview pattern: metadata in a database, bytes in object storage. A photo service stores the image in S3 under a key, and a Postgres row holds the key, owner, dimensions, and permissions. Uploads and downloads should use presigned URLs so clients transfer directly with S3 and your servers never proxy the bytes, and a CDN in front of the bucket serves hot objects from edge locations.

Inverted indexes: how search works

A database index finds rows by exact key or range; it cannot efficiently answer 'documents containing the words cheap AND flights'. The inverted index solves this by mapping each term to the sorted list of document IDs containing it (a postings list), like a book's index at web scale. Querying intersects or unions postings lists: the AND of two terms is a merge of two sorted lists, which is fast even over millions of documents.

Building the index requires text analysis: tokenize the text, lowercase it, drop stop words, and apply stemming so 'running' matches 'run', plus optional synonym expansion. The same analysis must apply to queries. Postings can also store term positions (for phrase queries like 'new york'), and frequencies for ranking.

Ranking is what separates search from lookup. Classic scoring is TF-IDF, refined into BM25 (the default in Lucene, Elasticsearch, and OpenSearch): a document scores higher when the query term appears often in it, the term is rare across the corpus, and the document is short. Modern stacks add a second stage, retrieving the top few hundred candidates with BM25 and re-ranking with machine-learned models, and increasingly hybrid search that combines keyword retrieval with vector similarity from embeddings.

Elasticsearch and search at scale

Elasticsearch (and OpenSearch, its fork) packages Lucene, the inverted-index library, into a distributed system. An index is split into shards, each a full Lucene index; shards have replicas for availability and read throughput. A query fans out to one copy of every shard, each shard returns its top K candidates, and a coordinating node merges them into the global top K, classic scatter-gather. This is why shard count matters: 1,000 shards means every query does 1,000 sub-queries, and massive over-sharding is the most common Elasticsearch operational mistake; a common guideline keeps individual shards between 10 and 50 GB.

Writes in Lucene follow an LSM-like pattern: documents buffer in memory and are written as immutable segments, which background merges consolidate. A document only becomes searchable after a refresh, which defaults to every 1 second, so Elasticsearch is near-real-time, not real-time, and it should be treated as eventually consistent search over your data, not as a primary store.

The canonical architecture keeps the source of truth in a database and syncs to the search cluster asynchronously, usually via change data capture (Debezium reading the database's replication log into Kafka, consumed by an indexer) or dual-writing through a queue. That pipeline introduces indexing lag, typically seconds, and requires idempotent indexing with versioning so replays and reordering do not corrupt documents. Handling deletes and mapping changes (which often force a full reindex into a new index behind an alias) are the operational realities worth mentioning.

Designing full-text search into a system

When an interviewer adds 'users can search products' to a design, resist reaching for Elasticsearch first. Postgres full-text search with a GIN index on a tsvector handles surprising scale, millions of rows with tens of milliseconds queries, with zero extra infrastructure, and SQLite FTS5 or MySQL FULLTEXT cover smaller cases. Reach for a dedicated engine when you need heavy relevance tuning, typo-tolerant autocomplete, faceted navigation, multi-language analysis, or query volume that would harm the primary database.

Size the problem out loud. A product catalog of 10 million items averaging 1 KB of searchable text is about 10 GB, an index that fits in one or two shards on a single node with a replica; a log-search cluster ingesting 1 TB per day is a completely different design with time-based indices, hot-warm-cold tiers on progressively cheaper hardware, and ILM policies that delete or archive old indices. Autocomplete is its own subproblem, usually served by edge n-gram indexes or an in-memory prefix trie rather than full queries.

Also know the adjacent options: Algolia and Typesense as managed low-latency search focused on instant results, vector databases and Lucene's HNSW support for semantic search over embeddings, and the pattern of caching frequent query results (search queries follow a power law, so a small cache absorbs a large share of traffic).

Tradeoffs