Hard

Design a News Feed (Twitter/Facebook)

Design the system that lets users post content and see a ranked, near-real-time feed of posts from people they follow. The defining challenge is the fan-out problem: how a single post reaches millions of followers' feeds efficiently, and how to blend fan-out on write with fan-out on read to handle celebrity accounts without melting the infrastructure.

1Requirements

Functional

• Users can publish posts (text up to 280 chars, images, video references).
• Users can follow and unfollow other users.
• Users see a feed of recent posts from accounts they follow, in reverse-chronological or ranked order.
• Feed supports infinite scroll with cursor-based pagination.
• Users can like and reply to posts, with counts visible in the feed.
• New posts from followees appear in the feed within seconds (near real time).

Non-functional

• Feed read latency under 200 ms at p99, since feed load is the app's front door.
• High availability (99.99%); a stale feed is acceptable, an error page is not.
• Eventual consistency is acceptable: a post may take a few seconds to reach all followers.
• Scale: hundreds of millions of DAU, with follower counts ranging from 10 to 100M+.
• The write path must absorb extreme skew: one celebrity post triggers work proportional to follower count.

2Back-of-envelope estimation

Feed read QPS	200M DAU x 10 feed loads/day = 2B reads/day ≈ 23K QPS, peak 2x ≈ 46K QPS
Post write QPS	200M DAU x 0.5 posts/day = 100M posts/day ≈ 1,200 QPS, peak 5x ≈ 6K QPS	Reads outnumber post writes ~20:1, before fan-out amplification
Fan-out amplification	Avg 200 followers x 100M posts/day = 20B feed-cache inserts/day ≈ 230K writes/sec	This is why fan-out is the core design problem
Celebrity worst case	1 post x 100M followers = 100M inserts; at 100K inserts/sec that is ~17 minutes of lag	Motivates the hybrid push/pull approach
Feed cache memory	200M users x 300 post IDs x 30 B ≈ 1.8 TB	Sharded Redis storing only IDs; post bodies hydrated from a separate post cache

3API design

POST /api/posts

Create a post. Body: { text, mediaIds? }. Returns the post object. Triggers async fan-out.

GET /api/feed?cursor={cursor}&limit=20

Fetch the viewer's home feed page. Cursor encodes (rank_score or timestamp, post_id) of the last item, so pagination is stable as new posts arrive.

POST /api/users/{userId}/follow

Follow a user. DELETE on the same path unfollows. Updates the social graph service.

POST /api/posts/{postId}/like

Like a post; counts are aggregated asynchronously and cached.

4High-level design

Clients talk to a gateway that routes to three main services: a post service (create/read posts), a social graph service (follow relationships), and a feed service (assemble the home timeline). Posts are written to a sharded post store and to a post cache, then the post ID is dropped onto a message queue for asynchronous fan-out, so the publish call returns quickly regardless of follower count.

Fan-out workers consume the queue. For a normal user's post, they query the graph service for follower IDs and push the post ID into each follower's feed cache, a Redis sorted set or list per user holding the most recent few hundred post IDs. This is fan-out on write (push): feeds are precomputed, so reading a feed is a single cache fetch, which is what makes 46K read QPS cheap.

For celebrity accounts above a follower threshold (say 100K), workers skip the push entirely. Instead, at read time the feed service pulls: it fetches the viewer's precomputed feed from cache, fetches recent posts from the short list of celebrities the viewer follows, merges the two streams by time or rank, and returns the page. This hybrid keeps write amplification bounded while keeping reads to a handful of cache lookups.

Feed responses hydrate post IDs into full content via the post cache, batch-fetch author profiles and like counts, then apply ranking (a lightweight ML scoring pass over the candidate set) before returning. Everything on the read path is cache-first; the databases (posts store, graph store) are the source of truth and backfill caches on miss.

Storage: posts in a sharded store keyed by post ID (Cassandra or sharded MySQL, partitioned by user_id so an author's posts colocate), the social graph in a store optimized for both follower and followee lookups (two adjacency tables or a graph store), and counters in a separate aggregated counters service fed by events.

5Data model

posts

post_id BIGINT PK (snowflake, time-sortable), author_id BIGINT, text VARCHAR(500), media_refs JSON, created_at TIMESTAMP

Sharded by author_id; snowflake IDs give free chronological ordering

follows

follower_id BIGINT, followee_id BIGINT, created_at TIMESTAMP, PK (follower_id, followee_id)

Second index or mirrored table keyed by followee_id for fan-out lookups

feed_cache (Redis)

key feed:{user_id}, sorted set of (post_id, score=timestamp or rank), trimmed to ~300 entries

Not durable; rebuilt on miss by pulling from followees' recent posts

post_counters

post_id BIGINT PK, like_count BIGINT, reply_count BIGINT, updated_at TIMESTAMP

Updated by async aggregation, never by synchronous increment on the read path

6Deep dives

Fan-out on write vs fan-out on read

Fan-out on write (push) precomputes every user's feed at post time: when Alice posts, insert her post ID into each of her followers' feed caches. Reads become O(1) cache fetches, latency is excellent, and it fits the read-heavy ratio. The costs: write amplification proportional to follower count, wasted work for inactive users whose feeds are computed but never read, and hot-spot writes when a big account posts.

Fan-out on read (pull) computes the feed at request time: fetch the viewer's followee list, fetch each followee's recent posts, merge and rank. Writes are O(1) and no work is wasted on inactive users, but every feed load costs hundreds of reads plus a merge, which at 46K QPS is untenable for latency and load.

No large feed system uses either extreme. The interview answer is: push by default because reads dominate, pull for the exceptional cases (celebrities, inactive users, cold caches). Also skip pushing to users inactive for, say, 30 days; rebuild their feed on demand via pull when they return.

The celebrity (hot user) problem

A user with 100M followers breaks pure push: a single post triggers 100M cache inserts, which takes minutes even at 100K inserts/sec, floods the queue ahead of normal users' posts, and briefly doubles global write load. Meanwhile followers see wildly different delivery times, and a burst of celebrity activity (a live event) can back the pipeline up for everyone.

The hybrid fix: mark accounts above a follower threshold as hot. Their posts are written to the post store and a hot posts cache only, with no fan-out. At read time, the feed service merges the viewer's pushed feed with a pull of recent posts from the (small) set of hot accounts the viewer follows. Since almost everyone follows only a handful of hot accounts, this pull adds only a few cache reads per feed load.

Edge cases worth mentioning: the threshold should have hysteresis so accounts crossing it do not flip-flop between modes; when an account transitions to hot you can simply stop pushing (old entries age out of the 300-entry feeds naturally); and the hot posts cache must itself be replicated because every feed read in the system may touch it.

Feed ranking and pagination

Reverse-chronological feeds are simple but modern feeds rank by predicted engagement. Architecturally, ranking is a second stage on the read path: gather a candidate set (the ~300 cached IDs plus pulled celebrity posts), hydrate lightweight features (author affinity, recency, engagement counts), score with a fast model under a strict latency budget (~50 ms), and return the top N. Keep the scorer stateless and feature fetches batched so the p99 stays inside 200 ms.

Pagination must be cursor-based, not offset-based. Offsets break when new posts prepend to the feed between page fetches, causing duplicates or gaps. A cursor encoding (score, post_id) of the last returned item lets the next page resume deterministically from that point in the sorted set.

Consistency expectations should be stated: it is fine if a follower sees a post 5 seconds late (eventual consistency via async fan-out), but a user must always immediately see their own post, so the client inserts it optimistically or the feed service unions the viewer's own recent posts into the response (read-your-writes).

Keeping counters and the graph fast

Like counts on viral posts receive tens of thousands of increments per second; doing a synchronous DB increment per like would serialize on one row. Instead, likes are events on a queue; aggregation workers batch increments (or use a sharded counter split across N rows summed on read) and publish totals into cache. Counts shown in feeds are seconds stale, which nobody notices.

The follow graph needs both directions: who does X follow (feed pull, profile) and who follows X (fan-out). Store both adjacency lists, sharded by the key you query on. Follower lists for hot accounts are huge, so fan-out workers stream them in chunks rather than loading 100M IDs into memory, and the graph service exposes a paginated followers iterator for exactly this purpose.

7Rapid implementation: build the MVP

Theory is table stakes. Here is how you would stand up a working version fast, on a budget, with the core algorithm in real code.

Stack

Next.js + Postgres + Redis on one $12 VPS; BullMQ (Redis-backed) as the fan-out queue, no Kafka needed for an MVP

01Create tables: posts (id BIGSERIAL, author_id, text, created_at) and follows (follower_id, followee_id, PK on the pair, plus an index on followee_id).
02Add a celebrity flag: is_celebrity boolean on users, set true above 10K followers by a nightly job (pick a low threshold so you can demo the hybrid path).
03Build POST /api/posts: insert the post, then enqueue a BullMQ fanout job with { postId, authorId } and return immediately.
04Write the fan-out worker: skip if the author is a celebrity, else page through follower ids 1000 at a time and LPUSH the post id into feed:{followerId}, LTRIM to 300.
05Build GET /api/feed: LRANGE the viewer's feed:{userId} list, pull recent post ids from celebrities the viewer follows via one SQL query, merge by created_at.
06Hydrate the merged id list with one SELECT ... WHERE id = ANY($1) and return posts sorted desc with a (created_at, id) cursor for pagination.
07Union the viewer's own posts from the last minute into the response so users always see their own post instantly (read-your-writes).
08Seed 10K fake users and 100K follows with a script, then verify a celebrity post appears in feeds without any fan-out writes.

Fan-out-on-write worker (skips celebrities)

typescript

import { Worker } from "bullmq";

new Worker("fanout", async (job) => {
  const { postId, authorId } = job.data;
  const author = await sql("SELECT is_celebrity FROM users WHERE id = $1", [authorId]);
  if (author[0].is_celebrity) return; // pulled at read time instead

  let cursor = 0;
  for (;;) {
    const followers = await sql(
      "SELECT follower_id FROM follows WHERE followee_id = $1 AND follower_id > $2 ORDER BY follower_id LIMIT 1000",
      [authorId, cursor]
    );
    if (followers.length === 0) break;
    const pipe = redis.pipeline();
    for (const f of followers) {
      pipe.lpush("feed:" + f.follower_id, String(postId));
      pipe.ltrim("feed:" + f.follower_id, 0, 299);
    }
    await pipe.exec();
    cursor = followers[followers.length - 1].follower_id;
  }
}, { connection: redis });

Hybrid feed read: cached push feed merged with celebrity pull

typescript

async function getFeed(userId: number, limit = 20) {
  // 1. precomputed feed from fan-out on write
  const pushedIds = (await redis.lrange("feed:" + userId, 0, 299)).map(Number);

  // 2. pull recent posts from celebrities this user follows
  const pulled = await sql(
    "SELECT p.id FROM posts p " +
    "JOIN follows f ON f.followee_id = p.author_id " +
    "JOIN users u ON u.id = p.author_id " +
    "WHERE f.follower_id = $1 AND u.is_celebrity " +
    "AND p.created_at > now() - interval '48 hours' " +
    "ORDER BY p.id DESC LIMIT 100",
    [userId]
  );

  // 3. merge, dedup, hydrate (snowflake-style ids sort by time)
  const ids = [...new Set([...pushedIds, ...pulled.map((r: any) => Number(r.id))])]
    .sort((a, b) => b - a)
    .slice(0, limit);
  const posts = await sql(
    "SELECT id, author_id, text, created_at FROM posts WHERE id = ANY($1)",
    [ids]
  );
  return ids.map((id) => posts.find((p: any) => Number(p.id) === id));
}

Bottlenecks & failure modes

⚠Celebrity fan-out floods the write pipeline; solve with the hybrid push/pull split at a follower threshold.
⚠Feed cache (1.8 TB) must be sharded; a resharding event or shard loss forces expensive feed rebuilds, so plan consistent hashing and replicas.
⚠Hot posts cache is read by nearly every feed request during a viral moment; replicate it and add per-server local caching.
⚠Synchronous counter updates on viral posts serialize on single rows; batch through queues or shard the counters.
⚠Fan-out queue backlog delays delivery for everyone; isolate queues by author tier so a hot account cannot starve normal traffic.

Key takeaways

▸The feed problem is the fan-out problem: push precomputes reads, pull avoids write amplification, and real systems blend both.
▸Set the threshold explicitly in interviews (e.g., push under 100K followers, pull above) and explain the read-time merge.
▸Feeds are eventually consistent except read-your-writes: users must see their own posts immediately.
▸Store IDs in feed caches and hydrate content separately; it keeps caches small and post edits consistent.
▸Cursor pagination and async counter aggregation are small details that separate senior answers from junior ones.

Brush up on the underlying topics

Caching Message Queues and Streaming Sharding and Partitioning Event-Driven Architecture Consistency and the CAP Theorem