Hard

Design Live Streaming (Twitch)

A live streaming platform where creators broadcast video that millions watch with a few seconds of delay, alongside a real-time chat. The pipeline is RTMP ingest, transcoding into a bitrate ladder, HLS segment packaging and CDN delivery, and a chat system whose fan-out can dwarf the video problem. The defining tradeoff is glass-to-glass latency versus scale and cost.

1Requirements

Functional

  • • Creators broadcast from OBS or similar encoders via RTMP(S) using a stream key.
  • • Viewers watch live with adaptive bitrate: quality adjusts automatically to their bandwidth (1080p down to 240p).
  • • Playback starts within 2 seconds of pressing play, at a live edge a few seconds behind the broadcaster.
  • • Each stream has a chat room; messages appear to all viewers in near real time, with moderation (bans, slow mode).
  • • Viewers discover live channels via directory and get live viewer counts.
  • • Streams are optionally recorded for VOD replay.

Non-functional

  • • Glass-to-glass latency of 3-8 seconds (LL-HLS territory); consistency of latency matters more than the absolute number.
  • • Support 100K concurrent streams and 10M concurrent viewers, with a single event peaking at 3M viewers on one stream.
  • • Ingest must survive encoder hiccups: brief disconnects resume the same stream without killing viewer sessions.
  • • Chat delivers messages within 1 second at up to 100K messages/minute in one room.
  • • The origin must be shielded: viewer traffic is served ~99% from CDN edge caches.
  • • Transcoding cost scales with streams, not viewers; delivery cost scales with viewers, not streams.

2Back-of-envelope estimation

Ingest bandwidth~600 Gbps
Transcoding compute~100K GPU-accelerated jobs
Egress bandwidth~30 Tbps
Segment request rate (one 3M-viewer stream)~1.5M req/s at edges, ~single-digit req/s at origin
Chat fan-out (one 3M-viewer room)~50M deliveries/s uncapped

3API design

rtmp://ingest.example.com/live/{streamKey}

RTMP ingest endpoint; the stream key authenticates the broadcaster and maps to a channel.

GET /v1/channels/{channel}/master.m3u8

Master HLS playlist listing the bitrate ladder renditions; the player picks based on measured bandwidth.

GET /hls/{channel}/{rendition}/segment_{n}.ts

Media segments (2 s each), served from CDN edge; playlist and segments are the entire video read path.

WSS /v1/chat/{channel}

WebSocket for chat: send messages, receive the room firehose (possibly sampled), moderation events.

GET /v1/channels?category=...&sort=viewers

Directory of live channels with approximate concurrent viewer counts.

4High-level design

Broadcasters push RTMP to the nearest ingest PoP (anycast or GeoDNS). The ingest server validates the stream key, and relays the source stream to the transcoding tier. Ingest keeps a short reconnect grace window so a flapping encoder resumes the same session instead of ending the broadcast.

Transcoders (GPU-accelerated ffmpeg pipelines) decode the source once and encode a bitrate ladder: for example 1080p60 at 6 Mbps, 720p at 3 Mbps, 480p at 1.5 Mbps, 360p at 800 Kbps, 240p at 400 Kbps, all with aligned keyframes every 2 seconds so players can switch renditions at segment boundaries. Transcoding cost is per stream, so small channels can get a reduced ladder (or source-only passthrough) to save GPUs, while partner channels get the full ladder.

The packager cuts each rendition into 2-second HLS segments, writes them to origin storage, and appends them to a rolling media playlist per rendition. Viewers fetch the master playlist once, then poll the media playlist and download segments over plain HTTPS. Because segments are immutable, static files, the CDN caches them perfectly: request collapsing means even 3M viewers of one stream produce only a handful of origin fetches per segment. Playlists get a 1-second TTL; segments are cached until evicted.

Chat is architecturally separate: a fleet of WebSocket gateway servers, each holding tens of thousands of connections, subscribed to per-channel topics on a pub/sub backbone (Redis pub/sub or Kafka). A message goes sender -> gateway -> pub/sub -> every gateway with subscribers in that room -> local fan-out over WebSockets. Giant rooms need protection: rate limits per user, slow mode, and firehose sampling where each gateway forwards only a representative fraction of messages because no human can read 800 messages per second anyway.

Viewer counts are approximate by design: gateways and players heartbeat, counts are aggregated with a streaming counter (or HyperLogLog for uniques) and published every few seconds. VOD recording is a parallel consumer of the segment stream: segments are appended to long-term storage and stitched into a VOD playlist when the stream ends.

5Data model

channels

channel_id, user_id, stream_key_hash, title, category, is_live, current_session_id, viewer_count_estimate, updated_at

stream_key stored only as a hash; viewer_count_estimate refreshed every few seconds, explicitly approximate.

stream_sessions

session_id, channel_id, started_at, ended_at, ingest_pop, source_resolution, renditions_json, vod_playlist_key

One row per broadcast; survives encoder reconnects within the grace window.

segments (object store + playlist state)

session_id, rendition, seq_number, duration_ms, storage_key, created_at

Immutable; the media playlist is generated from the latest N rows per rendition.

chat_messages

message_id, channel_id, user_id, body, badges, created_at, deleted_by

Persisted asynchronously for moderation/VOD replay; live delivery path never waits on this write.

6Deep dives

The latency vs scale tradeoff, made explicit

Live video latency and delivery scalability pull in opposite directions, and the segment length is the knob. HLS latency is roughly 3-4 segment durations (the player buffers a few segments to absorb jitter). With classic 6-second segments you get 20-30 seconds of latency but supreme cacheability and stability. With 2-second segments you get 6-10 seconds. LL-HLS pushes further by splitting segments into sub-second parts delivered with HTTP chunked transfer, reaching 2-5 seconds at the cost of many more requests and touchier CDN behavior. WebRTC achieves sub-second but abandons HTTP caching entirely: every viewer needs a stateful media session, so cost scales linearly with viewers and 3M-viewer streams become an SFU cascade problem.

Twitch's actual position is instructive: a few seconds of delay is fine for most content because the interaction loop is chat, and chat round trips are 1-2 seconds anyway. Matching video latency to the interaction medium, rather than minimizing it absolutely, is the mature answer. Auctions and betting need WebRTC; game streaming does not.

A second-order point interviewers reward: consistent latency beats low latency. If viewers drift (pausing, buffering), chat reactions desynchronize from the video. Players should quietly speed up playback by 2-5% when behind the target live edge to converge, which is invisible to users.

Why HLS over CDN wins the delivery economics

The delivery insight that makes 30 Tbps affordable: convert live video into immutable static files. A 2-second segment of the 720p rendition is identical for all 3 million viewers, so the CDN edge caches it once and serves it millions of times. With request collapsing (the edge holds concurrent requests for an uncached object and issues one origin fetch), origin load is per-segment-per-edge, essentially independent of viewer count. The origin serves maybe hundreds of requests per second while edges serve millions.

The playlist is the only mutable object. It gets a 1-second TTL and is tiny, so even aggressive polling is cheap. A subtle failure mode: if the packager stalls and playlists stop advancing, millions of players poll an unchanging playlist and then all stampede for the next segment when it appears. Jittering player poll intervals and having edges serve slightly stale playlists during origin failures both blunt this.

Multi-CDN is standard at this scale: no single CDN wants a surprise 30 Tbps, and per-region performance varies. A steering layer picks the CDN per session based on cost and measured throughput, and the player can fail over mid-stream because segments are addressable identically on any CDN.

Chat fan-out: the hidden hard problem

The math is unforgiving: a modest 17 messages/second in a 3M-viewer room implies 50 million message deliveries per second if delivered naively. Video does not have this problem because segments are shared; chat messages are per-connection pushes. The architecture is two-tier fan-out: publish each message once to a per-channel topic, have only the gateway servers with viewers in that room subscribe, and let each gateway multicast to its local WebSocket connections from a single in-memory copy. Publishing cost is O(gateways in room), delivery cost is amortized socket writes.

Even so, giant rooms need semantic load shedding. Humans cannot read more than roughly 10-20 messages per second, so beyond that the firehose is sampled: each gateway forwards a fair random fraction, always including messages from moderators, the streamer, and the viewer's own messages (which are echoed locally so the sender always sees their message). Slow mode (one message per user per N seconds) caps the publish rate at the source. These product features are actually backpressure mechanisms.

Moderation must propagate faster than messages: a ban or message deletion publishes a control event on the same topic at higher priority, and gateways drop queued messages from banned users before flushing. Persisting chat is off the hot path: the delivery pipeline writes to Kafka, and a consumer batches into storage for VOD replay and moderation audit.

Ingest resilience and the transcoding tier

Ingest is the one stateful, non-cacheable part of the video path, so it gets the reliability attention. Broadcasters connect to the nearest PoP; the stream key maps to a channel and a transcoding assignment. If the encoder disconnects (home internet blip), the ingest holds the session open for a grace window of 30-90 seconds, and the packager inserts a slate or freezes the last frame, so the viewer-side playlist keeps advancing and players do not tear down. Reconnection resumes the same session id and segment numbering.

7Rapid implementation: build the MVP

Theory is table stakes. Here is how you would stand up a working version fast, on a budget, with the core algorithm in real code.

Stack

OBS (broadcaster) + nginx-rtmp or Node Media Server in Docker + ffmpeg for the ladder + hls.js in the browser + Redis pub/sub with a Node WebSocket server for chat; runs on one $10 VPS.

  1. 01Run nginx with the rtmp module in Docker, configured with an application block that exec-invokes ffmpeg on publish.
  2. 02Point OBS at rtmp://localhost/live with a stream key; verify the raw stream arrives (ffprobe).
  3. 03Write the ffmpeg command that produces two renditions (720p and 360p) as HLS with 2-second segments and aligned keyframes (-g 60 at 30 fps, -sc_threshold 0), emitting a master playlist.
  4. 04Serve the HLS output directory with proper cache headers: segments immutable for 1 hour, playlists max-age=1.
  5. 05Build the player page with hls.js pointed at master.m3u8; confirm adaptive switching by throttling in devtools.
  6. 06Build chat: a Node ws server where each connection subscribes to a Redis channel per room; publish on message, fan out to local sockets on pub/sub delivery.
  7. 07Add slow mode (per-user token bucket in Redis) and a viewer counter (heartbeat keys with TTL, count via SCARD every 5 s).
  8. 08Measure glass-to-glass latency: show a clock on the broadcaster screen, compare with the player; tune segment count in the playlist to trade startup time vs latency.

Transcoding ladder + HLS packaging (single ffmpeg)

python
import subprocess

def start_transcode(stream_key: str, out_dir: str):
    # Two renditions, keyframes aligned every 2 s (gop 60 at 30 fps),
    # 2 s segments, rolling window of 6 segments per playlist.
    cmd = [
        "ffmpeg", "-i", "rtmp://localhost/live/" + stream_key,
        "-filter_complex",
        "[0:v]split=2[v1][v2];"
        "[v1]scale=w=1280:h=720[v1out];"
        "[v2]scale=w=640:h=360[v2out]",
        "-map", "[v1out]", "-c:v:0", "libx264", "-b:v:0", "3000k",
        "-g", "60", "-keyint_min", "60", "-sc_threshold", "0",
        "-map", "[v2out]", "-c:v:1", "libx264", "-b:v:1", "800k",
        "-map", "a:0", "-map", "a:0", "-c:a", "aac", "-b:a", "128k",
        "-f", "hls", "-hls_time", "2", "-hls_list_size", "6",
        "-hls_flags", "delete_segments+independent_segments",
        "-master_pl_name", "master.m3u8",
        "-var_stream_map", "v:0,a:0,name:720p v:1,a:1,name:360p",
        out_dir + "/stream_%v.m3u8",
    ]
    return subprocess.Popen(cmd)

Chat fan-out with Redis pub/sub and WebSockets

typescript
import { WebSocketServer, WebSocket } from "ws";
import { Redis } from "ioredis";

const pub = new Redis();
const sub = new Redis();
const rooms = new Map<string, Set<WebSocket>>(); // channel -> local sockets

sub.on("message", (channel, raw) => {
  // One pub/sub delivery per gateway, then local multicast:
  // the message is serialized once and written to every socket.
  const sockets = rooms.get(channel);
  if (!sockets) return;
  for (const ws of sockets) {
    if (ws.readyState === WebSocket.OPEN) ws.send(raw);
  }
});

const wss = new WebSocketServer({ port: 8081 });
wss.on("connection", (ws, req) => {
  const channel = "chat:" + new URL(req.url ?? "/", "http://x").searchParams.get("room");
  if (!rooms.has(channel)) {
    rooms.set(channel, new Set());
    sub.subscribe(channel); // subscribe only while we have local viewers
  }
  rooms.get(channel)!.add(ws);

  ws.on("message", async (data) => {
    const msg = JSON.stringify({ body: String(data).slice(0, 500), at: Date.now() });
    await pub.publish(channel, msg); // publish once; all gateways fan out
  });

  ws.on("close", () => {
    const set = rooms.get(channel)!;
    set.delete(ws);
    if (set.size === 0) { rooms.delete(channel); sub.unsubscribe(channel); }
  });
});

Slow mode: per-user rate limit with Redis

typescript
import { Redis } from "ioredis";
const redis = new Redis();

// Returns true if the user may send; enforces one message per
// slowModeSeconds per room using SET NX with expiry, which is atomic.
export async function allowMessage(
  room: string,
  userId: string,
  slowModeSeconds: number
): Promise<boolean> {
  if (slowModeSeconds <= 0) return true;
  const key = "slow:" + room + ":" + userId;
  const ok = await redis.set(key, "1", "EX", slowModeSeconds, "NX");
  return ok === "OK";
}

// Viewer count: heartbeat every 15 s from each player.
export async function heartbeat(room: string, viewerId: string) {
  await redis.set("viewer:" + room + ":" + viewerId, "1", "EX", 30);
}

export async function viewerCount(room: string): Promise<number> {
  let cursor = "0", count = 0;
  do {
    const [next, keys] = await redis.scan(cursor, "MATCH", "viewer:" + room + ":*", "COUNT", 1000);
    cursor = next;
    count += keys.length;
  } while (cursor !== "0");
  return count;
}

Bottlenecks & failure modes

  • ⚠Origin stampedes when a playlist stalls and recovers: millions of players synchronize their next request; mitigated by request collapsing, poll jitter, and stale-while-revalidate on playlists.
  • ⚠Transcoding GPU pool exhaustion during peak hours: mitigated by reduced ladders for small channels and passthrough-only mode as a degraded tier.
  • ⚠Single mega-room chat fan-out saturating gateway CPUs on socket writes; requires per-gateway multicast from one buffer, sampling, and slow mode.
  • ⚠Ingest PoP failure mid-broadcast: needs encoder reconnect to a backup ingest URL and session resumption without changing the viewer-facing playlist.
  • ⚠Cross-CDN consistency: a segment present on one CDN but not yet on another breaks mid-session failover; solved by origin-pull (both CDNs pull from the same origin) rather than push.

Key takeaways

  • ▸Segment duration is the master knob: latency ≈ 3-4 segment lengths, so 2 s segments give 6-10 s latency with full CDN cacheability; go WebRTC only when sub-second latency is genuinely required.
  • ▸HLS turns live video into immutable static files, and CDN request collapsing makes origin load independent of viewer count; the playlist is the only mutable, short-TTL object.
  • ▸Transcode once per stream into a keyframe-aligned bitrate ladder; transcoding cost scales with streams while delivery cost scales with viewers, and the architecture should keep those independent.
  • ▸Chat fan-out cost is messages x viewers and can exceed the video problem; two-tier fan-out (pub/sub to gateways, local multicast) plus sampling and slow mode are backpressure disguised as features.
  • ▸Design for consistent latency, not minimal latency: players should micro-adjust playback speed to hold the live edge so chat and video stay synchronized.

Brush up on the underlying topics

CDN (Content Delivery Network)Real-Time CommunicationCachingMessage Queues and StreamingLoad Balancing