Hard

Design a Video Platform (YouTube/Netflix)

Design a platform where creators upload videos and viewers stream them worldwide. The system splits into an asynchronous upload-and-transcode pipeline and a read-dominated streaming path built on adaptive bitrate protocols (HLS/DASH) and a CDN. The core challenges are parallelizing transcoding, serving petabytes of video with low startup latency, and keeping origin traffic tiny relative to what viewers consume.

1Requirements

Functional

• Creators can upload videos up to several GB, with resumable uploads.
• Videos are transcoded into multiple resolutions and bitrates automatically after upload.
• Viewers can stream videos with adaptive quality on any device and network.
• Viewers can search for videos and see metadata (title, views, thumbnails).
• Track view counts and watch time per video.
• Creators are notified when processing completes and the video is live.

Non-functional

• Streaming startup latency under 1-2 seconds and minimal rebuffering.
• High availability for playback (99.99%); uploads may degrade before playback ever does.
• Durability: a published video must never be lost (11 nines object storage).
• Scale: ~1B DAU watching, ~500 hours of video uploaded per minute.
• Cost efficiency: bandwidth and storage dominate; CDN offload and per-title encoding matter at this scale.
• Processing pipeline is async with a target publish time of minutes, not a hard latency bound.

2Back-of-envelope estimation

Upload volume	500 hours/min x 60 x 24 = 720K hours/day; at ~1.5 GB per source hour ≈ 1 PB/day raw ingest
Storage after transcoding	Each source becomes ~5-8 renditions; roughly 2-3x source size ≈ 2-3 PB/day of new derived storage	Cold tiering and per-title encoding are mandatory at this rate
Streaming egress	1B views/day x 5 min avg x 3 Mbps ≈ 1B x 112 MB ≈ 110 PB/day, ~10 Tbps sustained	Physically impossible from origin; CDN must serve 95%+ of bytes
Metadata QPS	1B DAU x 20 page/metadata hits ≈ 230K QPS on metadata, peak 2x	Cache-first: metadata is small and hot
Transcode compute	720K hours/day ingested x ~2x realtime per rendition x 6 renditions ≈ 8.6M compute-hours/day	Chunked parallel transcoding turns hours of latency into minutes

3API design

POST /api/videos

Initialize an upload: { title, description, size, checksum }. Returns a videoId and pre-signed multipart upload URLs pointing directly at object storage, bypassing app servers.

PUT {presignedUrl} (chunk upload)

Client uploads each chunk directly to object storage; ETags per part enable resume after failure. A final complete-upload call assembles parts and fires the processing event.

GET /api/videos/{videoId}

Fetch metadata: title, duration, status (processing/live), thumbnails, and the manifest URL for playback.

GET /manifests/{videoId}/master.m3u8

CDN-served HLS master playlist listing available renditions; the player picks variants and fetches segment playlists and .ts/.mp4 segments from the CDN.

POST /api/videos/{videoId}/events

Batched playback telemetry (view start, heartbeats, quality switches) feeding view counts and analytics.

4High-level design

Uploads never pass through application servers. The client requests pre-signed URLs and pushes chunks of the raw file directly into object storage (S3/GCS) using multipart upload, which gives resumability for free: on failure the client re-uploads only missing parts. When the upload completes, storage emits an event onto a message queue, and the video's metadata row flips to 'processing'.

The transcoding pipeline is a DAG of asynchronous workers driven by queues. A splitter breaks the source into ~5-10 second chunks aligned on keyframes (GOP boundaries); a fleet of transcode workers processes chunks in parallel, each producing every target rendition (e.g., 240p through 4K at appropriate bitrates); an assembler stitches results, generates HLS and DASH manifests, thumbnails, and captions; a validator checks output integrity. Parallelizing by chunk means a 2-hour movie transcodes in roughly the time of one chunk times pipeline overhead, minutes instead of hours.

Processed segments and manifests land in object storage, which acts as the CDN origin. Popular content is pushed or pulled into CDN edge caches worldwide. The playback path is: player fetches the master manifest from the CDN, chooses a rendition based on measured bandwidth, and streams small segments over plain HTTPS, switching renditions between segments as conditions change. Because everything is static files over HTTP, standard CDN infrastructure serves it with no special streaming servers.

The metadata path is a conventional read-heavy service: video metadata in a sharded database fronted by cache, search via an inverted index (Elasticsearch) fed by change events, and view counts aggregated from telemetry events through a stream processor rather than synchronous increments.

Netflix-style optimization for a smaller, ultra-popular catalog: precompute per-title encoding ladders (analyze each title's complexity to choose bitrates, saving ~20% bandwidth) and pre-position entire catalogs on appliances inside ISP networks (Open Connect) during off-peak hours, so peak-hour traffic barely touches the backbone. YouTube's long tail instead relies on pull-through caching with popularity-tiered retention.

5Data model

videos

video_id BIGINT PK, uploader_id BIGINT, title VARCHAR(255), description TEXT, duration_s INT, status VARCHAR(20), created_at TIMESTAMP, published_at TIMESTAMP NULL

Hot metadata; cached aggressively, source of truth for lifecycle state

video_renditions

video_id BIGINT, rendition VARCHAR(10) (e.g. 720p), bitrate_kbps INT, codec VARCHAR(10), manifest_path VARCHAR(512), segment_prefix VARCHAR(512), PK (video_id, rendition)

transcode_jobs

job_id UUID PK, video_id BIGINT, chunk_index INT, rendition VARCHAR(10), state VARCHAR(20), attempts INT, worker_id VARCHAR(64), updated_at TIMESTAMP

Tracks DAG progress; idempotent retries keyed by (video, chunk, rendition)

view_stats

video_id BIGINT, bucket_ts TIMESTAMP, views BIGINT, watch_time_s BIGINT, PK (video_id, bucket_ts)

Written by stream aggregation, not per-view increments

6Deep dives

The transcoding pipeline as a chunked DAG

Transcoding one large file serially is slow (often slower than realtime per rendition) and fragile: a failure at 90% wastes all the work. The fix is to split the source on GOP (group of pictures) boundaries into independent chunks, so each chunk can be decoded and re-encoded without neighbors. Chunks fan out across thousands of workers; each (chunk, rendition) task is an idempotent unit tracked in a job table, retried on failure, and safe to run twice.

Model the whole flow as a DAG: split → per-chunk transcode → audio processing → thumbnail and caption generation → manifest assembly → validation → publish. A DAG scheduler (Facebook described theirs as SVE; Temporal-style workflow engines work too) tracks state transitions and resumes from the last completed node after any crash.

Two practical notes for interviews: use spot/preemptible instances for the enormous but interruption-tolerant transcode fleet to cut cost, and prioritize the ladder so a watchable 360p rendition publishes first, letting the video go live in seconds while higher qualities backfill.

Adaptive bitrate streaming: HLS and DASH

Naive progressive download of one MP4 at one quality fails on variable networks: too high a bitrate causes rebuffering, too low wastes quality. Adaptive bitrate (ABR) streaming solves this by encoding each video at a ladder of bitrate/resolution pairs and cutting each rendition into small segments (2-10 seconds). A master manifest lists the renditions; per-rendition playlists list the segments.

The intelligence lives in the client. The player measures download throughput and buffer occupancy and picks the best rendition for each next segment, stepping down instantly when bandwidth drops and up when it recovers. Because segments align across renditions, switches are seamless. HLS (Apple, .m3u8, historically MPEG-TS segments) and DASH (open standard, fragmented MP4) are the two protocols; platforms typically serve both, and CMAF lets one set of media segments back both manifest formats, halving storage.

ABR is also what makes CDNs work here: every segment is an immutable static file over HTTP, so ordinary HTTP caches serve it, byte-range requests are unnecessary, and cache keys are stable forever.

CDN strategy and the economics of egress

The envelope math (roughly 110 PB/day, ~10 Tbps) makes the CDN the system's backbone, not an optimization. Segments flow origin → regional shield cache → edge PoP → viewer, and each layer absorbs misses from the one below, so origin egress ends up under a few percent of delivered bytes. Immutable segment URLs (content-addressed or versioned paths) allow infinite TTLs with zero invalidation logic.

Popularity is extremely skewed: a small fraction of titles produce most watch time. Netflix exploits this with Open Connect appliances placed inside ISP data centers and filled with the regional catalog during off-peak windows, meaning peak streaming traffic never crosses transit links. YouTube's billion-video long tail cannot be pre-positioned, so it uses pull-through caching with tiered retention and serves true cold tail requests from regional storage.

Cost levers worth naming: per-title encoding (tuning the ladder to content complexity saves ~20% of bits), newer codecs (VP9/AV1 save 30-50% over H.264 at the price of more encode compute, worth it only for popular titles), and cold-tiering derived renditions of rarely watched videos while keeping only the source, re-deriving on demand.

Resumable uploads and view counting

Multi-GB uploads on flaky networks will fail mid-way; restarting from zero is unacceptable. Multipart upload solves it: the client splits the file into parts (e.g., 10 MB), uploads each part independently to a pre-signed URL, and storage tracks received parts, so resume means asking which parts exist and sending the rest. Parts also upload in parallel, improving throughput. Pre-signed URLs keep petabytes of ingest off the application tier entirely.

View counting looks trivial but is a classic hot-row problem: a viral video takes thousands of view events per second, and synchronous UPDATE ... SET views = views + 1 serializes on that row. Route playback telemetry through a queue into a stream aggregator (Flink/Kafka Streams) that windows counts and flushes periodic deltas to storage and cache. Counts become near-real-time approximations, deduplicated per (user, video, session) to resist inflation, and exact totals reconcile in batch. Interviewers reward acknowledging that displayed counts are intentionally eventually consistent.

7Rapid implementation: build the MVP

Theory is table stakes. Here is how you would stand up a working version fast, on a budget, with the core algorithm in real code.

Stack

Next.js + Postgres + BullMQ worker running ffmpeg, files on S3-compatible storage (Cloudflare R2, zero egress fees) served through Cloudflare CDN, hls.js player

01Create tables: videos (id, title, status DEFAULT 'uploading', duration_s, created_at) and renditions (video_id, name, bandwidth, playlist_path).
02Build POST /api/videos to insert a row and return a pre-signed R2 multipart PUT URL so the browser uploads the source file directly to storage.
03On the upload-complete callback, flip status to 'processing' and enqueue a BullMQ transcode job with the videoId.
04In the worker, download the source and run ffmpeg once per rendition (start with 480p and 720p) producing HLS segments and a per-rendition .m3u8.
05Generate a master.m3u8 listing both renditions with BANDWIDTH and RESOLUTION attributes, upload all output under videos/{id}/ in R2, set status 'live'.
06Serve playback with hls.js pointed at the CDN URL of master.m3u8; the player handles rendition switching for free.
07Set Cache-Control: public, max-age=31536000, immutable on segments (they never change) and a short max-age on playlists.
08Count views by POSTing a beacon at 10s of playback into a view_events table; roll up per-video counts with a minutely cron, never increment synchronously.

Transcode worker: ffmpeg per rendition to HLS

typescript

import { execFile } from "node:child_process";
import { promisify } from "node:util";
const run = promisify(execFile);

const LADDER = [
  { name: "480p", height: 480, vBitrate: "1200k", bandwidth: 1400000 },
  { name: "720p", height: 720, vBitrate: "2800k", bandwidth: 3200000 },
];

async function transcode(videoId: string, srcPath: string, outDir: string) {
  for (const r of LADDER) {
    await run("ffmpeg", [
      "-i", srcPath,
      "-vf", "scale=-2:" + r.height,
      "-c:v", "libx264", "-b:v", r.vBitrate, "-preset", "fast",
      "-c:a", "aac", "-b:a", "128k",
      "-g", "48", "-keyint_min", "48", "-sc_threshold", "0", // aligned keyframes
      "-hls_time", "6", "-hls_playlist_type", "vod",
      "-hls_segment_filename", outDir + "/" + r.name + "_%04d.ts",
      outDir + "/" + r.name + ".m3u8",
    ]);
  }
}

Master HLS playlist generation

typescript

function buildMasterPlaylist(renditions: { name: string; bandwidth: number; height: number }[]): string {
  const lines = ["#EXTM3U", "#EXT-X-VERSION:3"];
  for (const r of renditions) {
    const width = Math.round((r.height * 16) / 9);
    lines.push(
      "#EXT-X-STREAM-INF:BANDWIDTH=" + r.bandwidth +
      ",RESOLUTION=" + width + "x" + r.height
    );
    lines.push(r.name + ".m3u8");
  }
  return lines.join("\n") + "\n";
}

// after transcoding, publish everything and go live
async function publish(videoId: string, outDir: string) {
  await uploadDir(outDir, "videos/" + videoId + "/"); // R2 put per file
  await sql("UPDATE videos SET status = 'live' WHERE id = $1", [videoId]);
}

Pre-signed direct-to-storage upload init

typescript

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

const r2 = new S3Client({
  region: "auto",
  endpoint: process.env.R2_ENDPOINT,
  credentials: { accessKeyId: process.env.R2_KEY!, secretAccessKey: process.env.R2_SECRET! },
});

export async function POST(req: Request) {
  const { title, contentType } = await req.json();
  const rows = await sql(
    "INSERT INTO videos (title, status) VALUES ($1, 'uploading') RETURNING id",
    [title]
  );
  const videoId = rows[0].id;
  const uploadUrl = await getSignedUrl(
    r2,
    new PutObjectCommand({ Bucket: "videos", Key: "sources/" + videoId, ContentType: contentType }),
    { expiresIn: 3600 }
  );
  return Response.json({ videoId, uploadUrl }); // browser PUTs the file itself
}

Bottlenecks & failure modes

⚠Origin bandwidth: serving even a few percent of 110 PB/day from origin is enormous; layered CDN caching with immutable URLs is the fix.
⚠Transcode backlog during upload spikes delays publishing; autoscale workers on queue depth and publish low renditions first.
⚠Hot metadata rows for viral videos (counts, comments) need async aggregation and cache-first reads.
⚠Storage growth of 2-3 PB/day forces lifecycle policies: cold-tier or drop unused renditions for tail content.
⚠A thundering herd on a just-published video from a huge creator can stampede CDN misses to origin; use request coalescing at shields and pre-warm edges for predictable premieres.

Key takeaways

▸Split the design cleanly: an async write pipeline (upload, transcode, publish) and a static-file read path (manifests, segments, CDN); they scale independently.
▸Chunked parallel transcoding on GOP boundaries turns hours into minutes and makes every unit of work idempotent and retryable.
▸Adaptive bitrate (HLS/DASH) puts quality decisions in the client and turns streaming into cacheable static HTTP, which is what makes CDNs sufficient.
▸Pre-signed direct-to-storage multipart uploads give resumability and keep bulk bytes off app servers.
▸At video scale, cost is architecture: CDN offload, per-title encoding, codec choice, and storage tiering are design decisions, not afterthoughts.

Brush up on the underlying topics

CDN (Content Delivery Network)Message Queues and Streaming Storage and Search Event-Driven Architecture Caching