DSA & System Design
Full-stackHard45 min

Design YouTube (Video Streaming)

Upload in resumable chunks, transcode into short segments in parallel, and let the player pick quality while a CDN serves the bytes.

Related:Async Processing and MessagingCaching Explain it back
Say it in 30 seconds
  • 12.5 Tbpspeak egress for 5M streams
  • 750 tasksto encode a 10 minute video
  • ~95%CDN hit rate keeps origin near 0.6 Tbps
Show what to say

“I'd split it in two. Uploads are resumable 8 MiB chunks straight to object storage, then a job fans out into 750 idempotent encode tasks: 4-second segments at five qualities. Playback is HLS: the player picks a quality per segment from measured throughput, and a CDN serves about 95% of the 12.5 Tbps peak.”

Trade-offs

  • Pre-encode all qualities vs on demand: instant playback, but storage and CPU spent on videos nobody watches.
  • Short vs long segments: 2 second segments adapt faster to a bad network, but cost more requests and overhead.
  • Client-side bitrate choice vs server-side: the player sees its own network, but needs a conservative safety margin to avoid stalls.

Bring it up before they ask

A dead transcoder: tasks are idempotent with deterministic output paths and expiring leases, so another worker simply runs the task again.

0 of 0 sections ticked
30-second answer

Upload in resumable chunks, transcode into short segments at several qualities in parallel, and let the player pick quality segment by segment (). Nearly all bytes are served by a , not by your servers.

Functional

  • Upload a video, even a large one over a flaky connection
  • Process it into formats every device can play
  • Stream it with fast start and no buffering stalls
  • Browse and search by title, owner and tags
  • Count views and likes without slowing playback

Non-functional

  • Playback starts in about 2 seconds
  • A 10 minute upload is playable within minutes
  • Peak of 5 million concurrent streams
  • Uploaded videos are never lost
  • Reads outnumber uploads about 1,000 to 1

Out of scope

  • Recommendations and ranking
  • Live streaming (a follow-up)
  • Ads, payments and copyright matching

Clarifying questions, scale and interface

Read more · 366 words
  • Scale? Uploads a day, daily viewers, peak concurrent streams.
  • Devices? Weak phones and fibre TVs need different qualities.
  • Live or on demand? On demand here. Live changes the pipeline.

Capacity estimate

QuantityCalculationResult
Uploads500K videos/day ÷ 86,400 s≈ 6 per second
Raw storage500K × 500 MB250 TB a day
Encoded storage5 qualities = 9.9 Mbps; 600 s × 9.9 Mbps ÷ 8 = 742.5 MB per video; × 500K371 TB a day
Views100M viewers × 5 videos ÷ 86,400 s≈ 5,800/s
Peak egress5M streams × 2.5 Mbps12.5 Tbps
Edge servers12.5 Tbps ÷ 10 Gbps each≈ 1,250
Origin egress5% CDN miss × 12.5 Tbps≈ 0.6 Tbps
Transcoding CPU500K × 600 s × 5 qualities ÷ 2x realtime ÷ 86,400 s≈ 8,700 cores

Storage, egress and encoding CPU are the big three. Egress rules out your own servers, so the is the design; an archive tier keeps storage cheap.

API

MethodPathNotes
POST/videos{ title, size, type } → { videoId, uploadId, chunkSize }
PUT/uploads/:idOne chunk, Content-Range: bytes 0-8388607/524288000
HEAD/uploads/:idHow many bytes are stored in order: the resume point
POST/videos/:id/completeOptional; also triggered by the storage event
GET/videos/:idMetadata and the playlist URL once READY
GET/v/:id/master.m3u8Served by the CDN, with /v/:id/720p/seg_00042.m4s
POST/videos/:id/viewA beacon, counted asynchronously

Data model

Bytes live in under v/{videoId}/{quality}/seg_{n}.m4s. Metadata is by video id. View counts go through into counters and are flushed to the table in batches.

CREATE TABLE videos (
  id          bigint PRIMARY KEY,                -- from an ID generator, so shards need no coordination
  owner_id    bigint NOT NULL,
  title       text   NOT NULL,
  status      text   NOT NULL CHECK (status IN ('UPLOADING', 'PROCESSING', 'READY', 'FAILED')),
  duration_s  int,
  created_at  timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE upload_sessions (
  upload_id   uuid PRIMARY KEY,
  video_id    bigint NOT NULL REFERENCES videos (id),
  size_bytes  bigint NOT NULL,
  stored_in_order bigint NOT NULL DEFAULT 0      -- the resume point
);

Architecture and CDN

The pipeline has two halves. Upload and are heavy but rare.

Read more · 141 words

Playback is light per viewer but happens a thousand times more often. Play Upload and process, Watch a video, Network gets worse and Upload interrupted.

and

Segment files never change, so a CDN can keep them for a long time. Popular videos sit at the edge; the long tail is pulled in on first view. At a 95% the origin sees about 0.6 Tbps instead of 12.5.

Press Play to watch it run · 9 steps

Uploads go in chunks straight to object storage. An event queues a transcoding job that cuts the video into short segments and encodes each one at five qualities in parallel. Viewers fetch a playlist from the metadata API, then pull segments from a CDN, which asks the video store only on a miss.

Video streaming architectureUploads go in chunks straight to object storage. An event queues a transcoding job that cuts the video into short segments and encodes each one at five qualities in parallel. Viewers fetch a playlist from the metadata API, then pull segments from a CDN, which asks the video store only on a miss.init, chunksput chunksrowuploadedjobread sourcesegmentsREADYwatch pagelookupplaylist, segmentsmissUploader appresumableCreates the video row and an upload session, and hands out chunk URLs. The bytes go to object storage, not through these servers.Upload APIsessions + URLsRaw storeoriginal filesStateless workers. Each takes one task: encode segment N of video V at quality Q. Same input, same output path, so a task can safely run twice.Transcoders~8,700 coresObject storage holding every encoded segment and the HLS playlists. Segment files never change, so they can be cached for a very long time.Video storesegments + playlistsMetadata APItitles, statusMetadata DBsharded by videoJob queueone job per videoViewer appadaptive playerServes nearly all traffic from servers near the viewer. About 5% of requests miss and go back to the video store.CDN edge~95% hit rate

Tap any box to see what it is and what it talks to.

  • Client
  • Service
  • Object storage
  • Worker
  • Database
  • Queue
  • CDN
Open in playground
Text description
  1. The app starts an upload: size 500 MiB, type video/mp4. The API creates the video row and an upload session.
  2. The row starts in status UPLOADING. The viewer pages will refuse it until it is READY.
  3. The app sends 8 MiB chunks. 500 MiB is 63 chunks, each sent to the raw store with a signed URL.
  4. The raw store holds the original file. The API servers never hold the video bytes in memory.
  5. When the last chunk lands, an 'object created' event queues one job for the video.
  6. A coordinator splits the job into 750 encode tasks: 150 four-second segments at 5 qualities.
  7. Many transcoders read the source at once, each cutting its own piece.
  8. Each task writes a segment to a path that depends only on the video, quality and index. Then one task writes the playlists.
  9. The row flips to READY. With this much parallelism a 10 minute video is ready in minutes, not hours.

Upload and transcoding

Big uploads fail, so make them resumable: 8 MiB chunks go straight to , and a failed upload continues from the last byte stored.

Read more · 56 words

Then cut the video into 4-second segments and encode each segment at each quality as its own task. A 10 minute video at 5 qualities is 750 tasks. Each writes a path set by video, quality and index, so a retry is harmless. Pick a branch: all succeed, a worker dies, or the source is corrupt.

Press Play to watch it run · 9 steps

A coordinator splits the video into segments and fans out encode tasks to workers. Tasks are idempotent: a failed task is retried on another worker while the others carry on. Only after all segments exist does the packager write the playlists.

A transcoding jobA coordinator splits the video into segments and fans out encode tasks to workers. Tasks are idempotent: a failed task is retried on another worker while the others carry on. Only after all segments exist does the packager write the playlists.job: video V, 600 sPlan 750 tasks: 150 segments x 5 qualitiesencode V/720p/7encode V/720p/8alt [all succeed]put segment 7put segment 8donedoneAfter the 750th task, the package task runsput master + playlistsstatus = READY
  • Queue
  • Service
  • Worker
  • Object storage
  • Database
Text description
  1. Job queue → Coordinator: job: video V, 600 s
  2. Coordinator sends (async) to Worker A: encode V/720p/7
  3. Coordinator sends (async) to Worker B: encode V/720p/8
  4. Worker A → Video store: put segment 7
  5. Worker B → Video store: put segment 8
  6. Worker A replies to Coordinator: done
  7. Worker B replies to Coordinator: done
  8. Coordinator → Video store: put master + playlists
  9. Coordinator → Metadata DB: status = READY

Adaptive bitrate

The player, not the server, chooses quality.

Read more · 86 words

It times each segment download and picks the best quality that fits within 80% of its smoothed .

QualityBitrate
240p0.4 Mbps
360p0.8 Mbps
480p1.2 Mbps
720p2.5 Mbps
1080p5 Mbps

Down is fast, up is slow: a thin buffer steps down at once, while a step up needs a healthy buffer and goes one level at a time.

How to approach it

  1. Clarify

    Scale, devices, length, on demand or live.

  2. Estimate

    Egress 12.5 Tbps, storage 600+ TB a day, 8,700 encoding cores.

  3. Draw both halves

    Upload and processing, then playback through a .

  4. Deep dives

    Resumable upload, tasks, , .

  5. Failure and cost

    Worker crashes, hot videos, storage tiers.

Read more · 294 words

Step-by-step answer

  1. Split the problem in two

    "Upload and processing: heavy, rare, asynchronous. Playback: light each, but huge in total." Different shapes, different designs.

  2. Make uploads resumable and direct

    Signed chunk URLs send bytes to , not through your API servers. The server keeps the resume point.

  3. Queue one job, fan out many tasks

    The storage event queues a job. A coordinator plans 750 tasks with leases and . A dead worker's lease expires and another takes the task.

  4. Package for streaming

    Write HLS or DASH playlists and fragmented MP4 segments. Keep segment boundaries aligned across qualities so switching is seamless.

  5. Serve through a CDN

    Immutable segments, long cache times, an origin shield between edge and storage to collapse duplicate misses.

  6. Start fast

    Begin at a low quality, then climb. Preload the next segment. First frame in about 2 seconds beats the first frame in high quality.

  7. Count views off the hot path

    A beacon goes to . Counters aggregate in memory and flush in batches. Showing a number a minute stale is fine.

  8. Control cost

    Move old, unpopular videos to cheaper storage. Encode rare qualities only when a viewer asks. Drop the 240p rendition if almost nobody uses it.

Implementation

Tabs: the resumable upload client, the plan, the player's quality chooser, and playlist generation.

Read more · 44 words
  • The upload test breaks the connection once and proves every byte is sent exactly once.
  • planJob() yields 750 tasks for a 10 minute video; the test also reproduces the 8,681-core estimate.
  • pickRendition() tests cover a bandwidth drop, a thin buffer, and one-step upswitching.
// Resumable chunked upload, client side. The server remembers how many bytes it has (in order);
// after any failure the client asks "where are you?" and continues from there instead of starting over.
export const CHUNK = 8 * 1024 * 1024;          // 8 MiB: big enough to be efficient, small enough that a failure wastes little

export function chunkPlan(size, chunk = CHUNK) {
  const parts = [];
  for (let start = 0; start < size; start += chunk) parts.push({ start, end: Math.min(start + chunk, size) });
  return parts;
}

// `session.offset()` asks the server how many bytes it has; `session.put(start, end)` sends one chunk and may throw.
export async function upload(size, session, { maxRetries = 5, sleep = (ms) => new Promise((r) => setTimeout(r, ms)) } = {}) {
  let failures = 0;
  let sent = 0;
  for (;;) {
    const have = await session.offset();                    // truth lives on the server
    if (have >= size) return { sent };
    const todo = chunkPlan(size).filter((p) => p.end > have);
    try {
      for (const p of todo) {
        await session.put(Math.max(p.start, have), p.end);
        sent += p.end - Math.max(p.start, have);
      }
    } catch (err) {
      if (++failures > maxRetries) throw err;
      await sleep(Math.min(30_000, 2 ** failures * 250));   // exponential backoff; real code adds jitter
    }
  }
}

Failure modes and trade-offs

Read more · 260 words
What breaksEffectMitigation
Upload connection dropsCreator restarts a big uploadResumable chunks; resume point on the server
Transcoder dies mid-taskSegment missingTask lease expires; retry on another worker; same output path
Corrupt source fileEvery attempt failsCap retries; mark FAILED with a reason; tell the creator
Viral video, cold edgeOrigin overloadedOrigin shield; request collapsing; pre-warm
CDN region outageViewers there bufferFail over to another CDN or region
Queue backs upSlow time to publishAutoscale workers; priority lanes for short or paid videos

Trade-offs

ChoiceGainCost
Pre-encode all qualitiesInstant, simple playbackStorage and CPU for rarely watched videos
Encode on demandSaves storageFirst viewers wait
Short segments (2 s)Faster adaptationMore requests and overhead
Long segments (10 s)Fewer requestsSlow to react to a bad network

Follow-up questions

How would live streaming change this?

Segments are made continuously, so encoding must keep up with realtime and playlists update every few seconds.

How do you make the first frame appear fast?

Start at a low quality with a small, edge-cached first segment, and open connections early.

How do you handle a hot new video?

Pre-warm the edges and add an origin shield so many misses become one fetch.

Check yourself

Question 1 of 3

Score 2 out of 3 or better to mark this page complete.

1. Why cut a video into 4-second segments at several qualities instead of encoding one file per quality?