Full-stackHard45 min

Design a News Feed (Twitter or Instagram)

The key decision is fan-out on write versus fan-out on read, and how a React UI consumes the result.

Say it in 30 seconds
  • 580 to 116K/sposts per second become fan-out writes
  • ~100K followerscelebrity threshold: pull instead of push
  • 800 idscap on each user's cached feed
Show what to say

“I'd use hybrid fan-out. A normal user's post is pushed into each active follower's capped Redis feed, so a read is one lookup. Celebrities, over about 100K followers, are pulled and merged at read time, so one post never costs millions of writes. Reads hydrate ids through a post cache and page with a score cursor.”

Trade-offs

  • Push vs pull: push makes reads one lookup but write cost grows with followers; pull writes once but every read merges timelines.
  • Cache every feed vs active users only: no rebuilds, but terabytes of RAM for people who never look.
  • Lower vs higher celebrity threshold: less write amplification, but more read-time merging.

Bring it up before they ask

Raise the celebrity problem yourself: pure push breaks for a 30M-follower account, since at 100K writes per second one post takes 5 minutes to reach everyone.

New to the words? Read the one-minute primer

0 of 0 sections ticked
30-second answer

Hybrid : copy a normal user's post into each follower's ready-made feed, so reading is one lookup. For celebrities (over ~100K followers), skip the copy and merge their posts in when the feed opens.

Functional

  • Follow and unfollow
  • Post text and media
  • A home feed, ranked or reverse-chronological
  • Like and comment
  • New posts appear without a manual refresh

Non-functional

  • Feed p99 under 200 ms
  • Followers see new posts within seconds
  • Celebrities with millions of followers
  • 300M daily active users
  • Available over consistent: a feed a few seconds stale is fine

Out of scope

  • Ads and ad ranking
  • Search and hashtags
  • Direct messages

Scope and scale

Ask:

Read more · 138 words
  • Ranked or chronological? Ranking adds a scoring step.
  • Follow graph? Average follows, biggest follower count, a follow limit?
  • Freshness? Seconds, or is a minute fine?
  • Celebrities? Millions of followers? Yes: this is the crux.
  • Scale? Daily users, feed opens, posts per day.
QuantityCalculationResult
Feed reads300M × 10/day≈ 35K/s (~100K/s peak)
New posts50M/day≈ 580/s
Fan-out writes50M posts × 200 average followers≈ 116K/s
Cached feed memory300M × 800 ids × ~40 B≈ 9.6 TB

Reads outnumber posts 60 to 1, so do the work at write time. The bill: 580 posts/s become 116K writes/s, and 9.6 TB of RAM. So only active users' feeds.

Interactive

Capacity calculator

Move any slider: these are the numbers to say out loud.

More assumptions

Ballparks at 10 actions each: 100K DAU ≈ 11.6 QPS · 1M DAU ≈ 115.7 QPS · 10M DAU ≈ 1.2K QPS · 100M DAU ≈ 11.6K QPS · 1B DAU ≈ 115.7K QPS.

Average QPS
38,194
37,568 reads · 626 writes
Peak QPS
114,583
3× average
Storage
296 TB
54.1 GB/day × 5y × 3 replicas
Read egress
751 MB/s
≈ 6 Gbps
App servers
96
at 60% of 2,000 req/s each
Cache (hot 20%)
6.5 TB
upper bound for a day's reads

Formula: QPS = 300M × 11 ÷ 86,400 = 38,194.

API and data model

Read more · 181 words
MethodPathNotes
POST/postsCreates a post; fan-out runs asynchronously
GET/feed?cursor=&limit=20{ posts, nextCursor }
POST/DELETE/users/:id/followFollow or unfollow
POST/posts/:id/likeOptimistic on the client
GET/feed/new-count?after=TSFor the "new posts" banner

nextCursor is the last post's timestamp; null means nothing older exists.

Posts and follows live in a durable store. Ids are time-ordered.

Feeds are derived data in :

KeyTypeContents
feed:USER_IDsorted setscore = createdAt ms, member = post id; newest 800
posts:AUTHOR_IDsorted setThe author's own timeline, for the celebrity merge
post:IDstring (JSON)Post body cache, 1 h TTL

A time-scored makes the score cursor work: "20 posts older than T" stays correct as new posts arrive.

{
  "posts": [
    { "id": "p_981", "author": { "id": "u_7", "name": "Maya" }, "text": "Sunset", "media": ["img/981.jpg"], "likes": 12, "liked": false, "createdAt": 1760000000000 }
  ],
  "nextCursor": 1759998000000
}
CREATE TABLE posts (
  id         bigint PRIMARY KEY,
  author_id  bigint NOT NULL,
  body       text,
  media      jsonb,
  created_at timestamptz NOT NULL DEFAULT now(),
  deleted    boolean NOT NULL DEFAULT false
);
CREATE INDEX posts_by_author ON posts (author_id, created_at DESC);

CREATE TABLE follows (
  follower_id bigint NOT NULL,
  followee_id bigint NOT NULL,
  created_at  timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (follower_id, followee_id)
);
CREATE INDEX follows_by_followee ON follows (followee_id, follower_id);  -- "who follows X?" for fan-out

Architecture

Each user's feed is a ready-made list of post ids, kept in fast memory.

Read more · 51 words

Writes store the post, queue the copying and return, so the author never waits. Reads touch only . Play Normal user posts, Celebrity posts and User opens feed.

A cache is sticky notes on your monitor: the answers you need most, within arm's reach.

Press Play to watch it run · 8 steps

Posts by normal authors are pushed into each active follower's cached feed by fan-out workers. Celebrity posts are only written to the author's timeline and merged in at read time. Reads hydrate post ids through a post cache.

News feed with hybrid fan-outPosts by normal authors are pushed into each active follower's cached feed by fan-out workers. Celebrity posts are only written to the author's timeline and merged in at read time. Reads hydrate post ids through a post cache.POST /postsinsertZADD≤ 100KZADD ×NGET /feedpullhydrateon missmediaAuthorPost serviceFan-out queue1 msg per 1KFan-out workers×N, autoscaledThe source of truth for post bodies, sharded by post id.Posts DB+ read replicasEvery author has a capped timeline of their newest 800 post ids; the feed API reads only the celebrities' ones at read time.Celeb timelinesposts:<author>One sorted set per active user, score = createdAt, capped at 800 ids.Feed cachefeed:<user> ZSETReaderFeed APImerge + hydratePost cachepost:<id>CDNimages + video

Tap any box to see what it is and what it talks to.

  • Client
  • Service
  • Queue
  • Worker
  • Database
  • Cache
  • CDN
Open in playground
Text description
  1. Maya, who has 300 followers, posts a photo. POST /posts reaches the post service.
  2. The post is written to the Posts DB, the source of truth.
  3. Its id is also added to Maya's own author timeline, posts:maya. Every author has one, capped at 800 ids.
  4. Maya is under the 100K-follower threshold, so fan-out jobs are enqueued: one message per 1,000 followers, so just one here.
  5. Maya gets 201 Created right away and sees her own post. Delivery to followers continues in the background.
  6. A fan-out worker takes the job and drops followers who haven't opened the app in 30 days.
  7. For each active follower: ZADD feed:ID with the post's timestamp, then trim to the newest 800. Pipelined, a batch of 1,000 takes milliseconds.
  8. Within a second or two the post sits at the top of every active follower's cached feed.

Push, pull or hybrid

The whole question turns on this. Compare the two pure strategies first.

Read more · 131 words

Push copies each post into every follower's feed, so reads are one lookup. Pull stores a post once, but every feed open merges many timelines. The hybrid pushes for most authors and pulls only the few celebrities each user follows.

Do the sum first. Say the workers manage about 100K feed writes a second:

AuthorFollowersPush takes
Typical user200Instant
Popular account100KAbout 1 second
Celebrity30MAbout 5 minutes

Push the first two. Pull the third: fans see the post when they next open the feed. A user who follows 5 celebrities merges only 5 extra timelines. An app where nobody has many followers can skip and just pull.

Press Play to watch it run · 6 steps

Push writes a post id into every follower's feed so a read is one lookup. Pull writes once to the author's timeline and merges every followed author's timeline at read time.

Fan-out on write vs fan-out on readPush writes a post id into every follower's feed so a read is one lookup. Pull writes once to the author's timeline and merges every followed author's timeline at read time.× followers× 11 read× followeesAuthorPost serviceFollower feeds1 ZSET/followerAuthor timelines1 ZSET/authorFeed APIReader (Ben)

Tap any box to see what it is and what it talks to.

  • Client
  • Service
  • Cache
Open in playground
Text description
  1. An author with 200 followers posts.
  2. Fan-out on write: the post id is written into all 200 follower feeds. 200 writes for one post, done in the background.
  3. Later, Ben opens his feed.
  4. One read: the newest 20 ids from feed:ben. His feed is already assembled.
  5. The ids come back in about a millisecond.
  6. Cheap, fast reads; expensive writes. An author with 30M followers would need 30M writes for every post.
Interactive

Fan-out calculator

Push copies a post into every feed; pull builds feeds on read.

Writes per post
2K
one per follower
Reach everyone
40 ms
at 50K writes/s
Feed memory
384 KB/day
128 KB per post
Writes per day
6K
< 1/s on average

Recommendation: Push (fan-out on write)

2K feed writes per post finish in 40 ms. Every follower then reads one precomputed list: the cheapest read path there is.

Push: time to reach every follower
0204060ms05001K1.5K2K2.5KFollowers
Show data table
Push: time to reach every follower
FollowersPush delivery time (ms)
00
50010
1K20
1.5K30
2K40
2.5K50
Rule of thumb and assumptions

Push for accounts under ~100K followers, pull (merge at read time) above it. The line is straight: delivery time grows with every follower you add. Each feed entry costs ~64 B in the cache (post id, score and sorted-set overhead); the pull side assumes 10 feed opens per follower per day; a feed should feel fresh within 5 s.

Step-by-step answer

  1. Clarify

    About 60 reads per post justifies work at write time.

  2. Push: fan-out on write

    Write each post id into every follower's feed. Reads are one lookup, but 10M followers means 10M writes.

  3. Pull: fan-out on read

    Merge timelines at read time. Writes are cheap, but reads are too slow at 35K/s.

  4. Hybrid: push for most, pull for celebrities

    Tune the ~100K threshold on write cost versus read cost. Users follow few celebrities, so the merge stays small.

  5. Feed storage

    A per user, scored by time, capped at 800. Skip users inactive for 30 days.

    redis.zadd(`feed:${userId}`, ts, postId);
    redis.zremrangebyrank(`feed:${userId}`, 0, -801);             // keep the newest 800
  6. Read path: score cursor, merge, hydrate

    Read a page by score cursor, plus the same window from each followed celebrity. Merge, , sort, then fetch post bodies and drop deleted ones.

    const max = cursor ? `(${cursor}` : '+inf';                     // "(" = strictly older than the cursor
    redis.zrevrangebyscore(`feed:${userId}`, max, '-inf', 'WITHSCORES', 'LIMIT', 0, limit);
  7. Ranking

    Start chronological, then score candidates: affinity × (likes + 2·comments + 3·shares + 1) / (ageHours + 2)^1.5. A learned model can replace the formula later.

  8. Freshness

    Poll /feed/new-count every 30 to 60 s and show a banner. Never insert above the reader.

  9. Unfollow and delete

    Filter at read time, then clean up asynchronously. Never block the user on cleanup.

  10. Client

    A virtualized list (see Build an Infinite Scroll Feed) and optimistic likes that roll back on error.

Read more · 29 words

Implementation

Server tabs: , the read path, helpers.

Read more · 139 words

Client tabs: an optimistic like.

feed.js fixes two classic bugs: offsets that repeated posts (now a score cursor), and no celebrity merge (now mergeTimelines).

Then build this design in the playground and check it against the rubric:

  • post service → queue → fan-out workers → feed ;
  • celebrity timelines readable by the feed service;
  • a post cache and a database with ;
  • a for media.

See Caching and Async messaging.

// Write path with hybrid fan-out
const CELEBRITY_THRESHOLD = 100_000;

export async function publishPost(post) {                         // { id, authorId, createdAt, … }
  await db.posts.insert(post);
  await redis.zadd(`posts:${post.authorId}`, post.createdAt, post.id);   // author timeline (used for pull)
  await redis.zremrangebyrank(`posts:${post.authorId}`, 0, -801);
  if ((await graph.followerCount(post.authorId)) > CELEBRITY_THRESHOLD) return;   // merged at read time instead
  for await (const batch of graph.followersInBatches(post.authorId, 1_000)) {
    await queue.publish('fanout', { postId: post.id, ts: post.createdAt, to: batch });   // 1 message per 1,000 followers
  }
}

// Fan-out worker. With ioredis enableAutoPipelining, these calls are batched per node (works with Redis Cluster).
export async function fanoutWorker({ postId, ts, to }) {
  const active = await activeSince(to, 30);                       // skip users inactive for 30 days
  await Promise.all(active.flatMap(userId => [
    redis.zadd(`feed:${userId}`, ts, postId),
    redis.zremrangebyrank(`feed:${userId}`, 0, -801),             // keep the newest 800
  ]));
}
// Optimistic like with rollback
import { useMutation, useQueryClient } from '@tanstack/react-query';
import { likeInFeed } from './optimistic-like.js';

export function useLike() {
  const qc = useQueryClient();
  return useMutation({
    mutationFn: postId => fetch(`/api/posts/${postId}/like`, { method: 'POST' }).then(r => { if (!r.ok) throw new Error('like failed'); }),
    onMutate: async postId => {
      await qc.cancelQueries({ queryKey: ['feed'] });               // an in-flight refetch must not overwrite the optimistic state
      const previous = qc.getQueryData(['feed']);
      qc.setQueryData(['feed'], data => likeInFeed(data, postId));
      return { previous };
    },
    onError: (_err, _postId, ctx) => qc.setQueryData(['feed'], ctx?.previous),   // roll back
    // No blanket invalidate: refetching an infinite feed reloads every page. Reconcile the one post if needed.
  });
}

Failure modes and trade-offs

Read more · 141 words
What breaksEffectMitigation
Fan-out queue backlogPosts arrive lateAutoscale workers; authors see their own post at once
A worker dies mid-batchSome followers miss the postRedeliver; ZADD is idempotent
A Redis feed is lostEmpty feedRebuild by pulling
A celebrity's timeline is hammeredA hot keyShort in-process cache; replicate the key
A post is deleted after fan-outA ghost postFilter at read time
UnfollowOld posts lingerFollow check at read time

Choices to defend:

ChoiceGainCost
Ranked, not chronologicalBetter engagementRanking compute
Lower celebrity thresholdLess write costMore read-time merging
Cache every feedNo rebuildsTerabytes of RAM for idle users
SSE, not polling, for bannersInstantA connection per user

Follow-up questions

How would you rank posts?

Score pushed and pulled candidates, then mix so one author doesn't fill the page.

How do you handle unfollows and deleted posts?

Filter at read time, then clean the later, never blocking the user.

How do you cache viral posts?

post bodies in and in-process. Spread like counters across sub-keys.

What does a new user see?

Follow suggestions and a "popular" feed, until their own graph fills a feed.

How do you handle inactive users?

Skip them in . On login, rebuild their feed once by pulling, then switch back to push.

How do you shard the feed cache?

By user id, so one read touches one . Workers batch writes per node.

An author goes from 90K to 110K followers. What changes?

New posts are pulled, not pushed. Old pushed posts stay in feeds, so the merge by post id.

Check yourself

Question 1 of 3

Score 2 out of 3 or better to mark this page complete.

1. Why skip fan-out on write for celebrities?

Patterns behind this

The coding patterns this design is built on. Practise them and the design questions get easier.