Design a News Feed (Twitter or Instagram)
The key decision is fan-out on write versus fan-out on read, and how a React UI consumes the result.
- 580 to 116K/sposts per second become fan-out writes
- ~100K followerscelebrity threshold: pull instead of push
- 800 idscap on each user's cached feed
Show what to say
“I'd use hybrid fan-out. A normal user's post is pushed into each active follower's capped Redis feed, so a read is one lookup. Celebrities, over about 100K followers, are pulled and merged at read time, so one post never costs millions of writes. Reads hydrate ids through a post cache and page with a score cursor.”
Trade-offs
- Push vs pull: push makes reads one lookup but write cost grows with followers; pull writes once but every read merges timelines.
- Cache every feed vs active users only: no rebuilds, but terabytes of RAM for people who never look.
- Lower vs higher celebrity threshold: less write amplification, but more read-time merging.
Bring it up before they ask
Raise the celebrity problem yourself: pure push breaks for a 30M-follower account, since at 100K writes per second one post takes 5 minutes to reach everyone.
New to the words? Read the one-minute primer
Hybrid : copy a normal user's post into each follower's ready-made feed, so reading is one lookup. For celebrities (over ~100K followers), skip the copy and merge their posts in when the feed opens.
Functional
- Follow and unfollow
- Post text and media
- A home feed, ranked or reverse-chronological
- Like and comment
- New posts appear without a manual refresh
Non-functional
- Feed p99 under 200 ms
- Followers see new posts within seconds
- Celebrities with millions of followers
- 300M daily active users
- Available over consistent: a feed a few seconds stale is fine
Out of scope
- Ads and ad ranking
- Search and hashtags
- Direct messages
Scope and scale
Ask:
Read more · 138 words
- Ranked or chronological? Ranking adds a scoring step.
- Follow graph? Average follows, biggest follower count, a follow limit?
- Freshness? Seconds, or is a minute fine?
- Celebrities? Millions of followers? Yes: this is the crux.
- Scale? Daily users, feed opens, posts per day.
| Quantity | Calculation | Result |
|---|---|---|
| Feed reads | 300M × 10/day | ≈ 35K/s (~100K/s peak) |
| New posts | 50M/day | ≈ 580/s |
| Fan-out writes | 50M posts × 200 average followers | ≈ 116K/s |
| Cached feed memory | 300M × 800 ids × ~40 B | ≈ 9.6 TB |
Reads outnumber posts 60 to 1, so do the work at write time. The bill: 580 posts/s become 116K writes/s, and 9.6 TB of RAM. So only active users' feeds.
API and data model
Read more · 181 words
| Method | Path | Notes |
|---|---|---|
| POST | /posts | Creates a post; fan-out runs asynchronously |
| GET | /feed?cursor=&limit=20 | { posts, nextCursor } |
| POST/DELETE | /users/:id/follow | Follow or unfollow |
| POST | /posts/:id/like | Optimistic on the client |
| GET | /feed/new-count?after=TS | For the "new posts" banner |
nextCursor is the last post's timestamp; null means nothing older exists.
Posts and follows live in a durable store. Ids are time-ordered.
Feeds are derived data in :
| Key | Type | Contents |
|---|---|---|
feed:USER_ID | sorted set | score = createdAt ms, member = post id; newest 800 |
posts:AUTHOR_ID | sorted set | The author's own timeline, for the celebrity merge |
post:ID | string (JSON) | Post body cache, 1 h TTL |
A time-scored makes the score cursor work: "20 posts older than T" stays correct as new posts arrive.
{
"posts": [
{ "id": "p_981", "author": { "id": "u_7", "name": "Maya" }, "text": "Sunset", "media": ["img/981.jpg"], "likes": 12, "liked": false, "createdAt": 1760000000000 }
],
"nextCursor": 1759998000000
}CREATE TABLE posts (
id bigint PRIMARY KEY,
author_id bigint NOT NULL,
body text,
media jsonb,
created_at timestamptz NOT NULL DEFAULT now(),
deleted boolean NOT NULL DEFAULT false
);
CREATE INDEX posts_by_author ON posts (author_id, created_at DESC);
CREATE TABLE follows (
follower_id bigint NOT NULL,
followee_id bigint NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (follower_id, followee_id)
);
CREATE INDEX follows_by_followee ON follows (followee_id, follower_id); -- "who follows X?" for fan-outArchitecture
Each user's feed is a ready-made list of post ids, kept in fast memory.
Read more · 51 words
Writes store the post, queue the copying and return, so the author never waits. Reads touch only . Play Normal user posts, Celebrity posts and User opens feed.
Tap any box to see what it is and what it talks to.
- Client
- Service
- Queue
- Worker
- Database
- Cache
- CDN
Text description
- Maya, who has 300 followers, posts a photo. POST /posts reaches the post service.
- The post is written to the Posts DB, the source of truth.
- Its id is also added to Maya's own author timeline, posts:maya. Every author has one, capped at 800 ids.
- Maya is under the 100K-follower threshold, so fan-out jobs are enqueued: one message per 1,000 followers, so just one here.
- Maya gets 201 Created right away and sees her own post. Delivery to followers continues in the background.
- A fan-out worker takes the job and drops followers who haven't opened the app in 30 days.
- For each active follower: ZADD feed:ID with the post's timestamp, then trim to the newest 800. Pipelined, a batch of 1,000 takes milliseconds.
- Within a second or two the post sits at the top of every active follower's cached feed.
Push, pull or hybrid
The whole question turns on this. Compare the two pure strategies first.
Read more · 131 words
Push copies each post into every follower's feed, so reads are one lookup. Pull stores a post once, but every feed open merges many timelines. The hybrid pushes for most authors and pulls only the few celebrities each user follows.
Do the sum first. Say the workers manage about 100K feed writes a second:
| Author | Followers | Push takes |
|---|---|---|
| Typical user | 200 | Instant |
| Popular account | 100K | About 1 second |
| Celebrity | 30M | About 5 minutes |
Push the first two. Pull the third: fans see the post when they next open the feed. A user who follows 5 celebrities merges only 5 extra timelines. An app where nobody has many followers can skip and just pull.
Tap any box to see what it is and what it talks to.
- Client
- Service
- Cache
Text description
- An author with 200 followers posts.
- Fan-out on write: the post id is written into all 200 follower feeds. 200 writes for one post, done in the background.
- Later, Ben opens his feed.
- One read: the newest 20 ids from feed:ben. His feed is already assembled.
- The ids come back in about a millisecond.
- Cheap, fast reads; expensive writes. An author with 30M followers would need 30M writes for every post.
Step-by-step answer
Clarify
About 60 reads per post justifies work at write time.
Push: fan-out on write
Write each post id into every follower's feed. Reads are one lookup, but 10M followers means 10M writes.
Pull: fan-out on read
Merge timelines at read time. Writes are cheap, but reads are too slow at 35K/s.
Hybrid: push for most, pull for celebrities
Tune the ~100K threshold on write cost versus read cost. Users follow few celebrities, so the merge stays small.
Feed storage
A per user, scored by time, capped at 800. Skip users inactive for 30 days.
redis.zadd(`feed:${userId}`, ts, postId); redis.zremrangebyrank(`feed:${userId}`, 0, -801); // keep the newest 800Read path: score cursor, merge, hydrate
Read a page by score cursor, plus the same window from each followed celebrity. Merge, , sort, then fetch post bodies and drop deleted ones.
const max = cursor ? `(${cursor}` : '+inf'; // "(" = strictly older than the cursor redis.zrevrangebyscore(`feed:${userId}`, max, '-inf', 'WITHSCORES', 'LIMIT', 0, limit);Ranking
Start chronological, then score candidates:
affinity × (likes + 2·comments + 3·shares + 1) / (ageHours + 2)^1.5. A learned model can replace the formula later.Freshness
Poll
/feed/new-countevery 30 to 60 s and show a banner. Never insert above the reader.Unfollow and delete
Filter at read time, then clean up asynchronously. Never block the user on cleanup.
Client
A virtualized list (see Build an Infinite Scroll Feed) and optimistic likes that roll back on error.
Read more · 29 words
Implementation
Server tabs: , the read path, helpers.
Read more · 139 words
Client tabs: an optimistic like.
feed.js fixes two classic bugs: offsets that repeated posts (now a score cursor), and no celebrity merge (now mergeTimelines).
Then build this design in the playground and check it against the rubric:
- post service → queue → fan-out workers → feed ;
- celebrity timelines readable by the feed service;
- a post cache and a database with ;
- a for media.
See Caching and Async messaging.
// Write path with hybrid fan-out
const CELEBRITY_THRESHOLD = 100_000;
export async function publishPost(post) { // { id, authorId, createdAt, … }
await db.posts.insert(post);
await redis.zadd(`posts:${post.authorId}`, post.createdAt, post.id); // author timeline (used for pull)
await redis.zremrangebyrank(`posts:${post.authorId}`, 0, -801);
if ((await graph.followerCount(post.authorId)) > CELEBRITY_THRESHOLD) return; // merged at read time instead
for await (const batch of graph.followersInBatches(post.authorId, 1_000)) {
await queue.publish('fanout', { postId: post.id, ts: post.createdAt, to: batch }); // 1 message per 1,000 followers
}
}
// Fan-out worker. With ioredis enableAutoPipelining, these calls are batched per node (works with Redis Cluster).
export async function fanoutWorker({ postId, ts, to }) {
const active = await activeSince(to, 30); // skip users inactive for 30 days
await Promise.all(active.flatMap(userId => [
redis.zadd(`feed:${userId}`, ts, postId),
redis.zremrangebyrank(`feed:${userId}`, 0, -801), // keep the newest 800
]));
}// Optimistic like with rollback
import { useMutation, useQueryClient } from '@tanstack/react-query';
import { likeInFeed } from './optimistic-like.js';
export function useLike() {
const qc = useQueryClient();
return useMutation({
mutationFn: postId => fetch(`/api/posts/${postId}/like`, { method: 'POST' }).then(r => { if (!r.ok) throw new Error('like failed'); }),
onMutate: async postId => {
await qc.cancelQueries({ queryKey: ['feed'] }); // an in-flight refetch must not overwrite the optimistic state
const previous = qc.getQueryData(['feed']);
qc.setQueryData(['feed'], data => likeInFeed(data, postId));
return { previous };
},
onError: (_err, _postId, ctx) => qc.setQueryData(['feed'], ctx?.previous), // roll back
// No blanket invalidate: refetching an infinite feed reloads every page. Reconcile the one post if needed.
});
}Failure modes and trade-offs
Read more · 141 words
| What breaks | Effect | Mitigation |
|---|---|---|
| Fan-out queue backlog | Posts arrive late | Autoscale workers; authors see their own post at once |
| A worker dies mid-batch | Some followers miss the post | Redeliver; ZADD is idempotent |
| A Redis feed is lost | Empty feed | Rebuild by pulling |
| A celebrity's timeline is hammered | A hot key | Short in-process cache; replicate the key |
| A post is deleted after fan-out | A ghost post | Filter at read time |
| Unfollow | Old posts linger | Follow check at read time |
Choices to defend:
| Choice | Gain | Cost |
|---|---|---|
| Ranked, not chronological | Better engagement | Ranking compute |
| Lower celebrity threshold | Less write cost | More read-time merging |
| Cache every feed | No rebuilds | Terabytes of RAM for idle users |
| SSE, not polling, for banners | Instant | A connection per user |
Follow-up questions
How would you rank posts?
Score pushed and pulled candidates, then mix so one author doesn't fill the page.
How do you handle unfollows and deleted posts?
Filter at read time, then clean the later, never blocking the user.
How do you cache viral posts?
post bodies in and in-process. Spread like counters across sub-keys.
What does a new user see?
Follow suggestions and a "popular" feed, until their own graph fills a feed.
How do you handle inactive users?
Skip them in . On login, rebuild their feed once by pulling, then switch back to push.
How do you shard the feed cache?
By user id, so one read touches one . Workers batch writes per node.
An author goes from 90K to 110K followers. What changes?
New posts are pulled, not pushed. Old pushed posts stay in feeds, so the merge by post id.
Check yourself
Question 1 of 3Score 2 out of 3 or better to mark this page complete.
Patterns behind this
The coding patterns this design is built on. Practise them and the design questions get easier.