Design a News Feed: Fan-Out, Ranking and the Celebrity Problem

Open any social app and the first thing you see is a feed: recent posts from the people you follow, possibly reordered by what the app thinks you will care about. Pixly's feed was mentioned in almost every earlier article as "the hard read path". Here we finally design it.

The core difficulty is a fan-in, fan-out problem. A feed read needs posts from hundreds of accounts. A post must appear in the feeds of everyone who follows its author, which for most authors is a few hundred people and for a few celebrities is many millions. Every design choice in this case study is about where to pay that cost: when someone posts, or when someone reads.

• How to estimate posting, reading and fan-out.
• Fan-out on read and fan-out on write, and where each breaks.
• The hybrid model for celebrity accounts.
• Storing precomputed feeds, and hydrating them at read time.
• Ranking, with a fallback to chronological order.
• Pagination, deletions, unfollows and privacy.
• What changes at social-media scale.

Users follow other accounts (one-way, like most photo and microblogging apps). The home feed shows posts from followed accounts, newest or most relevant first, with infinite scrolling. Posts contain text and photos (photo files are already handled by object storage and a CDN, article 10).

Functional requirements: create posts; follow and unfollow; read the home feed with pagination; show like and comment counts on each item.

Non-functional requirements

We start with Pixly's numbers from articles 06 and 07, then scale up in section 8.

10 million DAU; 50 million registeredFeed reads: 10 per DAU per day -> 100 million/day, about 1,160/s average, about 3,500/s peakPosts: 2 million/day -> about 23/s average, about 70/s peakAverage followers per posting account: 200 (median much lower; a long tail of large accounts)Fan-out on write (if every post is pushed to every follower):  2 million posts x 200 followers = 400 million feed insertions/day  -> about 4,600/s average, about 14,000/s peakCelebrity: one account with 10 million followers posts once  -> 10 million insertions from a single postFeed storage (precomputed): keep the latest 500 post ids per active user  10 million active users x 500 ids x 8 bytes = 40 GB (about 80 GB with structure overhead)

Conclusions: reads (3,500 per second) need to be fast; fan-out on write is affordable on average (14,000 inserts per second at peak) but the celebrity tail is not; feed storage fits comfortably in an in-memory store.

GET /v1/feed?cursor=...&limit=20200 {  "items": [    {"post_id": "p_8812", "author": {"id": 77, "name": "Meera", "avatar_url": "..."},     "caption": "Monsoon in Munnar", "media": [...], "like_count": 1203, "comment_count": 48,     "liked_by_me": false, "created_at": "..."}  ],  "next_cursor": "opaque"}POST /v1/posts                          (idempotency key; returns 201 with post_id)PUT  /v1/users/{id}/followers/me        (follow, idempotent)DELETE /v1/users/{id}/followers/me      (unfollow)DELETE /v1/posts/{id}                   (author only)

The cursor is opaque (article 04). For a ranked feed, it identifies a position in a snapshot of the ranked list made for this browsing session, so scrolling does not skip or repeat items as new posts arrive.

posts        (post_id PK, time-ordered id; author_id; caption; media keys; visibility; created_at; deleted_at)             index (author_id, created_at DESC)            -> "this author's recent posts"follows      (follower_id, followee_id, created_at)        PK (follower_id, followee_id)followers    (followee_id, follower_id)                     reverse index for fan-outuser_flags   (user_id, is_celebrity, last_active_at, is_private)Feed store (precomputed), one entry per user:  key: feed:{user_id}  value: sorted set of post_id with score = created_at (or a ranking score)  capped at 500 entries; expires if the user is inactive for 30 daysCounts: like and comment counters per post (cached, updated asynchronously, article 09)

A sorted set per user is a natural fit for a feed of ids ordered by time or score: Redis sorted sets keep members ordered by score and support range reads, insertions and trimming efficiently (Redis sorted sets). The precomputed feed is a materialised view (Microsoft's Materialized View pattern describes such precomputed, query-optimised views and their eventual consistency (Materialized View pattern)), so it can always be rebuilt from posts and follows.

Partition keys: posts by post_id (and the author index by author_id); follows by follower_id; followers by followee_id; the feed store by user_id.

The simplest design builds the feed when it is requested:

GET /v1/feed:  1. followees = SELECT followee_id FROM follows WHERE follower_id = me        (say 200)  2. for each followee: recent posts by author (from the author index or cache)  3. merge by time (or rank), take the top 20  4. hydrate: post details, authors, counts

It is always fresh, wastes no work on inactive users, and handles celebrities trivially (their posts are read like anyone else's). This is fan-out on read (also called the pull model).

At 3,500 feed reads per second with 200 followees each, step 2 alone is 700,000 author-timeline reads per second, and a user following 5,000 accounts makes a single feed request touch 5,000 timelines. Latency suffers (tail latency over many parallel reads, article 02) and the post store carries enormous read load for what is the same work repeated: the same popular authors' timelines read over and over.

The alternative, fan-out on write (the push model), moves the work to posting time: when a post is created, insert its id into every follower's precomputed feed. A feed read becomes one range read of feed:{me}, plus hydration. Reads are fast and cheap.

But fan-out on write breaks at the other end:

• Celebrities: one post from a 10-million-follower account creates 10 million inserts. At a few hundred thousand inserts per second it takes many seconds to minutes, delays everyone else's fan-out, and repeats with every post.
• Wasted work: many followers are inactive; pushing to their feeds is wasted.
• Write amplification: storage and network costs scale with followers, not with posts.

Combine the two:

• Normal authors (fewer than a threshold of followers, say 100,000): push each new post id into the feeds of active followers (active in the last 30 days).
• Celebrity authors (above the threshold): do not push. Their posts are pulled at read time.
• At read time: read the precomputed feed (pushed items) and the recent posts of the celebrities this user follows (usually a small number), merge, rank, and return.

REQUEST PATH (read)Client -> Feed service:   1. ZREVRANGE feed:{me} (pushed post ids, newest first)   2. celebrity followees of me (small list, cached) -> their recent post ids (cached per celebrity)   3. merge -> candidates -> rank (or sort by time) -> page of 20   4. hydrate: multi-get post objects, authors, counts, my-likes; filter deleted and not-allowed   5. return page + cursorASYNC PATH (write)Post service: insert post + outbox event "PostCreated" (article 21)  -> Fan-out workers (if author is not a celebrity):       page through followers (active only), ZADD post_id into each feed:{follower},       trim each feed to 500 entries  -> Celebrity timeline cache updated (if author is a celebrity)

Each celebrity's recent posts are cached once and read by millions of feed requests, so pulling them is cheap: it is a hot key served from cache (with local caches for the very hottest, article 09).

• Consume PostCreated events from a stream partitioned by author (article 11).
• Read the author's followers in pages (for example 1,000 at a time) and write to feeds in batches (pipelining many ZADD calls).
• Skip inactive followers; their feed is rebuilt on demand if they return (below).
• Be idempotent: adding the same post id to a sorted set twice is harmless, which makes redelivery safe.
• Scale with lag: at 14,000 inserts per second at peak, a modest pool of workers keeps fan-out lag within seconds.

If a user who was inactive for months opens the app, feed:{me} does not exist. Build it on demand with fan-out on read (the baseline algorithm) for the first request, store the result, and from then on the user receives pushes again. The first load is slower; every later load is fast.

The feed store holds only ids. Hydration turns 20 ids into a page:

• Multi-get post objects from a post cache (article 32), falling back to the database for misses.
• Multi-get author summaries and counts.
• Compute liked_by_me with one batched lookup.
• Filter posts that are deleted, from blocked or private accounts the viewer may no longer see, or otherwise not allowed. Fetch a few extra ids to fill the page after filtering.

Hydration happens on every read, so it is the place to enforce rules that must be current (deletion and privacy), while the precomputed list may be slightly stale.

A ranked feed adds two stages:

1. Candidate generation: the merged ids (pushed feed plus celebrity posts), a few hundred candidates.
2. Scoring: compute a relevance score per candidate from features such as recency, the viewer's past interactions with the author, engagement on the post, and content type, using a model served by a ranking service with a strict latency budget (for example 50 ms).

Keep it robust:

• If the ranking service is slow or down, fall back to chronological order (circuit breaker, article 23). A slightly less relevant feed is far better than an error.
• Precompute expensive features offline (affinity between users, post quality signals) and keep them in a fast feature store.
• Snapshot the ranked list per session (store the ordered ids for the session briefly), so pagination is stable while new posts arrive.

• Deleted posts: marked deleted in posts; hydration filters them immediately. A background job removes their ids from feeds lazily (or never; trimming removes them eventually).
• Unfollow: future posts are no longer pushed. Existing items from that author can be filtered at read time (check the follow relationship for authors on the page) and cleaned up asynchronously.
• Privacy changes and blocks: enforced at hydration, so they take effect immediately even though feeds are precomputed.

Read-your-writes (article 19): when a user posts, insert the id into their own feed synchronously (or merge "my recent posts" into my feed reads), so the author never wonders where their post went, while followers receive it via asynchronous fan-out.

Now suppose 500 million DAU, each reading the feed 15 times a day, and 100 million posts a day:

Feed reads: 7.5 billion/day -> about 87,000/s average, about 260,000/s at peak (factor 3)Posts: 100 million/day -> about 1,160/s average, about 3,500/s peakFan-out (normal authors, average 300 active followers): 3,500 x 300 = about 1 million inserts/s at peakFeed store: 500 million x 500 ids x 8 B = 2 TB (about 4 TB with overhead), sharded by user_id

What changes:

• The feed store is a sharded cluster (article 32) of a few dozen to a hundred nodes, plus replicas; it may move to a disk-backed wide-column store with a cache in front, keeping only the most recent items in memory.
• Fan-out at a million inserts per second needs a large, autoscaled worker fleet reading from a partitioned stream, and an aggressive "active follower" filter.
• The celebrity threshold is tuned by measured cost: pushing to more than N followers costs more than pulling.
• Ranking becomes a major subsystem with its own feature pipelines, model training and serving; candidate generation also pulls in recommended posts from accounts the user does not follow (an extension beyond this design).
• Multi-region: feeds are built and stored per region near users; posts and follow graphs are replicated across regions asynchronously, so a post from another region appears a little later.

Clients -> CDN (media) ; API gateway -> Feed service -> feed store (sharded, by user)                                         |-> celebrity timeline cache                                         |-> ranking service (fallback: chronological)                                         `-> hydration: post cache, user cache, counts cache, likesPost service -> posts DB -> outbox -> stream (partitioned by author)   -> fan-out workers (normal authors -> active followers' feeds)   -> celebrity timeline cache updaterFollow service -> follows/followers stores (sharded by follower and by followee)

• Followers' feeds: eventually consistent; a post appears within seconds (fan-out lag).
• The author's own feed: read-your-writes, synchronously updated.
• Counts: eventually consistent, updated asynchronously.
• Privacy, blocks and deletion: enforced at read time, so effectively immediate even though feed lists are stale.
• Pagination: stable within a session through the ranked snapshot or cursor.

• Visibility checks at hydration for private accounts, followers-only posts and blocked users (object-level authorisation, article 28).
• Spam and abuse: rate-limit posting and following (mass-follow bots), detect spam accounts, and down-rank or remove harmful content.
• Scraping: rate-limit feed reads per user and IP, cap page sizes and pagination depth.
• No leakage through counts or ids: do not reveal private accounts' posts through counts or error messages.

"The feed shows posts from accounts a user follows, ranked, with p99 under 300 ms and new posts visible to followers within about ten seconds. For 10 million daily users that's about 3,500 feed reads a second and 70 posts a second at peak, but posts fan out to an average of 200 followers, and some celebrities have millions.

I use a hybrid model. When a normal author posts, the post service writes the post and an outbox event; fan-out workers insert the post id into each active follower's feed, a capped sorted set of ids in a sharded Redis cluster. Celebrity posts are not pushed; their recent posts are cached once and merged at read time. Inactive users get their feed rebuilt on demand.

A feed read takes the pushed ids, merges celebrity posts, ranks a few hundred candidates with a ranking service that falls back to chronological order, and hydrates the page from post, user and count caches, filtering deleted posts and anything the viewer isn't allowed to see. Authors see their own posts immediately; everything else is eventually consistent. The design scales by sharding feeds by user, scaling fan-out workers on stream lag, and tuning the celebrity threshold by measured cost."

Normal: 1,000,000 posts x 300 = 300 millionCelebrities: 2,000 x 3 posts x 5,000,000 = 30 billionTotal: about 30.3 billion insertions/day (about 350,000/s on average, bursty)

Raw: 500 million x 300 x 16 bytes = 2.4 x 10^12 bytes = 2.4 TBWith 1.5x overhead: 3.6 TBNodes: 3.6 TB / 64 GB = about 57 primaries; with one replica each, about 114 nodes

• Next: Design Search Autocomplete (Typeahead), another read path that depends on precomputation.
• Related: the event-driven read models of article 27 and the caching patterns of article 09 underpin this design.