Fan-out on write vs fan-out on read: designing a social feed
Fan-out on write vs fan-out on read, with the arithmetic: why push wins for ordinary accounts, where a 10M-follower post explodes, and how a hybrid merges both.
A user opens the app and expects a timeline: the newest posts from the few hundred accounts they follow, sorted, in well under a second. Somewhere behind that screen a decision was made about when the work of assembling that list happens. Do it when the post is written, and every post costs one insert per follower. Do it when the timeline is read, and every refresh costs one lookup per followee. Both are correct. Both are also wrong for a different slice of the same user base, which is why every large feed ends up running both.
When does the timeline get assembled?
Pay at post time, or pay at read time
Fan-out on write
push into follower inboxes at post time
1 read = 1 list 200 inserts, one batch, about 20 ms
10,000,000 inserts
10,000 batches of 1,000; ten seconds even with twenty workers
Fan-out on read
pull from every followee at request time
1 read = N lookups ... 492 more, most returning nothing
500 lookups per refresh, nothing stored twice
Fan-out on read: the obvious design
Store posts in a table keyed by author. When a user asks for a timeline, fetch the accounts they follow, fetch the recent posts from each, merge, sort by time, return the top 50. Nothing is duplicated, a deleted post disappears everywhere at once, and an edit is free because there is only one copy.
The cost is on the read. Give the system 10 million daily active users who each refresh ten times a day, and say the average person follows 500 accounts.
timeline reads: 10,000,000 users x 10 refreshes = 100,000,000/day (about 1,160/s)
lookups per read: 500 followees
lookups: 100,000,000 x 500 = 50 billion/day (about 580,000/s)Every one of those lookups is "newest posts by author X since time T", and most return nothing, because most accounts posted nothing in the last hour. You are paying 580,000 queries a second, plus a 500-way merge, to discover that. Read latency scales with how many accounts a user follows, so the most engaged users get the slowest feeds. Caching helps only as much as the followee set repeats across users, which for a personalized feed is not much.
Fan-out on write: pay at post time
Flip it. When a post is created, look up the author's followers and append the post id to each follower's timeline: an inbox per user, materialized in advance. A timeline read becomes one fetch of one list. The 100 million reads a day become 1,160 cheap list reads a second, which a cache cluster barely notices.
The bill moves to the write, and the write amplification factor is exactly the follower count.
posts: 10,000,000/day
average followers: 200
inbox inserts: 10,000,000 x 200 = 2 billion/day (about 23,000/s)
one post, 200 followers: 200 inserts = one batch, about 20 ms
one post, 10M followers: 10,000,000 insertsTwenty-three thousand inserts a second across a sharded cache is ordinary. The last line is not. Ten million inserts in batches of 1,000 is 10,000 batches; at 20 ms per batch that is 200 seconds of work for one worker, and even twenty workers in parallel need ten seconds before the last follower sees the post. Meanwhile those workers are serving nobody else. One account with 10 million followers posting three times an hour generates more inbox writes than a million ordinary users combined.
Fan-out on write turns follower count into write amplification. It is the right trade for almost every account and a catastrophe for the few accounts everyone follows.
This is the celebrity problem, and it has a clean answer once you stop looking for one mechanism.
The hybrid: push for the many, pull for the few
Draw a line at some follower count. Accounts below it are fanned out on write: their posts land in follower inboxes within milliseconds. Accounts above it are never fanned out. Instead, each user's follow list records which followees are high-fan-out, and the read path pulls those accounts' recent posts and merges them into the cached inbox at request time.
The numbers work because the two sets have opposite shapes. Ordinary accounts are numerous but have few followers, so pushing is cheap per post. High-follower accounts are rare, so a user follows a handful at most, and pulling from five authors is a five-way merge, not a 500-way one. The read stays fast and the write never explodes.
The post-created event is a fact several handlers care about (search indexing, notifications, fan-out), so it belongs on a topic rather than a work queue. The fan-out worker is one subscription, consuming in follower batches:
public sealed class FanOutWorker(
ChannelReader<PostCreated> posts,
IFollowerStore followers,
ITimelineCache timelines,
ILogger<FanOutWorker> logger) : BackgroundService
{
private const int BatchSize = 1_000;
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
await foreach (var post in posts.ReadAllAsync(stoppingToken))
{
var started = Stopwatch.GetTimestamp();
await foreach (var batch in followers.GetActiveFollowerBatchesAsync(
post.AuthorId, BatchSize, stoppingToken))
{
await timelines.PushToManyAsync(batch, post.Id, post.CreatedAt, stoppingToken);
}
logger.LogInformation(
"Fanned out post {PostId} in {Elapsed}",
post.Id,
Stopwatch.GetElapsedTime(started));
}
}
}GetActiveFollowerBatchesAsync is doing two jobs. It streams followers in pages, so a
million-follower account is never loaded into memory as one list, and it skips dormant accounts: a
follower who has not opened the app in 30 days does not need a materialized inbox, and rebuilding
one on their return is a single pull. Skipping dormant users is the cheapest optimization in the
whole system. PushToManyAsync pipelines the inserts and caps every inbox:
LPUSH timeline:{userId} {postId}
LTRIM timeline:{userId} 0 799Eight hundred ids at 8 bytes is about 6 KB per user before overhead; call it 10 KB. Ten million active users is 100 GB of inbox across the cache cluster, a real but ordinary bill. Without the dormant-account skip, all 50 million registered users would cost 500 GB. Redis documents LTRIM for exactly this capped-list pattern, and the caching strategies guide covers what happens when a node holding those lists disappears.
The read path merges the inbox with the pulled high-fan-out posts:
public async Task<IReadOnlyList<long>> ReadTimelineAsync(
long userId,
int take,
CancellationToken cancellationToken)
{
var inbox = await _timelines.GetNewestAsync(userId, take, cancellationToken);
var celebrities = await _follows.GetHighFanOutFolloweesAsync(userId, cancellationToken);
if (celebrities.Count == 0)
{
return inbox.Select(entry => entry.PostId).ToList();
}
var pulled = await _posts.GetRecentByAuthorsAsync(celebrities, take, cancellationToken);
return inbox
.Concat(pulled)
.DistinctBy(entry => entry.PostId)
.OrderByDescending(entry => entry.CreatedAt)
.Take(take)
.Select(entry => entry.PostId)
.ToList();
}The inbox holds ids, not content, so a final step hydrates the 50 winning ids from a post cache.
That is also what makes edits free. Paging further back uses the last (CreatedAt, PostId) pair as
a cursor; offset pagination on a merged, moving list returns
duplicates and skips posts.
Ranking is a different layer
Everything above is retrieval: which few hundred post ids are candidates for this user right now. Ranking (recency, engagement, whatever the product decides) runs after retrieval, over those candidates only. Keeping them separate means the fan-out machinery never changes when the ranking model does, and the ranking service never has to know how the ids arrived.
The chores nobody draws on the whiteboard
Materialized inboxes are copies, and copies drift. Deleted posts are filtered at read time against a small tombstone set rather than chased through millions of inboxes. Unfollows either remove that author's ids from one inbox (cheap, one user) or become a read-time filter until the inbox rolls over naturally. A user who follows a new account gets a one-off backfill pull. The fan-out queue needs the usual care: at-least-once delivery means an inbox can receive the same id twice, which is why the read path deduplicates, and queue-based load leveling is what keeps a burst of posts from becoming a burst of cache writes.
Measure four things: fan-out lag (post created to present in the last follower's inbox, p99), timeline read p99, cache memory per active user, and how often a read falls through to the database. When fan-out lag climbs, the high-fan-out threshold is too high or the workers are too few. When memory climbs, the dormant-account cutoff is too generous.
Practicing the trade-off
You can read about the celebrity problem in one sitting. What sticks is drawing the push design, watching it fall over when one account has ten million followers, and then fixing it. Katabench's System Design Studio has a Social Feed build course that goes exactly there: publish the first post, fan out home timelines, handle celebrity fan-out, then moderate and operate the feed. The first three of those challenges are free with an account. Deterministic rules check fan-out and required paths, and a capacity simulation reports p99 latency and which component saturates first under a modeled traffic scenario, so the ten-million-follower post becomes a number on a report instead of a war story. How it works explains the grading model behind it.