Design A News-Feed System
System design for a scalable news feed.

On this page
- Problem statement
- Requirements and scope
- Functional requirements
- Non-functional requirements
- Back-of-the-envelope estimation
- Core design decisions
- Fan-out on Write & Fan-out on Read (Hybrid model)
- Chronological ordering
- Write path
- Read path
- Database choices
- CDN for media
- Load balancing
- Feed pagination
- Kafka scalability
- Data retention in Redis
- Auth and rate limiting
- Search
- Bottlenecks and trade-offs
Problem statement
Design a news feed system similar to Facebook or Inshorts, where users can publish posts and see a personalized, chronological feed of content from people they follow.
Requirements and scope
Functional requirements
- Users can create a post:
POST /api/v1/newPost - Users can fetch their feed:
GET /api/v1/posts - Feed is ordered chronologically
Non-functional requirements
- 10 million DAU
- Highly available and low-latency feed reads
- Eventual consistency is acceptable for feed delivery
Back-of-the-envelope estimation
| Metric | Value |
|---|---|
| DAU | 10M |
| Read : Write ratio | 10 : 1 |
| Avg post size | 200 KB (including its media) |
| Write throughput | 60 writes/sec |
| Read throughput | 580 reads/sec |
| New data written/day | ~1 TB/day |
| Read data served/sec | ~2.3 GB/sec |
Where those numbers come from:
Writes: 60 posts/sec × 86,400 sec ≈ 5.2M posts/day (about 1 post per 2 users)
Storage: 5.2M posts × 200 KB ≈ 1 TB/day
Reads: 580 feed loads/sec × ~20 posts per page × 200 KB ≈ 2.3 GB/sec
Almost all of that 200 KB is media, which lives in object storage behind a CDN (covered below). The post record itself, text plus media URLs, is only around a kilobyte.
The high read throughput makes caching non-negotiable.
Core design decisions
Fan-out on Write & Fan-out on Read (Hybrid model)
Fan-out on Write (Push Model)
When User A makes a post, you immediately push that post to the feed of all of User A's followers.
So if A has 1000 followers, you write that post to 1000 users' pre-built feed cache at write time.
Result: When a follower opens their feed, it's already ready, just read from cache. Super fast reads.
Problem: What if a celebrity (100M followers) posts something? 100M cache writes in seconds. This is called the Hotkey / Celebrity problem.
Fan-out on Read (Pull Model)
When a user opens their feed, you go and fetch posts from everyone they follow at that moment and assemble the feed.
Result: Reads are slower since you're computing on the fly, but writes are cheap.
Problem: At 580 reads/sec, with each read pulling posts from maybe 500 accounts the user follows, that's a massive DB fan-out on every request.
The Safe Spot (Hybrid)
Most real systems like Facebook/Twitter use a hybrid approach:
- Regular users -> Fan-out on Write
- Celebrities / High follower accounts -> Fan-out on Read
So a follower's feed is the pre-built list pushed by regular accounts, plus the latest posts from the few celebrities they follow, pulled and merged in at read time.
Fan-out on write Fan-out on read
Work happens When a post is made When a feed is opened
Reads Fast, already built Slow, assembled on the fly
Writes Expensive for big Cheap
follower counts
Best for Regular users Celebrities
Chronological ordering
Feeds are stored in Redis as sorted sets, keyed by UserFeed:{userId}, with the timestamp as the score. No ranking algorithm needed for a news-focused product.
ZADD UserFeed:42 1716540000 post:9001 # fan-out worker adds a post
ZREVRANGE UserFeed:42 0 19 # newest 20 postIds for the feed
Write path
- Client calls
POST /api/v1/newPost-> hits the API Gateway - Gateway routes to the Post service, which writes the post to Post DB (NoSQL) and Post Cache (Redis)
- Post service publishes an event to Kafka
- Fan-out workers consume from Kafka asynchronously:
- Query Graph DB to fetch all followers of the author
- For each follower, push the new postId into their Redis feed sorted set
- Notification service also consumes from Kafka and sends push alerts
Read path
- Client calls
GET /api/v1/posts-> hits the API Gateway - Gateway routes to the Get service
- Get service fetches the user's pre-built feed from Feed Cache (Redis) i.e. a list of postIds, and merges in recent posts from any celebrities the user follows
- On cache hit: hydrate each postId with full post content from Post Cache
- On cache miss: rebuild the feed from Post DB, by pulling recent posts from the accounts the user follows, and backfill the cache
flowchart LR
C[Client] --> GW[API Gateway]
GW -->|POST newPost| PS[Post service]
PS --> PDB[(Post DB)]
PS --> PC[(Post Cache)]
PS --> K[[Kafka]]
K --> FW[Fan-out workers]
FW --> G[(Graph DB)]
FW --> FC[(Feed Cache)]
K --> NS[Notification service]
GW -->|GET posts| GS[Get service]
GS --> FC
GS --> PC
Database choices
| Store | Technology | Why |
|---|---|---|
| Post DB | NoSQL (e.g. Cassandra) | High write throughput, flexible schema |
| Graph DB | Neo4j / adjacency list | Efficient follower/following traversal |
| Feed Cache | Redis sorted set | O(log n) inserts, instant ranked reads |
| Post Cache | Redis | Low-latency post hydration |
CDN for media
At 2.3 GB/sec of read throughput, serving images and videos directly from the Post DB would be catastrophic. All media assets are stored in an object store (e.g. S3) and served via a CDN like CloudFront. The post record in the DB stores only the media URL, not the binary content itself. Cache-Control headers ensure images are served from edge nodes closest to the user, drastically reducing origin load and latency.
Load balancing
A load balancer sits between the API Gateway and each request-serving microservice, the Post service and the Get service. This ensures no single instance becomes a bottleneck and allows horizontal scaling. Health checks at the load balancer layer handle instance failures transparently. Fan-out workers and the Notification service don't need one, since they pull their work from Kafka rather than receiving requests, and the consumer group spreads the load across them (see below).
Feed pagination
The GET /api/v1/posts endpoint cannot return all posts in a single response. Cursor-based pagination is used instead of offset-based, because offsets become expensive as the feed grows. Each response returns a nextCursor (the timestamp of the last post returned). The next request passes this cursor, and the Get service fetches the next N postIds from Redis starting from that score. This keeps pagination O(log n) regardless of feed size.
# next page: 20 posts older than the cursor (the "(" makes the bound exclusive)
ZREVRANGEBYSCORE UserFeed:42 (1716540000 -inf LIMIT 0 20
One detail to watch: two posts can share the same timestamp, and an exclusive bound would then skip the second one. So in practice the cursor also carries the last postId (or the score gets a unique tiebreaker added to it), so a page boundary never skips or repeats a post.
Kafka scalability
Kafka topics are partitioned by userId of the post author. This ensures all events from the same user land on the same partition, preserving ordering. Fan-out worker pods are part of a consumer group, each pod owns one or more partitions and scales horizontally as write throughput grows. Partition count can be increased without downtime as the system scales, with one catch: adding partitions changes which partition a given userId maps to, so ordering is only guaranteed within the old layout and the new one, not across the switch. Many teams create more partitions than they need up front to avoid this.
Data retention in Redis
Redis memory is finite. Each user's feed sorted set is capped at the most recent 1000 postIds. When a fan-out worker pushes a new postId, it runs a ZREMRANGEBYRANK to trim entries beyond the cap.
ZREMRANGEBYRANK UserFeed:42 0 -1001 # keep only the newest 1000
Requests for older posts fall through to the Post DB, which acts as the source of truth: the older part of the feed is rebuilt from the posts of the accounts the user follows, the same way as a cache miss.
Auth and rate limiting
Authentication happens at the API Gateway layer using JWT tokens. Every request is validated before reaching any downstream service. Rate limiting is also enforced at the gateway, write endpoints like POST /api/v1/newPost are throttled per user to prevent abuse and protect Kafka from ingest spikes. Read endpoints have a more generous limit but are still bounded to protect the Get service.
Search
Feed reads and post creation do not serve search. A separate Elasticsearch cluster is maintained for this purpose. Whenever a post is written to the Post DB, a change-data-capture (CDC) pipeline syncs it to Elasticsearch asynchronously. Users searching for posts, topics, or keywords hit a dedicated Search service backed by Elasticsearch, completely decoupled from the feed read path. (The same derived read model idea from the Microservices series.)
Bottlenecks and trade-offs
- The celebrity problem: If a user has millions of followers, fan-out on write turns one post into millions of cache writes at once. Since this is a news platform with limited celebrity accounts, this is less severe, and the hybrid model above (fan-out on read for high-follower accounts) handles the ones that do exist.
- Feed staleness: Because fan-out is async via Kafka, there is a small delay between a post being published and it appearing in followers' feeds. This is acceptable for a news product.
- Storage: Storing postId lists per user in Redis rather than full post content keeps memory usage manageable. Full content is only fetched during feed hydration.