HLD · 7 min read

Design A News-Feed System

System design for a scalable news feed.

On this page
  1. Problem statement
  2. Requirements and scope
  3. Functional requirements
  4. Non-functional requirements
  5. Back-of-the-envelope estimation
  6. Core design decisions
  7. Fan-out on Write & Fan-out on Read (Hybrid model)
  8. Chronological ordering
  9. Write path
  10. Read path
  11. Database choices
  12. CDN for media
  13. Load balancing
  14. Feed pagination
  15. Kafka scalability
  16. Data retention in Redis
  17. Auth and rate limiting
  18. Search
  19. Bottlenecks and trade-offs

Problem statement

Design a news feed system similar to Facebook or Inshorts, where users can publish posts and see a personalized, chronological feed of content from people they follow.

Requirements and scope

Functional requirements

  • Users can create a post: POST /api/v1/newPost
  • Users can fetch their feed: GET /api/v1/posts
  • Feed is ordered chronologically

Non-functional requirements

  • 10 million DAU
  • Highly available and low-latency feed reads
  • Eventual consistency is acceptable for feed delivery

Back-of-the-envelope estimation

MetricValue
DAU10M
Read : Write ratio10 : 1
Avg post size200 KB (including its media)
Write throughput60 writes/sec
Read throughput580 reads/sec
New data written/day~1 TB/day
Read data served/sec~2.3 GB/sec

Where those numbers come from:

text
Writes:  60 posts/sec × 86,400 sec  ≈ 5.2M posts/day  (about 1 post per 2 users)
Storage: 5.2M posts × 200 KB        ≈ 1 TB/day
Reads:   580 feed loads/sec × ~20 posts per page × 200 KB ≈ 2.3 GB/sec

Almost all of that 200 KB is media, which lives in object storage behind a CDN (covered below). The post record itself, text plus media URLs, is only around a kilobyte.

The high read throughput makes caching non-negotiable.

Core design decisions

Fan-out on Write & Fan-out on Read (Hybrid model)

Fan-out on Write (Push Model)

When User A makes a post, you immediately push that post to the feed of all of User A's followers.

So if A has 1000 followers, you write that post to 1000 users' pre-built feed cache at write time.

Result: When a follower opens their feed, it's already ready, just read from cache. Super fast reads.

Problem: What if a celebrity (100M followers) posts something? 100M cache writes in seconds. This is called the Hotkey / Celebrity problem.

Fan-out on Read (Pull Model)

When a user opens their feed, you go and fetch posts from everyone they follow at that moment and assemble the feed.

Result: Reads are slower since you're computing on the fly, but writes are cheap.

Problem: At 580 reads/sec, with each read pulling posts from maybe 500 accounts the user follows, that's a massive DB fan-out on every request.

The Safe Spot (Hybrid)

Most real systems like Facebook/Twitter use a hybrid approach:

  • Regular users -> Fan-out on Write
  • Celebrities / High follower accounts -> Fan-out on Read

So a follower's feed is the pre-built list pushed by regular accounts, plus the latest posts from the few celebrities they follow, pulled and merged in at read time.

text
                 Fan-out on write          Fan-out on read
Work happens     When a post is made       When a feed is opened
Reads            Fast, already built       Slow, assembled on the fly
Writes           Expensive for big         Cheap
                 follower counts
Best for         Regular users             Celebrities

Chronological ordering

Feeds are stored in Redis as sorted sets, keyed by UserFeed:{userId}, with the timestamp as the score. No ranking algorithm needed for a news-focused product.

redis
ZADD UserFeed:42 1716540000 post:9001        # fan-out worker adds a post
ZREVRANGE UserFeed:42 0 19                   # newest 20 postIds for the feed

Write path

  1. Client calls POST /api/v1/newPost -> hits the API Gateway
  2. Gateway routes to the Post service, which writes the post to Post DB (NoSQL) and Post Cache (Redis)
  3. Post service publishes an event to Kafka
  4. Fan-out workers consume from Kafka asynchronously:
    • Query Graph DB to fetch all followers of the author
    • For each follower, push the new postId into their Redis feed sorted set
  5. Notification service also consumes from Kafka and sends push alerts

Read path

  1. Client calls GET /api/v1/posts -> hits the API Gateway
  2. Gateway routes to the Get service
  3. Get service fetches the user's pre-built feed from Feed Cache (Redis) i.e. a list of postIds, and merges in recent posts from any celebrities the user follows
  4. On cache hit: hydrate each postId with full post content from Post Cache
  5. On cache miss: rebuild the feed from Post DB, by pulling recent posts from the accounts the user follows, and backfill the cache
mermaid
flowchart LR
    C[Client] --> GW[API Gateway]
    GW -->|POST newPost| PS[Post service]
    PS --> PDB[(Post DB)]
    PS --> PC[(Post Cache)]
    PS --> K[[Kafka]]
    K --> FW[Fan-out workers]
    FW --> G[(Graph DB)]
    FW --> FC[(Feed Cache)]
    K --> NS[Notification service]
    GW -->|GET posts| GS[Get service]
    GS --> FC
    GS --> PC

Database choices

StoreTechnologyWhy
Post DBNoSQL (e.g. Cassandra)High write throughput, flexible schema
Graph DBNeo4j / adjacency listEfficient follower/following traversal
Feed CacheRedis sorted setO(log n) inserts, instant ranked reads
Post CacheRedisLow-latency post hydration

CDN for media

At 2.3 GB/sec of read throughput, serving images and videos directly from the Post DB would be catastrophic. All media assets are stored in an object store (e.g. S3) and served via a CDN like CloudFront. The post record in the DB stores only the media URL, not the binary content itself. Cache-Control headers ensure images are served from edge nodes closest to the user, drastically reducing origin load and latency.

Load balancing

A load balancer sits between the API Gateway and each request-serving microservice, the Post service and the Get service. This ensures no single instance becomes a bottleneck and allows horizontal scaling. Health checks at the load balancer layer handle instance failures transparently. Fan-out workers and the Notification service don't need one, since they pull their work from Kafka rather than receiving requests, and the consumer group spreads the load across them (see below).

Feed pagination

The GET /api/v1/posts endpoint cannot return all posts in a single response. Cursor-based pagination is used instead of offset-based, because offsets become expensive as the feed grows. Each response returns a nextCursor (the timestamp of the last post returned). The next request passes this cursor, and the Get service fetches the next N postIds from Redis starting from that score. This keeps pagination O(log n) regardless of feed size.

redis
# next page: 20 posts older than the cursor (the "(" makes the bound exclusive)
ZREVRANGEBYSCORE UserFeed:42 (1716540000 -inf LIMIT 0 20

One detail to watch: two posts can share the same timestamp, and an exclusive bound would then skip the second one. So in practice the cursor also carries the last postId (or the score gets a unique tiebreaker added to it), so a page boundary never skips or repeats a post.

Kafka scalability

Kafka topics are partitioned by userId of the post author. This ensures all events from the same user land on the same partition, preserving ordering. Fan-out worker pods are part of a consumer group, each pod owns one or more partitions and scales horizontally as write throughput grows. Partition count can be increased without downtime as the system scales, with one catch: adding partitions changes which partition a given userId maps to, so ordering is only guaranteed within the old layout and the new one, not across the switch. Many teams create more partitions than they need up front to avoid this.

Data retention in Redis

Redis memory is finite. Each user's feed sorted set is capped at the most recent 1000 postIds. When a fan-out worker pushes a new postId, it runs a ZREMRANGEBYRANK to trim entries beyond the cap.

redis
ZREMRANGEBYRANK UserFeed:42 0 -1001     # keep only the newest 1000

Requests for older posts fall through to the Post DB, which acts as the source of truth: the older part of the feed is rebuilt from the posts of the accounts the user follows, the same way as a cache miss.

Auth and rate limiting

Authentication happens at the API Gateway layer using JWT tokens. Every request is validated before reaching any downstream service. Rate limiting is also enforced at the gateway, write endpoints like POST /api/v1/newPost are throttled per user to prevent abuse and protect Kafka from ingest spikes. Read endpoints have a more generous limit but are still bounded to protect the Get service.

Feed reads and post creation do not serve search. A separate Elasticsearch cluster is maintained for this purpose. Whenever a post is written to the Post DB, a change-data-capture (CDC) pipeline syncs it to Elasticsearch asynchronously. Users searching for posts, topics, or keywords hit a dedicated Search service backed by Elasticsearch, completely decoupled from the feed read path. (The same derived read model idea from the Microservices series.)

Bottlenecks and trade-offs

  • The celebrity problem: If a user has millions of followers, fan-out on write turns one post into millions of cache writes at once. Since this is a news platform with limited celebrity accounts, this is less severe, and the hybrid model above (fan-out on read for high-follower accounts) handles the ones that do exist.
  • Feed staleness: Because fan-out is async via Kafka, there is a small delay between a post being published and it appearing in followers' feeds. This is acceptable for a news product.
  • Storage: Storing postId lists per user in Redis rather than full post content keeps memory usage manageable. Full content is only fetched during feed hydration.