HLD · 17 min read

Design Reddit

System design for a forum/link-aggregation platform like Reddit.

On this page
  1. 1. Requirements
  2. Functional Requirements
  3. Out of Scope
  4. Non-Functional Requirements
  5. 2. Capacity Estimation
  6. 3. API Gateway
  7. 4. Post Service and Upload Pipeline
  8. Upload Flow
  9. Data Model
  10. Bottlenecks in the Post Service
  11. 5. Voting Service
  12. Full Vote Flow
  13. Votes Table (PostgreSQL)
  14. Bottlenecks in the Voting Service
  15. 6. Feed Service
  16. Redis Sorted Sets per Subreddit per Sorting Mode
  17. Fan-out Strategy
  18. Home Feed Read Flow
  19. Subreddit Feed Read Flow
  20. Bottlenecks in the Feed Service
  21. 7. Comment Service
  22. Adjacency List Data Model
  23. Lazy Loading
  24. Comment Vote Flow
  25. Bottlenecks in the Comment Service
  26. 8. Search Service
  27. Storage Engine - Elasticsearch
  28. Elasticsearch Documents
  29. Keeping Elasticsearch in Sync
  30. Bottlenecks in Search
  31. 9. Full Data Flow Summary
  32. 10. Resilience and Fault Tolerance
  33. 11. Technology Choices Summary

1. Requirements

Functional Requirements

  • Create posts (text, image, video)
  • Upvote and downvote posts and comments
  • Threaded nested comments
  • Join subreddits and follow users
  • Home feed based on joined subreddits
  • Subreddit feed with sorting modes: Hot, New, Top, Controversial
  • Search for posts, subreddits, and users
  • Create a subreddit

Out of Scope

  • Chat (same WebSocket architecture as Tinder)
  • Reddit Awards and coins
  • Moderation tools
  • Reddit Ads

Non-Functional Requirements

  • High availability across feed, voting, and media serving
  • Low latency feed reads under 100ms
  • Votes must never be lost even under traffic spikes
  • Hotness scores must reflect recent activity with time decay
  • Eventual consistency acceptable for vote counts and feed ranking
  • Strong consistency required for vote idempotency (a user cannot vote twice on the same post)

2. Capacity Estimation

MetricValue
DAU50 million
Posts per day10 million (20% of DAU)
Average post size~2 MB weighted average across text (a few KB), images (500 KB to 1 MB), and videos (up to 10 MB)
Post storage per day10M × 2 MB = 20 TB/day
Votes per day50M DAU × 5 votes each = 250 million votes/day = ~2,900 votes/second on average, several times that at peak, with viral spikes far higher
Comments per dayRoughly 100 million (2 comments per DAU on average)

The dominant write load is votes at around 3,000 per second, not posts. Every vote potentially triggers a hotness score recomputation. This is the primary design constraint for the voting and feed systems. Media storage at 20 TB/day is significant but manageable with S3 and CDN caching.

3. API Gateway

All client requests enter through the API Gateway which handles authentication, rate limiting, and routing. The same constraint applies as in every previous design: the API Gateway is never in the path of binary media payloads. Image and video uploads bypass the Gateway after authentication using a presigned S3 URL for direct client-to-S3 upload.

4. Post Service and Upload Pipeline

Upload Flow

When a user creates a post with media:

  1. Client authenticates via API Gateway and requests a presigned S3 URL from the Post Service
  2. Client uploads media directly to S3 using the presigned URL
  3. S3 emits an upload event which triggers the Transcoding Service via Kafka
  4. The Transcoding Service processes media:
    • Images: generates thumbnail, medium, and full resolution variants, applies compression and WebP conversion
    • Videos: generates multiple quality variants (360p, 480p, 720p, 1080p), chunks into HLS segments for adaptive streaming, generates a thumbnail frame
  5. All variants are written back to S3 under a deterministic path: media/{post_id}/{variant}/
  6. Transcoding completes and publishes a post.transcoded event to Kafka
  7. Two consumers pick up the event independently:
    • Metadata Worker writes post metadata to the Posts DB and populates the Post Metadata Cache in Redis
    • Search Indexing Worker indexes the post into Elasticsearch

For text-only posts, steps 1 through 6 are skipped entirely. The Post Service writes directly to the Posts DB and publishes a post.created event to Kafka. (This is the same pipeline as Instagram's upload flow.)

Data Model

Posts Table (PostgreSQL)

text
Posts
post_id         UUID, primary key
subreddit_id    UUID, foreign key -> Subreddits
user_id         UUID, foreign key -> Users
title           VARCHAR
content         TEXT (null for media posts)
media_url       TEXT (null for text posts)
media_type      ENUM('text', 'image', 'video')
upvotes         INTEGER
downvotes       INTEGER
score           INTEGER
hot_score       FLOAT
posted_at       TIMESTAMP

upvotes, downvotes, score, and hot_score are denormalized counters maintained asynchronously via Kafka consumers. This avoids expensive aggregation queries at read time.

Subreddits Table (PostgreSQL)

text
Subreddits
subreddit_id    UUID, primary key
name            VARCHAR, unique
description     TEXT
member_count    INTEGER
created_by      UUID, foreign key -> Users
created_at      TIMESTAMP

SubredditMembers Table (PostgreSQL)

text
SubredditMembers
user_id         UUID, foreign key -> Users
subreddit_id    UUID, foreign key -> Subreddits
joined_at       TIMESTAMP
PRIMARY KEY (user_id, subreddit_id)

This is a many-to-many join table. Joining a subreddit is an INSERT, leaving is a DELETE. "Give me all subreddits user X has joined" is a simple WHERE user_id = X query. The fan-out worker asks the reverse question, "who are the members of subreddit Y", so subreddit_id needs its own index as well.

Bottlenecks in the Post Service

The Post Service itself is stateless and horizontally scalable. The Transcoding Service is the bottleneck for media posts since generating multiple video quality variants is CPU-intensive. This is handled by running the Transcoding Service as a horizontally scalable worker pool pulling jobs from a Kafka topic. Text posts have no bottleneck.

5. Voting Service

Voting is the highest volume write path in the system at around 3,000 votes per second on average, with extreme spikes on viral posts. The voting system must handle burst traffic without data loss, prevent duplicate votes, and keep hotness scores up to date.

mermaid
flowchart LR
    C[Client] -->|POST /votes| VS[Vote Service]
    VS -->|"1. record vote, get previous one"| R[(Redis)]
    VS -->|"2. update counters"| R
    VS -->|"3. vote.created"| K[[Kafka]]
    K --> VP[Vote Persistence Worker] --> PG[(PostgreSQL Votes)]
    K --> SR[Score Recomputation Worker] --> HOT[(Redis hot sorted set)]
    K --> CS[Controversial Score Worker] --> CON[(Redis controversial sorted set)]

Full Vote Flow

Step 1 - Receive Vote

Client sends POST /votes with { user_id, post_id, direction: 'up' | 'down' } to the Vote Service via API Gateway.

Step 2 - Idempotency Check in Redis

Before any processing, the Vote Service records the vote in Redis and, in the same atomic command, gets back whatever the user's previous vote on this post was:

redis
SET vote:user_123:post_456 up GET      # returns the old value: nil, "up" or "down"
  • Nothing before: a new vote, process it.
  • Same direction before: a duplicate, the request is ignored.
  • Opposite direction before: the user is switching their vote (upvote to downvote), so the delta is computed and processed.

Doing the check and the write in one command matters: a separate "read, then write" would leave a gap where two retries of the same vote both see "no previous vote" and both get counted. This idempotency check happens entirely in Redis on the hot path without touching the database.

Step 3 - Increment Vote Counter in Redis

Vote counters are updated immediately in Redis:

redis
INCR post:post_456:upvotes
DECR post:post_456:downvotes     # only when switching from a downvote

Clients read vote counts from Redis, not from the database. This ensures near real-time vote count updates on viral posts without any database read pressure.

Step 4 - Publish to Kafka

The Vote Service publishes a vote.created event to Kafka immediately after the Redis update, and only confirms the vote to the client once Kafka has acknowledged it. This decouples the hot path from all downstream processing and provides durability. Even if downstream services are temporarily slow, no vote is lost.

Step 5 - Kafka Consumers

Three consumers process the event independently:

Vote Persistence Worker writes the vote to the Votes Table in PostgreSQL asynchronously. This is the permanent record of every vote.

Score Recomputation Worker recomputes the hotness score for the post. Reddit's hotness algorithm is a function of the net upvote score and the time elapsed since the post was created. Newer posts with fewer upvotes can outrank older posts with more upvotes because the time decay factor penalizes age aggressively. The recomputed score is written back to the Redis Sorted Set:

redis
ZADD subreddit:programming:hot <new_hot_score> post_456

The subreddit's Hot feed is updated instantly without any read-time computation.

Reddit's original, open-sourced version of this formula shows a neat trick:

text
hot = sign(score) × log10(max(|score|, 1)) + (posted_at_seconds / 45000)

Instead of making old posts' scores shrink as time passes, it gives every new post a slightly bigger head start, about one extra point every 12.5 hours. One point is worth 10× the votes on the log scale, so a post from 12.5 hours ago needs ten times the net upvotes to tie with a new one. The effect is the same as time decay, but a post's hot score only changes when it gets a vote, never just because time passed, so there is no need to recompute every post's score on a timer.

Controversial Score Worker separately computes the controversial score. A post is controversial when it has a high total vote count but a nearly equal upvote/downvote split. Reddit's version is roughly (upvotes + downvotes) ^ (smaller / larger), which is large only when there are many votes and the split is close to even. This score is written to a separate sorted set:

redis
ZADD subreddit:programming:controversial <controversial_score> post_456

Step 6 - Periodic Reconciliation

Redis vote counters are the fast read path. PostgreSQL is the source of truth. A background job periodically reconciles Redis counters against the database to catch any drift caused by Redis restarts or missed events.

Votes Table (PostgreSQL)

text
Votes
vote_id       UUID, primary key
user_id       UUID, foreign key -> Users
post_id       UUID, foreign key -> Posts
direction     ENUM('up', 'down')
voted_at      TIMESTAMP
UNIQUE (user_id, post_id)

The UNIQUE constraint on (user_id, post_id) enforces vote idempotency at the database level as a safety net, even though the Redis check handles it on the hot path. A vote switch becomes an update of the existing row's direction, not a second row.

Bottlenecks in the Voting Service

The Redis operations are sub-millisecond and handle the peak vote throughput comfortably. Kafka absorbs burst traffic from viral posts. The Score Recomputation Worker can become a bottleneck if a single post receives thousands of votes per second, since each vote triggers a recomputation. This is mitigated by debouncing recomputation in the worker: instead of recomputing on every single vote event, the worker batches vote events for the same post within a short window (say 5 seconds) and recomputes once per window. This reduces recomputation overhead by orders of magnitude on viral posts while keeping scores fresh. (The bulk processing pattern in action.)

6. Feed Service

Reddit's feed is subreddit-based, not follow-based. A user joins 50 subreddits. The home feed shows the best posts across all 50 subreddits, ranked by the selected sorting mode.

Redis Sorted Sets per Subreddit per Sorting Mode

Each subreddit maintains four Redis Sorted Sets, one per sorting mode:

KeyScored by
subreddit:{subreddit_id}:hothot_score (upvotes + time decay)
subreddit:{subreddit_id}:newposted_at timestamp
subreddit:{subreddit_id}:topNet upvote count
subreddit:{subreddit_id}:controversialControversial score

"Top" on Reddit also comes with a time range (today, this week, all time), which in practice means one Top set per range, each trimmed to the posts inside it.

These are updated in real time by Kafka consumers as votes come in and new posts are created. Feed reads are always served from Redis and never touch the Posts DB directly in the happy path.

Fan-out Strategy

Reddit uses a hybrid fan-out model similar to Instagram but based on subreddit size rather than user follower count:

Fan-out on write (small subreddits): When a post is created in a small subreddit (below a member threshold, say 100,000 members), the Feed Fanout Worker writes the post directly into the home feed sorted sets of all members. Each member's home feed is a Redis Sorted Set:

  • Key: feed:{user_id}:home
  • Member: post_id
  • Score: hot_score

The hot_score copied into each home feed is the one the post had at fan-out time. As the post collects votes, those copies fall behind, so the Feed Service re-ranks the candidates it reads using each post's current hot score from the subreddit set.

Fan-out on read (large subreddits): For large subreddits like r/programming or r/worldnews with millions of members, fan-out on write is impractical. Instead, their subreddit sorted sets are read directly at feed request time and merged with the user's precomputed home feed. A post going viral in r/worldnews does not trigger writes to millions of home feed sorted sets.

Home Feed Read Flow

When a user opens Reddit:

  1. Feed Service fetches the user's joined subreddits from the SubredditMembers cache in Redis
  2. For small subreddits: reads from the precomputed feed:{user_id}:home sorted set
  3. For large subreddits: reads top posts directly from subreddit:{subreddit_id}:hot
  4. Merges and re-ranks results across all subreddits
  5. Returns top 25 post IDs to the client
  6. Feed Service hydrates post IDs via Redis MGET post:1 post:2 ... post:25
  7. On cache miss for any post, falls back to Posts DB and writes back to Redis
  8. Client renders feed with full post metadata and media URLs pointing to CDN

Subreddit Feed Read Flow

When a user opens a specific subreddit with a sorting mode:

  1. Feed Service reads directly from subreddit:{subreddit_id}:{sort_mode} sorted set
  2. ZREVRANGE subreddit:programming:hot 0 24 returns top 25 post IDs instantly
  3. Post metadata hydrated via Redis MGET
  4. Media served via CDN

Bottlenecks in the Feed Service

The merge step for large subreddits at read time adds latency proportional to the number of large subreddits a user follows. This is mitigated by capping the number of large subreddit reads per request and caching the merged result in Redis with a short TTL (30-60 seconds) per user. Home feed sorted sets for inactive users are evicted from Redis and rebuilt from the Posts DB on next login.

7. Comment Service

Threaded nested comments are Reddit's most distinctive data structure challenge. A comment can have replies, those replies can have replies, to arbitrary depth. Each comment has its own upvote/downvote score.

Adjacency List Data Model

The comment tree is stored using the Adjacency List pattern in PostgreSQL. Each comment stores a reference to its parent:

text
Comments
comment_id      UUID, primary key
post_id         UUID, foreign key -> Posts
parent_id       UUID, foreign key -> Comments (NULL for top-level comments)
user_id         UUID, foreign key -> Users
content         TEXT
upvotes         INTEGER
downvotes       INTEGER
score           INTEGER
created_at      TIMESTAMP

A parent_id of NULL means the comment is a direct reply to the post. A non-null parent_id means it is a reply to another comment. This single column captures the entire tree structure without a graph database.

text
Post 1
├── c1 "Great write-up"          parent_id = NULL
│   ├── c3 "Agreed"              parent_id = c1
│   │   └── c5 "Same here"       parent_id = c3
│   └── c4 "Not convinced"       parent_id = c1
└── c2 "Source?"                 parent_id = NULL

A composite index on (post_id, parent_id, score DESC) makes the two primary queries fast, since both filter on the post and the parent, and the index already holds the rows in score order:

sql
-- Top-level comments for a post, sorted by score
SELECT * FROM Comments
WHERE post_id = X AND parent_id IS NULL
ORDER BY score DESC
LIMIT 20;

-- Replies to a specific comment, sorted by score
SELECT * FROM Comments
WHERE post_id = X AND parent_id = 'comment_123'
ORDER BY score DESC
LIMIT 10;

(Including post_id in the replies query lets it use the same index. Filtering on parent_id alone could not, because parent_id is not the index's first column.)

Lazy Loading

Reddit does not load the entire comment tree in one request. The full tree for a popular post could have hundreds of thousands of comments. Instead it uses lazy loading:

  1. Fetch top 20 top-level comments sorted by score
  2. For each top-level comment, fetch the top 3 replies
  3. Deeper levels show a "load more replies" button which triggers a separate request

This keeps initial page load fast regardless of comment tree depth.

Step 2 doesn't need 20 separate queries. A window function fetches the top 3 replies for all 20 parents at once:

sql
SELECT * FROM (
  SELECT c.*,
         ROW_NUMBER() OVER (PARTITION BY parent_id ORDER BY score DESC) AS rank
  FROM Comments c
  WHERE post_id = X AND parent_id IN (/* the 20 top-level comment ids */)
) replies
WHERE rank <= 3;

Comment Vote Flow

Comment votes follow the exact same Kafka pipeline as post votes. Each comment has its own score counter maintained by the Score Recomputation Worker. The comment sorted order within a thread is recomputed asynchronously after each vote batch.

Hot comments for popular posts are cached in Redis with a short TTL to avoid repeated PostgreSQL queries for the same comment thread.

Bottlenecks in the Comment Service

The Adjacency List pattern works well for fetching one level at a time via lazy loading. The bottleneck is deep recursive fetches if a client tries to load the full tree in one shot. This is prevented by enforcing pagination and depth limits at the API layer. PostgreSQL handles comment writes at the volume Reddit sees comfortably since comments are far less frequent than votes.

If whole subtrees ever do need to be fetched in one query, alternatives like a materialized path (storing each comment's full ancestry, e.g. c1/c3/c5, so a subtree is a simple prefix match) trade slightly more work on write for much cheaper subtree reads.

8. Search Service

Reddit search covers three entities: posts (by title and content), subreddits (by name and description), and users (by username).

Storage Engine - Elasticsearch

Elasticsearch handles full-text search, partial matching, and relevance ranking across all three entity types. Reddit search is primarily keyword-based rather than engagement-weighted like Instagram, but subreddit size and post score are still used as ranking signals.

Elasticsearch Documents

Post document:

json
{
  "post_id": "123",
  "title": "Why Rust is taking over systems programming",
  "content": "A deep dive into memory safety...",
  "subreddit": "programming",
  "subreddit_id": "456",
  "author": "harry_dev",
  "score": 15000,
  "posted_at": "2026-06-10T10:00:00Z"
}

Subreddit document:

json
{
  "subreddit_id": "456",
  "name": "programming",
  "description": "Computer programming discussion",
  "member_count": 5000000
}

Post score and subreddit member count act as ranking signals. A highly upvoted post in a large subreddit ranks above a low-score post in a tiny subreddit for the same search query.

Keeping Elasticsearch in Sync

The Search Indexing Worker consumes from the post.transcoded and post.created Kafka topics for new posts. Score updates are batched and synced to Elasticsearch periodically rather than on every vote, since exact scores in search results do not need to be real-time.

Hot search queries are cached in Redis with a short TTL to reduce Elasticsearch load for popular searches.

Same as Instagram and Spotify. Redis caching for hot queries filters the majority of repeat traffic before hitting Elasticsearch. Horizontal scaling of the Elasticsearch cluster handles the remaining query volume.

9. Full Data Flow Summary

Post Upload Flow:

text
Client -> API Gateway (auth) -> Post Service -> presigned S3 URL
Client -> S3 (direct upload)
S3 -> Kafka -> Transcoding Service -> S3 (all variants)
Transcoding Service -> Kafka (post.transcoded)
  -> Metadata Worker      -> Posts DB + Redis Cache
  -> Search Worker        -> Elasticsearch
  -> Feed Fanout Worker   -> Redis home feed sorted sets (small subreddits)
                          -> Subreddit sorted sets (all subreddits)

Vote Flow:

text
Client upvotes post -> Vote Service
  -> Redis SET vote:user:post up GET (record vote, get previous one)
  -> Redis INCR post:post_id:upvotes
  -> Kafka (vote.created)
      -> Vote Persistence Worker    -> PostgreSQL Votes Table
      -> Score Recomputation Worker -> ZADD subreddit:id:hot <new_score> post_id
      -> Controversial Score Worker -> ZADD subreddit:id:controversial <score> post_id

Home Feed Read Flow:

text
Client opens Reddit -> Feed Service
  -> Redis: fetch joined subreddits for user
  -> Redis ZREVRANGE feed:{user_id}:home (small subreddit posts)
  -> Redis ZREVRANGE subreddit:{id}:hot (large subreddit posts, merged at read time)
  -> Redis MGET post:1 post:2 ... post:25 (hydrate metadata)
  -> CDN (media URLs served directly)

Subreddit Feed Read Flow:

text
Client opens r/programming -> Feed Service
  -> Redis ZREVRANGE subreddit:programming:hot 0 24
  -> Redis MGET (hydrate post metadata)
  -> CDN (media)

Comment Flow:

text
Client loads post -> Comment Service
  -> PostgreSQL: top-level comments WHERE post_id = X AND parent_id IS NULL ORDER BY score DESC LIMIT 20
  -> PostgreSQL: top 3 replies per top-level comment (one window-function query)
  -> Lazy load deeper levels on demand

Search Flow:

text
Client searches "rust programming" -> Search Service
  -> Redis (hot query cache)
  -> Elasticsearch (cache miss)
  -> Redis MGET (hydrate post metadata for results)

10. Resilience and Fault Tolerance

  • Redis Sorted Sets for feeds mean feed reads never touch the Posts DB in the happy path. If PostgreSQL is slow or temporarily unavailable, users continue seeing their feeds from Redis. New posts stop appearing until the DB recovers, but existing feed content is unaffected.
  • Kafka durability means no vote is ever lost even during traffic spikes. Votes sit in Kafka until the Score Recomputation Worker processes them. A viral post generating 100,000 votes per hour does not overwhelm the system because Kafka absorbs the burst and workers process at their own pace.
  • Vote debouncing prevents the Score Recomputation Worker from being overwhelmed on viral posts. Batching vote events in 5-second windows reduces recomputation overhead by orders of magnitude while keeping scores acceptably fresh.
  • Idempotency at two layers for votes: Redis check on the hot path and a PostgreSQL UNIQUE constraint as a safety net. A user can never vote twice on the same post even under retry conditions or network failures. (The same reasoning as idempotency keys for payments.)
  • Adjacency List lazy loading prevents comment fetches from becoming unbounded. Enforcing depth limits and pagination at the API layer protects PostgreSQL from runaway recursive queries.
  • Cassandra replication for any high-volume write stores ensures no data loss under node failures.
  • CDN caching for media means profile photos, post images, and video chunks are served from edge nodes globally with no origin involvement for cache hits.

11. Technology Choices Summary

ComponentTechnologyWhy
API GatewayKong / AWS API GatewayAuth, rate limiting, routing
Blob StorageAWS S3Immutable media files, durable, CDN-native
CDNCloudFront / AkamaiEdge caching for post media and profile photos
TranscodingHorizontally scaled worker pool via KafkaImage variants, HLS video chunking
DecouplingApache KafkaVote durability, post fanout, score recomputation, search indexing
Search EngineElasticsearchFull-text search, partial matching, score-weighted ranking
Posts / Users / Subreddits DBPostgreSQL sharded + replicasRelational structure, strong consistency, composite indexes
Feed StoreRedis Sorted Sets per subreddit per sort modeSub-millisecond ranked feed reads
Vote CountersRedis INCRNear real-time vote counts without DB pressure
Vote IdempotencyRedis + PostgreSQL UNIQUE constraintTwo-layer duplicate prevention
Home Feed CacheRedis Sorted Sets per userPrecomputed home feed for small subreddit fanout
Post Metadata CacheRedis MGETBatch hydration of post metadata in single round trip
Comment StoragePostgreSQL Adjacency ListThreaded nested comments, lazy loaded by depth
Analytics StoreCassandra / ClickHouseHigh write volume, time-series voting data