HLD · 20 min read

Design Twitter

System design for a microblogging platform like Twitter.

On this page
  1. 1. Requirements
  2. Functional Requirements
  3. Out of Scope
  4. Non-Functional Requirements
  5. 2. Capacity Estimation
  6. 3. API Gateway
  7. 4. User Service
  8. Profile Photo and Banner Upload
  9. Data Model
  10. Bottlenecks in the User Service
  11. 5. Tweet Service and Upload Pipeline
  12. Tweet Creation Flow
  13. Data Model
  14. Write Spike Handling
  15. Bottlenecks in the Tweet Service
  16. 6. Feed Service
  17. Algorithmic Feed
  18. Chronological Feed
  19. Feed Hydration
  20. Hot Key Problem
  21. Bottlenecks in the Feed Service
  22. 7. Trending Service
  23. Sliding Window Counter with Redis Sorted Sets
  24. Trending Computation Worker
  25. Hashtag Extraction Pipeline
  26. Personalized Trending
  27. Bottlenecks in the Trending Service
  28. 8. Search Service
  29. Storage Engine - Elasticsearch
  30. Elasticsearch Documents
  31. Keeping Elasticsearch in Sync
  32. Bottlenecks in Search
  33. 9. Interaction Service
  34. Like Flow
  35. Retweet Flow
  36. Reply Flow
  37. Bottlenecks in the Interaction Service
  38. 10. Notification Service
  39. 11. Full Data Flow Summary
  40. 12. Resilience and Fault Tolerance
  41. 13. Technology Choices Summary

1. Requirements

Functional Requirements

  • User profile management (name, username, profile photo, banner, bio, date of birth, followers, following)
  • Create tweets (text up to 280 characters, images, videos)
  • Like, reply, retweet, follow, unfollow
  • Home feed (algorithmic and chronological modes)
  • Trending page (global trending hashtags and topics)
  • Search for tweets, users, and hashtags
  • Notifications (likes, retweets, replies, follows, mentions)

Out of Scope

  • Direct Messages (same WebSocket architecture as Tinder)
  • Twitter Spaces (live audio, separate real-time system)
  • Twitter Blue / verified features
  • Ad targeting system
  • Analytics pipeline

Non-Functional Requirements

  • High availability across feed, tweet creation, and media serving
  • Low latency feed reads under 100ms
  • Handle coordinated write spikes during global events (World Cup goals, breaking news)
  • Hot key resilience for viral tweets read by millions simultaneously
  • Eventual consistency acceptable for like counts, retweet counts, and feed ranking
  • Strong consistency required for follow/unfollow relationships and tweet existence

2. Capacity Estimation

MetricValue
DAU200 million
Original tweets per day150 million (~1,700 tweets/sec on average), the same as 75% of DAU posting once a day
Tweet composition60% text only, 30% with images, 10% with videos
Text storage150M × 300 bytes = ~45 GB/day (every tweet has text; negligible)
Image storage30% × 150M × 1 MB = ~45 TB/day
Video storage10% × 150M × 10 MB = ~150 TB/day
Total storage per day~200 TB/day, dominated by video (before the extra transcoded variants)
Read/write ratioApproximately 3:1, counting likes, retweets, replies and follows as writes

The 3:1 read/write ratio is the most important number here. Unlike Instagram where celebrities produce and millions passively consume, Twitter's users are constantly retweeting, replying, and liking. Counted that way, as every interaction rather than just new tweets, the write path must be as robust and scalable as the read path, a constraint that does not exist in the same way for the other systems we have designed. (Timeline reads still vastly outnumber new tweets, which is why the feed is precomputed further down.)

3. API Gateway

All client requests enter through the API Gateway which handles authentication, rate limiting, and routing. The same constraint applies as every previous design: the API Gateway is never in the path of binary media payloads. Photo and video uploads bypass the Gateway after authentication using presigned S3 URLs for direct client-to-S3 upload.

Twitter's API Gateway has an additional concern compared to Instagram or Reddit: write spike absorption. During a major global event, millions of users simultaneously hit the tweet creation endpoint. Rate limiting at the Gateway level is the first line of defense, shedding excess traffic before it reaches downstream services.

4. User Service

The User Service manages all profile data including name, username, profile photo, banner image, bio, date of birth, follower count, and following count.

Profile Photo and Banner Upload

Same presigned URL pattern:

  1. Client authenticates via API Gateway and requests a presigned S3 URL from the User Service
  2. Client uploads directly to S3
  3. A Photo Processing Worker generates thumbnail and full resolution variants
  4. CDN sits in front of S3 and serves profile media globally with high cache hit rates since profile photos change rarely

Data Model

Users Table (PostgreSQL)

text
Users
user_id           UUID, primary key
username          VARCHAR, unique
display_name      VARCHAR
bio               TEXT
profile_pic_url   TEXT
banner_url        TEXT
date_of_birth     DATE
follower_count    INTEGER
following_count   INTEGER
tweet_count       INTEGER
verified          BOOLEAN
created_at        TIMESTAMP

follower_count, following_count, and tweet_count are denormalized counters updated asynchronously via Kafka consumers. Counting followers by querying the Follows table on every profile load would be catastrophically slow at Twitter's scale.

Follows Table (PostgreSQL)

text
Follows
follower_id     UUID, foreign key -> Users
followee_id     UUID, foreign key -> Users
created_at      TIMESTAMP
PRIMARY KEY (follower_id, followee_id)

A follow is an INSERT, an unfollow is a DELETE. "Give me everyone user X follows" is WHERE follower_id = X. Indexed on both columns for fast lookups in both directions. For second-degree relationships like suggested follows and mutual friends, a graph database like Neo4j or AWS Neptune is more appropriate since graph traversals across millions of nodes are native in graph DBs but expensive in relational joins.

Bottlenecks in the User Service

User profiles are read far more often than they are written. Redis caches hot profiles including verified accounts and celebrities whose profiles are fetched millions of times per day. The PostgreSQL read replicas handle the remaining traffic. The Follows table grows large at Twitter's scale with billions of rows and is sharded by follower_id to keep individual shard sizes manageable.

Sharding by follower_id makes "who does X follow" a single-shard query, but the opposite question, "who follows X", is exactly what the fan-out worker asks on every tweet, and it would now have to check every shard. The usual fix is to store each follow twice: once sharded by follower_id and once by followee_id, so both directions stay single-shard lookups.

5. Tweet Service and Upload Pipeline

Tweet Creation Flow

Text tweets:

  1. Client sends POST /tweets with content to the Tweet Service via API Gateway
  2. Tweet Service writes the tweet (through the Redis write buffer described below)
  3. Publishes tweet.created event to Kafka
  4. Kafka consumers fan out to feed, search, and notification systems

Media tweets:

  1. Client requests a presigned S3 URL from the Tweet Service
  2. Client uploads media directly to S3
  3. S3 emits an upload event (S3 event notifications, bridged into Kafka as media.uploaded)
  4. Transcoding Service picks up the event and processes:
    • Images: generates thumbnail, medium, and full resolution variants, applies WebP compression
    • Videos: generates multiple quality variants (360p, 480p, 720p, 1080p), chunks into HLS segments for adaptive streaming, generates a preview thumbnail
  5. All variants stored in S3 under media/{tweet_id}/{variant}/
  6. Transcoding Service publishes tweet.transcoded event to Kafka
  7. Three consumers pick up the event independently:
    • Metadata Worker writes tweet metadata to Tweets DB and populates Tweet Metadata Cache in Redis
    • Search Indexing Worker indexes the tweet into Elasticsearch
    • Feed Fanout Worker begins writing the tweet into followers' feed sorted sets

Data Model

Tweets Table (PostgreSQL)

text
Tweets
tweet_id        UUID, primary key
user_id         UUID, foreign key -> Users
content         VARCHAR(280)
media_url       TEXT (null for text tweets)
media_type      ENUM('none', 'image', 'video')
reply_to_id     UUID, foreign key -> Tweets (null for original tweets)
like_count      INTEGER
retweet_count   INTEGER
reply_count     INTEGER
view_count      INTEGER
posted_at       TIMESTAMP

like_count, retweet_count, reply_count, and view_count are all denormalized counters updated asynchronously. The source of truth for individual interactions lives in separate tables (Likes, Retweets, Replies) but counts are maintained on the Tweet row for fast reads.

A random UUID works as a primary key, but a time-sortable ID is a better fit for tweets. Twitter built Snowflake for exactly this: IDs are generated without coordination, sort by creation time, and keep new rows together at the end of the index instead of scattered randomly through it.

Retweets Table (PostgreSQL)

text
Retweets
retweet_id          UUID, primary key
original_tweet_id   UUID, foreign key -> Tweets
retweeter_id        UUID, foreign key -> Users
retweeted_at        TIMESTAMP

A retweet is a reference to an existing tweet and no new content is stored. The retweet_count on the original tweet row is incremented asynchronously by a Kafka consumer. When user A retweets a post, it fans out to user A's followers, not the original author's followers. The original author receives only a notification.

Likes Table (PostgreSQL)

text
Likes
like_id     UUID, primary key
tweet_id    UUID, foreign key -> Tweets
user_id     UUID, foreign key -> Users
liked_at    TIMESTAMP
UNIQUE (tweet_id, user_id)

The UNIQUE constraint enforces like idempotency at the database level. Like counts on the Tweet row are the fast read path and the Likes table is the source of truth.

Write Spike Handling

During a global event, millions of users tweet simultaneously. Writing every tweet synchronously to PostgreSQL under this load would cause the database to collapse. The mitigation is a write buffer in Redis, the write-behind pattern:

  1. Tweet is written to Redis immediately on creation
  2. tweet.created event published to Kafka
  3. Kafka consumer writes to PostgreSQL asynchronously off the hot path
  4. Redis write buffer absorbs the spike and PostgreSQL writes at a sustainable rate

Like and retweet counts follow the same pattern, incremented in Redis first and flushed to PostgreSQL asynchronously via Kafka.

Note that it's Kafka, not Redis, that makes this safe: the tweet is durably stored in Kafka's replicated log before the user gets a success response, so even if a Redis node dies, the PostgreSQL write still happens. Redis is there so the tweet is readable immediately, before PostgreSQL catches up.

Bottlenecks in the Tweet Service

The Transcoding Service is the bottleneck for media tweets. Horizontal scaling of the worker pool handles this. For text tweets, the bottleneck is the write spike problem described above, which is mitigated by the Redis write buffer and Kafka decoupling.

6. Feed Service

Twitter's feed has two modes that operate differently at the infrastructure level: algorithmic feed and chronological feed.

Algorithmic Feed

The algorithmic feed ranks tweets by relevance, engagement, and relationship closeness. This ranking is expensive to compute at read time, so it is precomputed via fan-out on write.

Fan-out strategy (the same hybrid model as the News Feed design):

  • Normal users (below follower threshold): Fan-out on write. When user A tweets, the Feed Fanout Worker writes the tweet into the feed sorted sets of all of user A's followers immediately.
  • Celebrity users (above follower threshold, say 1 million followers): Fan-out on read. Their tweets are stored in a Celebrity Tweets Cache in Redis. At feed read time, the Feed Service fetches the latest celebrity tweets and merges them with the precomputed feed.

Redis Sorted Set structure:

  • Key: feed:{user_id}:algorithmic
  • Member: tweet_id
  • Score: relevance score (combination of recency, engagement signals, relationship closeness)
redis
ZREVRANGE feed:user_123:algorithmic 0 19     # top 20 tweet IDs, instantly

One limitation to keep in mind: a score computed at fan-out time freezes the tweet's engagement at that moment, while a tweet that goes viral an hour later deserves to move up. Real ranking systems therefore treat the precomputed set as a candidate list, and re-rank the top few hundred candidates with fresh signals at read time.

Chronological Feed

The chronological feed is pure reverse time order with the latest tweets from followed accounts first. Because ranking is not needed, read time computation is viable and avoids the fan-out write problem for high-follower accounts entirely.

At read time:

  1. Fetch the list of accounts the user follows from Redis

  2. Query PostgreSQL:

    sql
    SELECT tweet_id FROM Tweets
    WHERE user_id IN (followed_accounts)
    ORDER BY posted_at DESC
    LIMIT 20;
    
  3. With a composite index on (user_id, posted_at), the database only reads the newest few tweets of each followed account instead of scanning their history. For users following thousands of accounts this is still real work, spread across Tweets shards, which is exactly why the next step matters

  4. Result cached in Redis per user with a short TTL: feed:{user_id}:chronological

Repeat opens of the chronological feed hit Redis, not PostgreSQL.

Feed Hydration

Both feed modes return tweet IDs. Full tweet content is hydrated via Redis MGET:

redis
MGET tweet:101 tweet:202 tweet:303 ... tweet:2020

Single round trip, all 20 tweet metadata records fetched simultaneously. On cache miss, falls back to PostgreSQL and writes back to Redis.

Hot Key Problem

When a celebrity with 50 million followers tweets something viral, that single tweet's cache key is read millions of times per minute. Standard Redis caching struggles because all reads hit the same Redis node for the same key (a hot key that no amount of sharding spreads out).

The solution is multi-layer caching (L1/L2/L3):

LayerWhereWhat it holds
L1Local in-memory cache on each application server (an LRU cache like Caffeine in Java)Viral tweets. Zero network hop, sub-microsecond reads
L2RedisTweets that are hot but not viral enough to warrant L1 caching on every server
L3PostgreSQLThe source of truth, hit only on full cache misses

A viral tweet read by 50 million users is served from thousands of app server local caches simultaneously. Redis is only hit when a server's local cache is cold. The hot key problem largely disappears. The price is that each server's L1 copy can be a few seconds stale, which is fine for like and retweet counts.

Bottlenecks in the Feed Service

The Fan-out Worker for the algorithmic feed is the primary bottleneck during high tweet volume from moderately popular accounts. The hybrid model mitigates this for celebrities. The merge step for celebrity tweets at read time adds latency proportional to the number of celebrity accounts a user follows, mitigated by caching the merged result with a short TTL.

The Trending page shows what the entire world is talking about right now, updated in near real-time. This is unique to Twitter.

Trending is not simply the hashtags with the most total tweets. #FIFA has billions of tweets over its lifetime and should not trend forever. Trending means a spike in activity right now compared to the historical baseline. A hashtag trending means its usage in the last hour is significantly above its average usage over the last 24 hours.

Sliding Window Counter with Redis Sorted Sets

For every hashtag extracted from an incoming tweet, the Trending Service maintains a Redis Sorted Set:

  • Key: hashtag:FIFA
  • Member: tweet_id (unique event identifier)
  • Score: Unix timestamp of the tweet

To count how many times #FIFA was used in the last hour:

redis
ZCOUNT hashtag:FIFA (now - 3600) now

This is an O(log N) operation, extremely fast even with millions of entries. Because the baseline needs the last 24 hours, entries are kept for 24 hours, and anything older is periodically cleaned up:

redis
ZREMRANGEBYSCORE hashtag:FIFA 0 (now - 86400)

This is a sliding window counter. The window moves forward with time automatically, keeping counts fresh without any manual reset.

Storing one entry per tweet for a full day can use a lot of memory for very popular hashtags. A lighter alternative is to keep one counter per hashtag per minute (INCR hashtag:FIFA:202606201005, with a TTL of a day) and sum the buckets you need, trading exactness at the window edges for far less memory.

A Trending Computation Worker runs every 1-5 minutes and:

  1. Fetches the top hashtags by recent tweet count using ZCOUNT across all active hashtag keys
  2. Computes the 24-hour baseline tweet count for each hashtag
  3. Ranks hashtags by spike ratio, the percentage jump above baseline, not raw count
  4. Writes the top 50 trending topics to a Redis key: trending:global with a short TTL
text
#FIFA:    last hour 90,000   24h hourly average 30,000   → 3×  (trending)
#Monday:  last hour 50,000   24h hourly average 48,000   → ~1× (not trending)

The client reads trending:global on every page load, a single Redis GET returning the precomputed list. No computation happens at read time.

Hashtag Extraction Pipeline

When a tweet is created, a Hashtag Extraction Worker consumes from the tweet.created Kafka topic, parses the tweet content for hashtags, and for each hashtag calls:

redis
ZADD hashtag:{tag} <timestamp> <tweet_id>

This is decoupled from the tweet creation hot path via Kafka. Tweet creation does not wait for hashtag extraction.

Twitter also shows personalized trending, topics relevant to your location and interests. This is computed by the same Worker but filtered by the user's location (country, city) and interest graph (accounts they follow, topics they engage with). Personalized trending results are cached per user with a longer TTL since they change less frequently than global trending.

The Hashtag Extraction Worker must process 150 million tweets per day, roughly 1,700 tweets per second. Each tweet can contain multiple hashtags. At this rate, the Worker pool must be horizontally scaled to keep up. The Redis Sorted Set operations are sub-millisecond each and not a concern. The Trending Computation Worker running every few minutes is a batch job and does not affect real-time performance.

8. Search Service

Twitter search covers three entities: tweets (by content and hashtag), users (by username and display name), and hashtags as first-class searchable objects.

Storage Engine - Elasticsearch

Elasticsearch handles full-text search, partial matching, and relevance ranking across all three entity types. Twitter search combines text match score with engagement signals. A highly liked tweet ranks above an obscure tweet for the same search query.

Elasticsearch Documents

Tweet document:

json
{
  "tweet_id": "123",
  "content": "Just shipped a new feature using Rust and WebAssembly",
  "hashtags": ["rust", "webassembly", "programming"],
  "author": "harry_dev",
  "author_follower_count": 15000,
  "like_count": 3200,
  "retweet_count": 890,
  "posted_at": "2026-06-20T10:00:00Z"
}

User document:

json
{
  "user_id": "456",
  "username": "harry_dev",
  "display_name": "Harry The Nerd",
  "bio": "Backend engineer. Writing about distributed systems.",
  "follower_count": 15000,
  "verified": false
}

Like count, retweet count, and follower count act as ranking signals. Verified accounts rank higher for user searches. Hashtags in tweets are indexed as separate terms so searching #rust returns all tweets containing that hashtag.

Keeping Elasticsearch in Sync

The Search Indexing Worker consumes from the tweet.created and tweet.transcoded Kafka topics. Engagement count updates are batched and synced to Elasticsearch periodically rather than on every interaction since exact counts in search results do not need to be real-time. (The same derived read model idea from the Microservices series.)

Hot search queries are cached in Redis with a short TTL to reduce Elasticsearch load.

Same pattern as all previous designs. Redis caching for hot queries filters the majority of repeat traffic before hitting Elasticsearch. Horizontal scaling of the Elasticsearch cluster handles remaining query volume.

9. Interaction Service

The Interaction Service handles likes, replies, retweets, and bookmarks. These are the highest volume write operations in the system after tweet creation itself.

Like Flow

  1. Client sends POST /likes with { user_id, tweet_id }
  2. Idempotency check and record in one step: SADD likes:tweet_123 user_456 in Redis. SADD returns 1 if the user was newly added and 0 if they had already liked it, in which case the request is ignored. (A separate SISMEMBER check followed by SADD would leave a gap where two retries both pass the check.)
  3. INCR tweet:tweet_123:like_count to update the counter in Redis
  4. Publish like.created event to Kafka
  5. Kafka consumers:
    • Like Persistence Worker writes to Likes Table in PostgreSQL asynchronously
    • Counter Sync Worker periodically syncs Redis like counts to the like_count column on the Tweets row in PostgreSQL
    • Notification Worker triggers a notification to the tweet author via APNs/FCM

Retweet Flow

  1. Client sends POST /retweets with { user_id, tweet_id }
  2. Idempotency check in Redis, a user can only retweet a tweet once
  3. Write to Retweets Table asynchronously via Kafka
  4. Increment retweet_count on the original tweet in Redis
  5. Feed Fanout Worker fans out the retweeted tweet to the retweeter's followers' feed sorted sets, not the original author's followers
  6. Notification sent to the original author

Reply Flow

Replies are tweets with a non-null reply_to_id. A reply is created through the same Tweet Service flow as an original tweet, with the additional reply_to_id field set. The original tweet's reply_count is incremented asynchronously via Kafka.

Bottlenecks in the Interaction Service

Likes are the highest volume operation. A viral tweet can receive thousands of likes per second. The Redis idempotency check and counter increment absorb this entirely in-memory. Kafka handles durability and decouples the hot path from PostgreSQL writes.

10. Notification Service

Same architecture as Instagram, and as the Notification System design. Kafka fanout from all interaction services, APNs for iOS, FCM for Android, with notification coalescing to prevent spam. "Harry and 4,999 others liked your tweet" instead of 5,000 individual push notifications.

The Notification Service subscribes to:

  • like.created topic
  • retweet.created topic
  • reply.created topic
  • follow.created topic
  • mention.created topic (extracted from tweet content by the Hashtag Extraction Worker)

Device tokens stored in a simple Redis hash per user. In-app notifications stored in a Notifications Table in PostgreSQL, served via WebSocket when the app is open.

11. Full Data Flow Summary

mermaid
flowchart LR
    C[Client] --> GW[API Gateway]
    GW --> TS[Tweet Service]
    TS --> WB[(Redis write buffer)]
    TS --> K[[Kafka]]
    K --> MW[Metadata Worker] --> PG[(PostgreSQL)]
    K --> SW[Search Worker] --> ES[(Elasticsearch)]
    K --> FW[Feed Fanout Worker] --> FS[(Redis feeds)]
    K --> HW[Hashtag Worker] --> TR[(Redis trending)]
    K --> NW[Notification Worker] --> PN[APNs / FCM]
    GW --> FD[Feed Service]
    FD --> FS
    FD --> TC[(Redis tweet cache)]

Tweet Creation Flow (text):

text
Client -> API Gateway (auth) -> Tweet Service
-> Redis write buffer (absorb spike)
-> Kafka (tweet.created)
    -> Metadata Worker       -> Tweets DB + Redis Metadata Cache
    -> Search Worker         -> Elasticsearch
    -> Feed Fanout Worker    -> Redis feed sorted sets (normal user followers)
                             -> Celebrity Tweet Cache (for celebrity authors)
    -> Hashtag Worker        -> ZADD hashtag:{tag} <timestamp> <tweet_id>
    -> Notification Worker   -> mentions extracted -> APNs / FCM

Tweet Creation Flow (media):

text
Client -> API Gateway (auth) -> Tweet Service -> presigned S3 URL
Client -> S3 (direct upload)
S3 -> Kafka -> Transcoding Service -> S3 (all variants)
Transcoding Service -> Kafka (tweet.transcoded)
    -> Metadata Worker    -> Tweets DB + Redis Cache
    -> Search Worker      -> Elasticsearch
    -> Feed Fanout Worker -> Redis feed sorted sets

Algorithmic Feed Read Flow:

text
Client -> Feed Service
-> Redis ZREVRANGE feed:{user_id}:algorithmic 0 19
-> Redis GET celebrity tweets (merged at read time for celebrity followees)
-> Redis MGET tweet:1 tweet:2 ... tweet:20 (hydrate metadata)
-> L1 local cache (viral tweets served from app server memory)
-> CDN (media URLs served directly)

Chronological Feed Read Flow:

text
Client -> Feed Service
-> Redis GET feed:{user_id}:chronological (cache hit)
-> PostgreSQL: SELECT tweet_id FROM Tweets WHERE user_id IN (...) ORDER BY posted_at DESC LIMIT 20
-> Redis MGET (hydrate metadata)
-> Write result back to Redis with short TTL

Like Flow:

text
Client likes tweet -> Interaction Service
-> Redis SADD likes:tweet_id user_id (idempotency: 0 means already liked)
-> Redis INCR tweet:tweet_id:like_count
-> Kafka (like.created)
    -> Like Persistence Worker  -> PostgreSQL Likes Table
    -> Counter Sync Worker      -> PostgreSQL Tweets.like_count (periodic batch)
    -> Notification Worker      -> APNs / FCM (coalesced)

Trending Flow:

text
tweet.created -> Kafka -> Hashtag Extraction Worker
-> ZADD hashtag:{tag} <timestamp> <tweet_id>
Trending Computation Worker (every 1-5 min)
-> ZCOUNT hashtag:{tag} (now-3600) now (last hour count)
-> Compare against 24hr baseline
-> ZREMRANGEBYSCORE (evict entries older than 24 hours)
-> SET trending:global [top 50 hashtags] with TTL
Client -> GET trending:global (single Redis read)

Search Flow:

text
Client -> Search Service
-> Redis (hot query cache)
-> Elasticsearch (cache miss)
-> Redis MGET (hydrate tweet metadata for results)

12. Resilience and Fault Tolerance

  • Redis write buffer for tweet creation and interactions absorbs thundering herd write spikes during global events. PostgreSQL never receives the raw spike and processes writes at a sustainable rate from the Kafka queue.
  • Multi-layer caching (L1/L2/L3) for viral tweets largely eliminates the hot key problem. App server local caches serve the vast majority of reads for viral content with zero network hops. Redis handles warm content. PostgreSQL is only hit on full cache misses.
  • Kafka durability throughout the pipeline means no tweet, like, or retweet is ever lost even if downstream services are temporarily unavailable. All consumers can fall behind and catch up on recovery without data loss.
  • Idempotency at two layers for likes and retweets: Redis Set check on the hot path and a PostgreSQL UNIQUE constraint as a safety net. A user can never like the same tweet twice even under retry conditions.
  • Sliding window counters in Redis for trending are self-maintaining. Old entries are cleaned up automatically via ZREMRANGEBYSCORE. The Trending Computation Worker is a stateless batch job that can be restarted at any time without data loss.
  • CDN caching for all media means profile photos, tweet images, and video chunks are served from edge nodes globally. Origin infrastructure is only hit on cache misses.
  • Read replicas on PostgreSQL for all tables ensure no single point of failure for reads. Writes go to the primary and reads fan out across replicas.
  • Chronological feed fallback means if the algorithmic feed precomputation pipeline is lagging, users can always switch to chronological mode which computes at read time from PostgreSQL independently of the fan-out pipeline.

13. Technology Choices Summary

ComponentTechnologyWhy
API GatewayKong / AWS API GatewayAuth, rate limiting, write spike shedding
Blob StorageAWS S3Immutable media files, durable, CDN-native
CDNCloudFront / AkamaiEdge caching for tweet media and profile photos
TranscodingHorizontally scaled worker pool via KafkaImage variants, HLS video chunking
DecouplingApache KafkaTweet fanout, like durability, trending extraction, search indexing
Search EngineElasticsearchFull-text search, hashtag indexing, engagement-weighted ranking
Users / Tweets / Likes DBPostgreSQL sharded + replicasRelational structure, strong consistency, composite indexes
Algorithmic Feed StoreRedis Sorted Sets per userPrecomputed ranked feed, sub-millisecond reads
Chronological Feed CacheRedis with short TTLCached read-time computed feed
Tweet Metadata CacheRedis MGETBatch hydration of tweet content in single round trip
Hot Tweet CacheL1 local in-memory (Caffeine LRU) + L2 RedisMulti-layer hot key mitigation
Write BufferRedisAbsorb thundering herd write spikes before Kafka and PostgreSQL
Trending StoreRedis Sorted Sets with sliding windowZCOUNT for time-windowed hashtag frequency
Interaction IdempotencyRedis Set + PostgreSQL UNIQUE constraintTwo-layer duplicate prevention
Follows GraphPostgreSQL + Neo4j / AWS NeptuneSimple follow in SQL, suggested follows via graph traversal