HLD · 23 min read

Design Medium

System design for an open publishing and social journalism platform like Medium

On this page
  1. 1. Requirements
  2. Functional Requirements
  3. Out of Scope
  4. Non-Functional Requirements
  5. 2. Capacity Estimation
  6. 3. API Gateway
  7. 4. User Profile Service
  8. Data Model
  9. Caching Strategy
  10. 5. Post Service
  11. Article Storage Architecture
  12. Article Content Structure (MongoDB)
  13. Articles Metadata Table (PostgreSQL)
  14. Tags and Topics
  15. Draft Autosave Flow
  16. Publish Flow
  17. Image Upload Flow
  18. Bottlenecks in the Post Service
  19. 6. Feed Service
  20. Fan-out on Write for All Users
  21. Weighted Score Formula
  22. Topic-Based Recommendations
  23. Already-Read Exclusion with Bloom Filter
  24. Feed Read Flow
  25. Bottlenecks in the Feed Service
  26. 7. Clap Service
  27. Redis Structure
  28. Full Clap Flow
  29. Claps Table (PostgreSQL)
  30. Bottlenecks in the Clap Service
  31. 8. Comment Service
  32. Data Model
  33. Bottlenecks in the Comment Service
  34. 9. Search Service
  35. Storage Engine - Elasticsearch
  36. Ranking Signals
  37. Keeping Elasticsearch in Sync
  38. 10. Reading History Service
  39. Data Model
  40. Write Flow
  41. Bottlenecks in the Reading History Service
  42. 11. Notification Service
  43. Notification Coalescing
  44. 12. Full Data Flow Summary
  45. 13. Resilience and Fault Tolerance
  46. 14. Technology Choices Summary

1. Requirements

Functional Requirements

  • Writer and reader profile management
  • Create, edit, publish, and delete articles (rich text, embedded images, code blocks)
  • Draft management with autosave
  • Clap on articles (up to 50 claps per user per article)
  • Threaded comments on articles
  • Follow and unfollow writers
  • Home feed (followed writers + topic-based recommendations)
  • Tags and topics per article
  • Search for articles, writers, and topics
  • Reading time estimation
  • Notifications (new article from followed writer, clap, comment, new follower)

Out of Scope

  • Premium membership and paywalled articles
  • Medium Partner Program (writer earnings)
  • Repost feature
  • Publications (multi-writer managed blogs)
  • Audio narration of articles

Non-Functional Requirements

  • High availability for article reading, the most frequent operation
  • Low latency feed reads under 100ms
  • Autosave must feel instantaneous, no perceptible write latency for writers
  • Eventual consistency acceptable for clap counts, feed ranking, and recommendations
  • Strong consistency required for article publish state (draft vs published)
  • Read/write ratio of approximately 1000:1, heavily optimized for reads
  • A user must never see an article they have already read in their feed

2. Capacity Estimation

MetricValue
DAU1 million
Articles published per day1-5% of DAU, ~10,000 articles per day at the low end
Text content per article~50 KB
Images per article~50% of articles have 2 images at 500 KB each
Text storage per day10,000 × 50 KB = ~500 MB/day (negligible)
Image storage per day10,000 × 0.5 × 2 × 500 KB = ~5 GB/day
Total storage per day~5 GB/day, dominated by images
Active drafts at any time~10× published articles = ~100,000 drafts being edited
Autosave writes100,000 active drafts × 1 autosave per 5 seconds = ~20,000 writes/second during peak writing hours
Read/write ratio~1000:1, a viral article is read tens of thousands of times after being written once

The autosave figure is a deliberate worst case: it assumes every one of those 100,000 drafts is open and being typed into at the same moment. Real concurrency is lower, but designing for the worst case is what keeps autosave instant on a busy evening.

The dominant challenge is not storage scale but autosave write volume and feed personalization quality. Medium is a content discovery platform as much as a publishing platform, and the feed experience determines whether readers return daily.

3. API Gateway

All client requests enter through the API Gateway which handles authentication, rate limiting, and routing. Media uploads (article images) bypass the Gateway after authentication using presigned S3 URLs for direct client-to-S3 upload. The API Gateway never handles binary media payloads.

Medium has a meaningful unauthenticated read traffic pattern. Many readers arrive via Google search or shared links without being logged in. The API Gateway routes unauthenticated article reads directly to the Article Read path without authentication overhead, while write operations (clap, comment, follow, publish) always require authentication.

Those logged-out reads are also the easiest traffic in the whole system to cache. A published article looks the same to every anonymous reader, so the rendered page can be cached at the CDN edge for a few minutes and served without touching the backend at all, which also makes it fast for search engine crawlers.

4. User Profile Service

Medium has two overlapping user types: writers who publish content and readers who consume it. Most users are both simultaneously.

Data Model

Users Table (PostgreSQL)

text
Users
user_id              UUID, primary key
username             VARCHAR, unique
email                VARCHAR, unique
display_name         VARCHAR
bio                  TEXT
profile_pic_url      TEXT
website_url          TEXT
twitter_handle       VARCHAR
follower_count       INTEGER
following_count      INTEGER
article_count        INTEGER
total_claps_received INTEGER
created_at           TIMESTAMP

follower_count, following_count, article_count, and total_claps_received are denormalized counters updated asynchronously via Kafka consumers. Aggregating these at read time from raw tables would be too slow at scale.

Follows Table (PostgreSQL)

text
Follows
follower_id     UUID, foreign key -> Users
followee_id     UUID, foreign key -> Users
created_at      TIMESTAMP
PRIMARY KEY (follower_id, followee_id)

A follow is an INSERT, an unfollow is a DELETE. "Give me all writers user X follows" is WHERE follower_id = X. Indexed on both columns for fast lookups in both directions.

UserTopics Table (PostgreSQL)

text
UserTopics
user_id         UUID, foreign key -> Users
topic           VARCHAR
interest_score  FLOAT (updated by recommendation pipeline)
created_at      TIMESTAMP
PRIMARY KEY (user_id, topic)

Explicit topic interests (user manually follows a topic) and inferred interests (derived from reading behavior by the recommendation pipeline) both live here. The interest_score is updated periodically by the ML pipeline based on reading history signals.

Caching Strategy

Writer profiles are cached in Redis since they are fetched on every article page view. Hot writers (those with many followers whose articles appear in many feeds) benefit most from caching. User topic interest scores are cached per user for fast feed assembly.

5. Post Service

The Post Service manages the full article lifecycle from draft creation through editing, publishing, and deletion.

Article Storage Architecture

Medium articles are not simple text strings. They are rich documents with headings, paragraphs, images, code blocks, pull quotes, and embedded links. This structure is stored as a JSON document in MongoDB, not as raw text in PostgreSQL.

MongoDB is chosen over PostgreSQL for article content because:

  • Flexible schema per article (different articles have different block structures)
  • Efficient storage and retrieval of large nested JSON documents
  • No need for complex joins on article content
  • Native support for partial document updates (editing one section without rewriting the entire document)

PostgreSQL stores article metadata (title, slug, status, author, clap count). MongoDB stores article content (the full rich text document). S3 stores embedded images referenced within the article.

Article Content Structure (MongoDB)

json
{
  "article_id": "uuid",
  "blocks": [
    { "type": "heading", "level": 1, "text": "Why I Switched to Rust" },
    { "type": "paragraph", "text": "It started on a Tuesday morning..." },
    { "type": "image", "url": "https://cdn.medium.com/...", "caption": "My workspace" },
    { "type": "code", "language": "rust", "content": "fn main() { ... }" },
    { "type": "paragraph", "text": "The borrow checker felt restrictive at first..." }
  ],
  "updated_at": "2026-06-20T10:00:00Z"
}

This block-based structure is similar to how Notion, Substack, and Medium's own editor (backed by a similar format) represent rich content.

Articles Metadata Table (PostgreSQL)

text
Articles
article_id        UUID, primary key
author_id         UUID, foreign key -> Users
title             VARCHAR
subtitle          VARCHAR
cover_image_url   TEXT
slug              VARCHAR, unique (null until published)
status            ENUM('draft', 'published', 'deleted')
reading_time      INTEGER (minutes, computed on publish)
clap_count        INTEGER (denormalized total)
comment_count     INTEGER (denormalized counter)
view_count        INTEGER (denormalized counter)
published_at      TIMESTAMP (null until published)
created_at        TIMESTAMP
updated_at        TIMESTAMP

Tags and Topics

ArticleTags Table (PostgreSQL)

text
ArticleTags
article_id    UUID, foreign key -> Articles
tag           VARCHAR
PRIMARY KEY (article_id, tag)

Tags are set by the writer. Topics are broader categories inferred or manually selected. Both are indexed in Elasticsearch for search and used by the recommendation pipeline for topic-based feed personalization.

Draft Autosave Flow

Medium autosaves every few seconds while a writer is typing. At 100,000 active drafts with saves every 5 seconds, this is ~20,000 writes per second during peak hours. Writing every autosave directly to MongoDB would overwhelm the database.

The autosave flow uses Redis as a write buffer (the write-behind pattern):

  1. Writer types, client debounces keystrokes and sends autosave request every 3-5 seconds

  2. Post Service writes draft content to Redis immediately:

    text
    Key:   draft:{article_id}
    Value: serialized article content JSON
    TTL:   7 days
    
  3. Response returned to client instantly (sub-millisecond Redis write)

  4. Autosave event published to Kafka: draft.updated

  5. Kafka consumer flushes draft content from Redis to MongoDB asynchronously every 30 seconds per draft

mermaid
sequenceDiagram
    participant W as Writer's editor
    participant P as Post Service
    participant R as Redis
    participant K as Kafka
    participant F as Flush worker
    participant M as MongoDB
    W->>P: autosave (every 3-5 s)
    P->>R: SET draft:{article_id}
    P-->>W: saved ✓
    P->>K: draft.updated
    K->>F: consume
    F->>R: read latest draft (at most every 30 s)
    F->>M: write draft

The writer never waits for MongoDB. Redis absorbs all autosave writes. MongoDB receives batched updates at a sustainable rate. Because the flush worker only writes the latest version of each draft every 30 seconds, the 20,000 writes per second reaching Redis turn into a few thousand writes per second reaching MongoDB.

On cache miss (Redis evicted the draft, writer returns after 7 days), the draft is fetched from MongoDB and re-populated in Redis.

One trade-off to be aware of: until the next flush, the newest edits exist only in Redis. If a Redis node is lost before it reaches MongoDB, up to ~30 seconds of typing can be lost. Turning on Redis AOF persistence shrinks that window, and the client keeps a local copy of the draft, so in practice a writer almost never loses work.

Publish Flow

When a writer clicks Publish:

  1. Post Service reads final draft content from Redis (or MongoDB on cache miss)
  2. Generates unique URL slug from title (e.g. why-i-switched-to-rust-a1b2c3)
  3. Computes reading time: word_count / 200 (average adult reading speed in wpm; Medium itself uses about 265 wpm)
  4. Writes final content to MongoDB (permanent published version)
  5. Updates Articles Table: status = 'published', slug = generated_slug, published_at = NOW()
  6. Publishes article.published event to Kafka
  7. Kafka consumers:
    • Search Indexing Worker indexes article into Elasticsearch
    • Feed Fanout Worker writes article into followers' feed sorted sets
    • Notification Worker notifies all followers of new article
    • Topic Index Worker updates topic sorted sets in Redis with new article

The order of steps 4 and 5 matters for the strong-consistency requirement on publish state. The content goes into MongoDB first, and only then does the status flip to published in PostgreSQL. That single PostgreSQL update is the moment of publishing. If the MongoDB write fails, the article simply stays a draft. If the order were reversed, a failure in between would leave a "published" article with no body.

Image Upload Flow

When a writer inserts an image into the editor:

  1. Client requests presigned S3 URL from Post Service
  2. Client uploads image directly to S3
  3. Image Processing Worker generates multiple resolution variants (thumbnail for feed cards, medium for article body, full for expanded view)
  4. CDN sits in front of S3, serving images globally with high cache hit rates

Bottlenecks in the Post Service

The autosave Redis write buffer is the key architectural decision that prevents MongoDB from being overwhelmed. The publish flow is low volume (10,000 per day) and has no bottleneck concerns. The image processing pipeline is the same pattern as all previous systems and scales horizontally via Kafka workers.

6. Feed Service

Medium's feed combines two signals: articles from writers the user follows and topic-based recommendations. This is more complex than Twitter's purely follow-based feed but simpler than Netflix's multi-stage recommendation pipeline.

Fan-out on Write for All Users

Unlike Instagram and Twitter where celebrities with millions of followers require fan-out on read, Medium's largest writers have at most a few hundred thousand followers. Fan-out on write for all users is perfectly viable. No hybrid model needed.

When an article is published, the Feed Fanout Worker writes it into the feed sorted set of every follower:

  • Key: feed:{user_id}
  • Member: article_id
  • Score: weighted relevance score (recency + follow boost + engagement signals)
  • TTL: 7 days

Weighted Score Formula

text
Score = base_engagement_score + follow_boost + recency_decay
  • base_engagement_score reflects how much engagement (claps, comments, reads) the article has received since publishing
  • follow_boost is added for articles from writers the user follows, ensuring followed content gets priority but does not completely dominate
  • recency_decay ensures newer articles score higher than older ones with the same engagement, preventing stale content from staying at the top

This weighted merge means the feed is a blend of follow-based and topic-based content rather than a strict ordering. A highly relevant fresh topic recommendation can surface above a stale article from a followed writer, which matches Medium's actual feed behavior.

Note that at fan-out time a brand-new article has almost no engagement yet, so the score stored in each follower's sorted set is mostly follow boost and recency. That's why the Feed Service re-ranks at read time (below) using the article's current engagement.

Topic-Based Recommendations

The recommendation pipeline runs periodically and computes topic-based article recommendations per user based on:

  • Reading history signals: topics of articles the user read to completion (high completion_pct) weighted more than articles they bounced from
  • Collaborative filtering: users who read the same articles as this user also read Y
  • Topic graph: adjacent topics (self-help and poetry share readers; backend engineering and system design share readers)
  • Followed writer topics: infer interest from the topics of writers the user follows

Top topic-based recommendations are written to Redis per user:

text
Key:   recommendations:{user_id}
Value: [article_id_1, article_id_2, ..., article_id_20]
TTL:   6 hours

At feed assembly time, the Feed Service merges the precomputed follow feed sorted set with the topic recommendations, applies the weighted scoring, and returns the top 20 article IDs.

Already-Read Exclusion with Bloom Filter

A user must never see an article they have already read in their feed. The Reading History Service maintains a Bloom Filter per user:

redis
BF.ADD seen:{user_id} article_id       # when user opens an article
BF.EXISTS seen:{user_id} article_id    # before inserting into feed sorted set

Before any article is inserted into a user's feed sorted set or returned as a recommendation, BF.EXISTS is checked. If the article has been read, it is excluded. No false negatives means a read article is never shown again. The small false positive rate (occasionally excluding an unread article) is acceptable. An article can also be read after it was inserted into someone's feed, which is why the same check runs again when the feed is read (step 3 below).

Feed Read Flow

When a user opens their Medium home feed:

  1. Feed Service fetches top 50 article IDs from feed:{user_id} sorted set via ZREVRANGE feed:user_123 0 49
  2. Merges with topic recommendations from recommendations:{user_id} Redis key
  3. Applies Bloom Filter exclusion for already-read articles (one BF.MEXISTS call for all candidates)
  4. Re-ranks merged list by weighted score
  5. Hydrates top 20 article IDs via Redis MGET for article metadata (title, subtitle, cover image, author, reading time, clap count)
  6. On cache miss for any article, falls back to PostgreSQL and writes back to Redis
  7. Returns rendered feed cards to client

Bottlenecks in the Feed Service

Fan-out on write for 100,000 followers of a top writer means writing to 100,000 sorted sets per article publish. At 10,000 articles per day with average 500 followers each, this is 5 million sorted set writes per day, which is very manageable with a horizontally scaled Fanout Worker pool. Redis memory for feed sorted sets is capped by the 7-day TTL and a maximum of 200 articles per feed sorted set.

7. Clap Service

The Clap Service handles Medium's unique engagement mechanism. A single reader can clap up to 50 times on a single article. Total claps displayed on an article is the SUM of all readers' clap counts, not a simple count of unique clappers.

Redis Structure

Per-user clap count (cap enforcement):

text
Key:   claps:{article_id}:{user_id}
Value: integer (1 to 50)
TTL:   30 days

Total clap count (display on article):

text
Key:   article:claps:{article_id}
Value: integer (running total across all users)

Every clap that passes the per-user cap check also increments article:claps:{article_id}. Reading total claps for display is a single GET article:claps:{article_id}. No SUM query needed at read time.

Full Clap Flow

  1. User taps clap button (can tap multiple times rapidly)
  2. Client debounces taps and batches them (sends accumulated clap count every 1 second)
  3. Clap Service receives { user_id, article_id, clap_increment }
  4. Read the user's current clap count, compute the allowed increment min(clap_increment, 50 - current_count), and apply it, all in one atomic step (see below)
  5. If allowed increment > 0:
    • INCRBY claps:{article_id}:{user_id} allowed_increment
    • INCRBY article:claps:{article_id} allowed_increment
    • Publish clap.created event to Kafka
  6. Kafka consumers:
    • Clap Persistence Worker upserts to Claps Table in PostgreSQL
    • Counter Sync Worker periodically syncs Redis total to clap_count on Articles Table
    • Notification Worker notifies article author (coalesced, not one notification per clap)

Steps 4 and 5 have to run as one atomic operation. With a plain GET followed by INCRBY, two clap batches from the same user arriving at the same moment (two open tabs, or a retry) can both read 45, both decide 5 more claps are allowed, and end at 55. Running the check and both increments inside a single Lua script closes that gap:

lua
-- KEYS[1] = claps:{article_id}:{user_id}, KEYS[2] = article:claps:{article_id}
-- ARGV[1] = clap_increment
local current = tonumber(redis.call('GET', KEYS[1]) or '0')
local allowed = math.min(tonumber(ARGV[1]), 50 - current)
if allowed > 0 then
  redis.call('INCRBY', KEYS[1], allowed)
  redis.call('INCRBY', KEYS[2], allowed)
end
return allowed

The 30-day TTL on the per-user key needs one more guard. Once it expires, a missing key reads as 0, so a reader returning after a month could clap another 50 times on the same article. So when the key is missing, the service first loads the user's real count from the Claps table (the source of truth) into Redis, and only then runs the script.

Claps Table (PostgreSQL)

text
Claps
user_id       UUID, foreign key -> Users
article_id    UUID, foreign key -> Articles
clap_count    INTEGER (1 to 50)
last_clapped  TIMESTAMP
PRIMARY KEY (user_id, article_id)

An UPSERT on (user_id, article_id) with clap_count = clap_count + allowed_increment handles both first clap and subsequent claps in a single operation:

sql
INSERT INTO Claps (user_id, article_id, clap_count, last_clapped)
VALUES ($1, $2, $3, NOW())
ON CONFLICT (user_id, article_id)
DO UPDATE SET clap_count   = LEAST(Claps.clap_count + EXCLUDED.clap_count, 50),
              last_clapped = NOW();

Bottlenecks in the Clap Service

Clap bursts on viral articles (thousands of readers clapping simultaneously) are absorbed entirely by Redis, where each script runs atomically in well under a millisecond. The Kafka consumer handles async persistence without any write pressure on PostgreSQL during the burst. Periodic reconciliation between Redis totals and PostgreSQL clap_count catches any drift.

8. Comment Service

Comments on Medium are threaded but shallower than Reddit. Medium encourages responses (separate articles written as replies) more than deep comment threads. Comments are low volume compared to claps.

Data Model

Comments Table (PostgreSQL)

text
Comments
comment_id    UUID, primary key
article_id    UUID, foreign key -> Articles
parent_id     UUID, foreign key -> Comments (null for top-level)
user_id       UUID, foreign key -> Users
content       TEXT
clap_count    INTEGER
created_at    TIMESTAMP
updated_at    TIMESTAMP

Same Adjacency List pattern as Reddit. parent_id = NULL means top-level comment on the article. Non-null parent_id means reply to another comment. Top-level comments sorted by clap count (most appreciated comments surface first). Replies sorted chronologically within a thread.

A composite index on (article_id, parent_id) makes both primary queries fast:

sql
-- Top-level comments
SELECT * FROM Comments
WHERE article_id = X AND parent_id IS NULL
ORDER BY clap_count DESC
LIMIT 20;

-- Replies to a comment
SELECT * FROM Comments
WHERE article_id = X AND parent_id = 'comment_123'
ORDER BY created_at ASC;

(Both queries filter on article_id first so they can use the same index.)

Hot article comment sections are cached in Redis with a short TTL. The daily most-read articles benefit significantly from comment caching.

Bottlenecks in the Comment Service

Comment volume is low at Medium's scale. PostgreSQL handles comment writes comfortably. The main concern is comment sections on viral articles receiving thousands of simultaneous reads. Redis caching of top-level comments absorbs this.

9. Search Service

Medium search covers three entities: articles (by title, content, and tags), writers (by name and username), and topics.

Storage Engine - Elasticsearch

Elasticsearch handles full-text search, partial matching, and relevance ranking across all article content. Medium search is primarily keyword and topic-based rather than engagement-weighted like Instagram, but clap count and read count are used as ranking signals.

Article document:

json
{
  "article_id": "123",
  "title": "Why I Switched to Rust After 10 Years of Python",
  "subtitle": "A pragmatic engineer's perspective",
  "author_id": "456",
  "author_name": "harry_dev",
  "tags": ["rust", "python", "programming", "backend"],
  "content_preview": "It started on a Tuesday morning when my service crashed for the third time...",
  "reading_time": 8,
  "clap_count": 4200,
  "view_count": 18000,
  "published_at": "2026-06-20T10:00:00Z"
}

Full article content is not indexed in Elasticsearch, only a content preview (first 500 characters) and tags. Indexing full bodies would make the index much larger and slower to keep updated, and most search intent is satisfied by title, tags, and preview matching.

Writer document:

json
{
  "user_id": "456",
  "username": "harry_dev",
  "display_name": "Harry Singh",
  "bio": "Backend engineer. Writing about distributed systems and Rust.",
  "follower_count": 12000,
  "article_count": 47
}

Ranking Signals

Medium search ranking combines:

  • Text match score (title match weighted higher than tag match, tag match weighted higher than preview match)
  • Clap count (more appreciated articles rank higher for the same query)
  • Recency (newer articles get a ranking boost for broad queries)
  • Author follower count (established writers rank slightly higher)

Keeping Elasticsearch in Sync

The Search Indexing Worker consumes from article.published and article.updated Kafka topics. Clap count and view count updates are batched and synced to Elasticsearch periodically rather than on every clap since exact engagement counts in search results do not need to be real-time. (The same derived read model idea from the Microservices series.)

10. Reading History Service

The Reading History Service tracks which articles each user has read and how deeply they read them. This data feeds the recommendation pipeline and powers the Bloom Filter for feed deduplication.

Data Model

ReadingHistory Table (PostgreSQL, partitioned by user_id)

text
ReadingHistory
user_id         UUID, foreign key -> Users
article_id      UUID, foreign key -> Articles
read_at         TIMESTAMP
completion_pct  FLOAT (0.0 to 1.0, how far the user scrolled)
time_spent_sec  INTEGER (actual reading time in seconds)

completion_pct is the most valuable signal. A user who reads 95% of an 8-minute article about Rust is far more interested in Rust than a user who opened and immediately bounced. This distinction drives topic interest scoring in the recommendation pipeline.

Write Flow

Every article view generates a read event. At 1 million DAU reading an average of 5 articles per day, this is 5 million read events per day. Writes are high volume and must not block the reading experience.

Read events are published to Kafka asynchronously from the client:

  1. User opens article -> client sends GET /articles/{slug} -> article served
  2. Client tracks scroll position and time spent in the background
  3. On article close or after 30 seconds: client sends read event { user_id, article_id, completion_pct, time_spent_sec }
  4. Reading History Service publishes to Kafka: article.read
  5. Kafka consumers:
    • History Persistence Worker writes to ReadingHistory Table
    • Bloom Filter Worker calls BF.ADD seen:{user_id} article_id
    • View Counter Worker increments INCR article:views:{article_id} in Redis
    • Recommendation Signal Worker updates UserTopics interest scores based on completion_pct

Bottlenecks in the Reading History Service

5 million read events per day is ~58 per second on average, well within Kafka's capacity. The Bloom Filter writes are sub-millisecond Redis operations. PostgreSQL partition by user_id keeps per-user history queries fast.

11. Notification Service

Same architecture as all previous systems, and as the Notification System design. Kafka fanout from all interaction services, APNs for iOS, FCM for Android, email via SES.

The Notification Service subscribes to:

  • article.published -> "Harry published a new article" to all followers
  • clap.created -> coalesced clap notification to article author ("Harry and 42 others clapped for your article")
  • comment.created -> comment notification to article author and parent comment author
  • follow.created -> "Someone followed you" notification

Notification Coalescing

Clap notifications must be coalesced aggressively. A viral article receiving 5,000 claps in an hour should not send 5,000 push notifications to the author. The Notification Service batches clap events within a 1-hour window and sends a single notification: "Your article received 5,247 claps in the last hour."

New article notifications from followed writers are delivered individually since each represents a distinct content event the reader wants to know about.

12. Full Data Flow Summary

Article Creation Flow (Draft):

text
Writer types -> client debounces -> autosave every 3-5 seconds
    -> Post Service -> Redis SET draft:{article_id} (sub-millisecond)
    -> Kafka (draft.updated)
    -> MongoDB Writer (async, flushes every 30 seconds per draft)

Image Upload Flow:

text
Writer inserts image -> Post Service -> presigned S3 URL
Writer -> S3 (direct upload)
S3 -> Kafka -> Image Processing Worker -> S3 (thumbnail, medium, full variants)
CDN caches all variants

Publish Flow:

text
Writer clicks Publish -> Post Service
    -> Generate slug, compute reading_time
    -> Write final content to MongoDB
    -> UPDATE Articles: status = 'published', published_at = NOW()  (the commit point)
    -> Kafka (article.published)
        -> Search Worker        -> Elasticsearch index
        -> Feed Fanout Worker   -> Redis ZADD feed:{follower_id} for all followers
        -> Notification Worker  -> APNs / FCM / SES to all followers
        -> Topic Index Worker   -> Redis topic sorted sets updated

Feed Read Flow:

text
User opens Medium -> Feed Service
    -> Redis ZREVRANGE feed:{user_id} 0 49 (follow-based articles)
    -> Redis GET recommendations:{user_id} (topic-based articles)
    -> Bloom Filter BF.MEXISTS seen:{user_id} for the candidates (exclude read articles)
    -> Weighted score merge and re-rank
    -> Redis MGET (hydrate article metadata for top 20)
    -> CDN (cover images served directly)

Article Read Flow:

text
User opens article -> CDN (cached page for logged-out readers) or Post Service
    -> Redis (article metadata cache)
    -> MongoDB (full article content)
    -> CDN (embedded images)
    -> Client tracks scroll + time spent
    -> On close: read event published to Kafka (article.read)
        -> History Persistence Worker   -> ReadingHistory Table (PostgreSQL)
        -> Bloom Filter Worker          -> BF.ADD seen:{user_id} article_id
        -> View Counter Worker          -> INCR article:views:{article_id}
        -> Recommendation Signal Worker -> UPDATE UserTopics interest scores

Clap Flow:

text
User claps -> client debounces and batches -> Clap Service
    -> (if the per-user key is missing) load the user's count from PostgreSQL
    -> Lua script: check the cap, INCRBY the user's count and the article total
    -> Kafka (clap.created)
        -> Clap Persistence Worker  -> UPSERT Claps Table (PostgreSQL)
        -> Counter Sync Worker      -> UPDATE Articles.clap_count (periodic batch)
        -> Notification Worker      -> coalesced clap notification to author

Search Flow:

text
User searches "rust backend" -> Search Service
    -> Redis (hot query cache)
    -> Elasticsearch (title + tag + preview match, ranked by claps + recency)
    -> Redis MGET (hydrate article metadata for results)

13. Resilience and Fault Tolerance

  • Redis autosave buffer ensures writers never lose work even if MongoDB is temporarily unavailable. Drafts survive in Redis with a 7-day TTL. The Kafka consumer catches up on MongoDB writes when the database recovers.
  • Bloom Filter for feed deduplication has no false negatives. A read article is never shown again. The small false positive rate (occasionally excluding an unread article) is completely acceptable and users never notice.
  • Fan-out on write for all users means feed reads are always fast regardless of database health. The feed sorted set in Redis is the primary read path. PostgreSQL is only hit on full cache misses.
  • Kafka durability throughout the pipeline means no article publish event, clap event, or read event is lost even if downstream services are temporarily unavailable. All consumers catch up from Kafka offsets on recovery.
  • Redis clap counters with async PostgreSQL sync absorb clap bursts on viral articles without any write pressure on the database. Periodic reconciliation catches any drift between Redis totals and PostgreSQL clap_count.
  • MongoDB for article content decouples content storage from metadata storage. A MongoDB slowdown affects article body reads but not feed assembly, search results, or profile pages which all use PostgreSQL and Redis.
  • CDN caching for all article images, and for whole article pages served to logged-out readers, means most read traffic is served from edge nodes globally. Article images change rarely (writers do not edit images after publishing) so cache hit rates are very high.
  • Elasticsearch eventual consistency for article indexing is acceptable. A newly published article may not appear in search results for a few seconds while the Search Indexing Worker processes the Kafka event. Readers discover new articles primarily through their feed, not through search.

14. Technology Choices Summary

ComponentTechnologyWhy
API GatewayKong / AWS API GatewayAuth, rate limiting, unauthenticated read routing
Article Content StoreMongoDBRich text JSON documents, flexible schema, large nested content
Article Metadata DBPostgreSQL sharded + replicasTitle, slug, status, counters, strong consistency for publish state
Draft Autosave BufferRedisSub-millisecond autosave writes, 7-day TTL, async flush to MongoDB via Kafka
Image StorageAWS S3Embedded article images, durable, CDN-native
CDNCloudFront / AkamaiEdge caching for article images and logged-out article pages
DecouplingApache KafkaArticle publish fanout, clap persistence, read events, notification triggers
Search EngineElasticsearchFull-text search on title, tags, preview, clap-weighted ranking
Feed StoreRedis Sorted Sets per userWeighted score merge of follow-based and topic-based articles
Topic RecommendationsRedis per userPrecomputed topic-based article recommendations, 6-hour TTL
Read DeduplicationRedis Bloom Filter per userAlready-read article exclusion from feed
Clap Counter (per-user cap)Redis + Lua scriptAtomic 50-clap limit enforcement
Clap Counter (article total)Redis INCRBYRunning total, single GET for display
Claps DBPostgreSQL UPSERTSource of truth for per-user clap counts
Reading History DBPostgreSQL partitioned by user_idCompletion rate, time spent, recommendation signals
User Profile CacheRedisWriter profiles, follower counts, topic interest scores
Notification DeliveryAPNs + FCM + AWS SESPush notifications and email, aggressive clap coalescing