HLD · 15 min read

Design Discord

System design for a discord-like platform

On this page
  1. 1. Requirements
  2. Functional Requirements
  3. Out of Scope
  4. Non-Functional Requirements
  5. 2. Capacity Estimation
  6. 3. High Level Services
  7. Gateway vs Message Service
  8. 4. API Gateway and Service Discovery
  9. 5. Message Send Path
  10. Message ID: Snowflake
  11. Ordering
  12. 6. Fan-out and the 500K Member Problem
  13. Why per-user queues fail
  14. Solution: fan-out per Gateway, not per user
  15. What still breaks and how to handle it
  16. 7. Message Storage
  17. Cassandra (ScyllaDB)
  18. Fix: time bucketing
  19. Supporting techniques
  20. 8. Presence and Typing Indicators
  21. Presence: heartbeat plus TTL
  22. The real problem: who is notified?
  23. Typing indicators: ephemeral
  24. Durability
  25. 9. Roles and Permissions
  26. Bitmask model
  27. Data model
  28. Effective permission, general to specific
  29. Making 35K checks per second fast
  30. Invalidation
  31. Security
  32. 10. Notifications
  33. 11. Search
  34. 12. Direct Messages
  35. 13. Full Data Flow Summary
  36. 14. Resilience and Fault Tolerance
  37. 15. Technology Choices Summary

1. Requirements

Functional Requirements

  • Create servers (guilds) and channels
  • Text messaging in channels with message history
  • Roles and per-channel permissions
  • Direct messages (1:1 and group), presence, and typing indicators
  • Notifications
  • Search across messages

Out of Scope

  • Voice and video (WebRTC and media servers, a separate system)
  • Bots and webhooks
  • Moderation tooling and audit logs
  • Go Live streaming

Non-Functional Requirements

  • Highly scalable, from 5-member friend servers to servers with millions of members
  • Low latency, messages delivered to online members within a few hundred milliseconds
  • Fault tolerance, no single point of failure on the message path
  • High throughput in busy servers and hot channels
  • Strict per-channel message ordering, every client sees the same order
  • Eventual consistency acceptable for presence, typing, and unread counters
  • Strong enforcement of permissions on the server side

2. Capacity Estimation

MetricValue
DAU500 million
Peak concurrent users~100 million (20% of DAU), each holding one WebSocket connection
Messages per day500M × ~2 = ~1 billion (most users read far more than they write)
Write QPS1B / 86,400 = ~11.5K per second average, ~35K per second at peak
Message size~1 KB including metadata (a sentence of text is 100 to 200 bytes)
Text storage1B × 1 KB = ~1 TB per day, ~365 TB per year, ~1.1 PB per year with replication factor 3
AttachmentsAssume 10% of messages carry a 1 MB file, so ~100 TB per day in S3 and CDN. Messages store only a URL
Gateway serversAt ~100K to 500K connections each, ~200 to 1,000 gateways
Fan-outOne message in a server with 500K online members is 500K deliveries. At 100 messages per second that is 50M deliveries per second for one server

Discord's load is highly skewed. Most servers have 10 to 50 members, while a few giants have hundreds of thousands online at once. The dominant challenges are holding 100M WebSocket connections and fan-out in giant servers, not raw storage.

3. High Level Services

ServiceResponsibility
User ServiceProfiles, settings, friend lists
Guild ServiceServers, channels, categories, invites (low write, merged Server and Channel service)
Permission ServiceRoles, per-channel overrides, cached permission checks
Gateway ServiceHolds persistent WebSocket connections, stateful, only moves bytes
Message ServiceStateless send and history path, validation, ID assignment, persistence
Presence ServiceOnline status and typing indicators, ephemeral
Media ServicePresigned S3 URLs for attachments
Notification ServicePush, unread counters, mentions
Search ServiceElasticsearch over messages

Gateway vs Message Service

The Gateway is the phone line: it keeps a live connection per user and carries bytes in both directions. It is stateful and scales with the number of connections (100M). The Message Service is the clerk: it checks rules, assigns an ID, saves the message, and publishes it. It is stateless and scales with messages per second (35K). Splitting them lets each tier scale independently. A crashed Gateway just means clients reconnect. A crashed Message Service instance just means a retry.

4. API Gateway and Service Discovery

REST requests (create server, edit profile, fetch history) pass through the API Gateway for authentication and rate limiting. Real-time traffic uses WebSockets.

When a client comes online, a load balancer and a service registry (etcd or Consul, both built on Raft) assign it a healthy Gateway. The registry also answers "which Gateway holds user X's connection?" for other services. On connect the Gateway authenticates the token, then the client sends heartbeats every ~30 seconds.

Binary media never passes through the API Gateway. Uploads use presigned S3 URLs.

5. Message Send Path

text
A user sends "hello" in #general
-> Gateway A (holds their WebSocket)
-> Message Service:
     1. Permission check (can they post in this channel?)
     2. Assign Snowflake ID
     3. Persist to Cassandra
     4. Publish to Kafka (keyed by channel_id)
-> Fan-out to Gateways subscribed to the channel
-> Each Gateway pushes to its local WebSocket connections

Message ID: Snowflake

A 64-bit integer:

text
| 41 bits: timestamp (ms since custom epoch) | 10 bits: worker ID | 12 bits: sequence |

The timestamp sits in the high bits, so IDs sort by time without a separate timestamp column. Worker ID gives uniqueness across machines without coordination. The sequence handles up to 4,096 IDs per millisecond per worker. Each Message Service instance generates IDs locally, so there is no central bottleneck. (The Unique ID Generator design covers Snowflake in detail. 41 + 10 + 12 = 63 bits, and the top bit is left as 0 so IDs stay positive.)

Ordering

Clients never decide order because their clocks disagree. The server does:

  1. Kafka is keyed by channel_id, so every message of a channel lands on the same partition. A partition is a strictly ordered log, so one channel always has a single, agreed order. (Many channels share each partition, since there are far more channels than partitions, but each channel's messages stay in order within it.)
  2. The Message Service stamps the Snowflake ID. Ties in the same millisecond are broken by worker ID and sequence.
  3. All clients sort by Snowflake ID and see the same order.

Since every client sorts by the same IDs, they always agree with each other. The IDs come from different Message Service instances, though, and their clocks can differ by a few milliseconds. So two messages sent at almost the same moment can end up ordered slightly differently from the order they truly arrived in, and very occasionally a message slots in just above one that's already on screen. Everyone still sees the same order, which is what the requirement asks for. (Why clocks can't be fully trusted.)

Unlike Google Docs, no Operational Transformation is needed. Chat messages are independent appends, not concurrent edits to shared state.

Tradeoff: one partition per channel caps throughput for an extremely hot channel. That is acceptable because a channel with thousands of messages per second is already unreadable.

6. Fan-out and the 500K Member Problem

Why per-user queues fail

Giving every recipient their own queue is fan-out on write (the News Feed design covers the same trade-off). At 100 members that is fine. At 500K online members, one message means 500K writes, and at 100 messages per second that is 50M writes per second for a single server.

Solution: fan-out per Gateway, not per user

500K online members are spread across roughly 500 gateways. Publish once per Gateway instead of once per user:

  1. Subscription map. Each Gateway keeps in memory which of its local users are viewing which channel.
  2. Subscribe on demand. A Gateway subscribes to the channel topic (Redis Pub/Sub or a Kafka consumer) when its first local user opens the channel, and unsubscribes when the last leaves.
  3. One publish per message to channel:42. Every subscribed Gateway receives it.
  4. Local fan-out. Each Gateway loops over its own connections for that channel and writes to each WebSocket. This is an in-memory loop of microseconds.
mermaid
flowchart TD
    MS[Message Service] -->|"publish once to channel:42"| PS[(Pub/Sub)]
    PS --> G1["Gateway 1"]
    PS --> G2["Gateway 2"]
    PS --> GN["… ~500 gateways"]
    G1 --> U1["800 WebSocket writes"]
    G2 --> U2["950 WebSocket writes"]
    GN --> UN["…"]

Work drops from 500K deliveries to about 500 publishes per message, a 1000x reduction.

What still breaks and how to handle it

  • Hot channel. All traffic for one channel goes through one Kafka partition and one pub/sub channel. Very large servers shard the pub/sub layer or add a relay tier (a tree of relays between the publisher and the gateways).
  • Per-Gateway write volume. 100 messages per second × 1,000 local users = 100K WebSocket writes per second on one Gateway. This is the practical ceiling.
  • Lazy delivery. Users not currently viewing a channel receive only a lightweight unread or mention counter, not the full message. Full messages go only to users with that channel open. This cuts real fan-out sharply.

7. Message Storage

Cassandra (ScyllaDB)

Messages are append-only and always read by channel in time order, which suits a wide-column store with horizontal scaling and replication factor 3. (Discord famously started on Cassandra and later moved to ScyllaDB, a faster Cassandra-compatible database.)

Naive key: channel_id

text
channel_id    BIGINT, partition key
message_id    BIGINT (Snowflake), clustering key DESC

Problems with this key:

  • Partitions grow without limit (a channel with 50M messages becomes a multi-GB partition)
  • Hot partitions for busy channels
  • Slow compaction, GC pauses, and painful node repair
  • Poor performance when scrolling deep into old history

Fix: time bucketing

text
Messages (Cassandra)
channel_id    BIGINT
bucket        INTEGER (floor(timestamp_ms / 10 days in ms))
message_id    BIGINT (Snowflake), clustering key DESC
author_id     BIGINT
content       TEXT
attachments   LIST<TEXT>
edited_at     TIMESTAMP
deleted       BOOLEAN
PRIMARY KEY ((channel_id, bucket), message_id)

The bucket is computed from the Snowflake timestamp with no extra lookup, and each partition stays bounded.

  • Write path: compute the bucket from the new ID and write to (channel_id, bucket).
  • Read path: read the newest 50 from the current bucket. If fewer than 50 are returned, step back through earlier buckets until 50 are collected. A cached "last message" pointer per channel avoids scanning empty buckets on quiet channels.

Supporting techniques

  • Hot-channel cache. The latest ~50 messages of active channels live in Redis, so most reads never touch Cassandra.
  • Soft deletes. Deletes write tombstones that hurt reads in busy partitions, so use a deleted flag and let compaction or TTL clean up later.
  • Attachments live in S3 behind a CDN. The row stores only the URL.

8. Presence and Typing Indicators

Presence: heartbeat plus TTL

text
Key:   presence:{user_id}
Value: { status: "online" | "idle" | "dnd", last_seen, gateway_id }
TTL:   60 seconds

The client sends a heartbeat every ~30 seconds, and the Gateway refreshes the TTL. If the app closes or the network drops, the key expires and the user shows offline. The client reports idle after 5 minutes without input. (The same pattern as the Chat System's presence.)

The real problem: who is notified?

One status flip for a user in 20 servers of 100K members each would mean 2M notifications. Fixes:

  • Notify only people who can see you right now. Friends, and members of servers whose member list is open on a client.
  • Lazy presence for big servers. Above a size threshold (say 100K members) do not push individual changes. The client requests presence only for the ~50 members visible in the sidebar and refreshes as the user scrolls.
  • Debounce. Wait a few seconds before broadcasting an offline transition so flaky connections do not flap.
  • Gateway-level fan-out. Publish one change per interested Gateway, and each Gateway fans out locally.

Typing indicators: ephemeral

  • The client sends a typing event when typing starts, then at most once every ~5 seconds while typing continues.
  • The Gateway publishes to the channel topic, and subscribed Gateways push "User is typing..." to clients viewing the channel.
  • The receiving client hides the indicator after ~8 to 10 seconds with no new event. No "stopped typing" event is needed.
  • Nothing is stored and nothing goes through Kafka. Losing a typing event is harmless, so Redis Pub/Sub (fire and forget) is enough.
  • In very large channels the indicator is throttled hard or disabled.

Durability

Presence is not stored durably. After a Redis restart, clients reconnect and heartbeats rebuild the state in about a minute. Only the user's preference (for example, invisible mode) lives in the User Service database.

9. Roles and Permissions

Bitmask model

Each permission is one bit:

text
VIEW_CHANNEL    = 1 << 0
SEND_MESSAGES   = 1 << 1
MANAGE_MESSAGES = 1 << 2
MANAGE_CHANNEL  = 1 << 3
ADMINISTRATOR   = 1 << 4

(These bit positions are illustrative. Discord's real API assigns different ones, but the idea is identical.)

A role is a number, and a check is one bitwise AND: (permissions & SEND_MESSAGES) != 0. No joins and no list scans.

Data model

text
Roles
role_id        BIGINT, primary key
guild_id       BIGINT
name           VARCHAR
permissions    BIGINT (bitmask)
position       INTEGER (hierarchy)

MemberRoles
guild_id, user_id, role_id

ChannelOverrides
channel_id     BIGINT
target_id      BIGINT (role_id or user_id)
target_type    ENUM('role', 'user')
allow          BIGINT (bits explicitly granted)
deny           BIGINT (bits explicitly denied)

Effective permission, general to specific

  1. base = OR of the permissions of all the user's roles (including the @everyone role every member has)
  2. If base has ADMINISTRATOR, allow everything and stop
  3. Apply the channel's @everyone override: base = (base & ~deny) | allow
  4. Apply overrides for the user's roles (OR the denies and allows together, then apply)
  5. Apply the override for the specific user
  6. The result is the effective bitmask for this user in this channel

The more specific rule always wins.

text
Roles give:              VIEW | SEND | MANAGE_MESSAGES
#announcements @everyone: deny SEND
Moderator role override:  allow SEND
→ a moderator can post in #announcements, everyone else can only read

Making 35K checks per second fast

text
Key:   perm:{guild_id}:{channel_id}:{user_id}:v{guild_perm_version}
Value: effective bitmask (8 bytes)
TTL:   10 minutes
  • Hot path is one Redis GET and one bitwise AND, sub-millisecond.
  • A small in-process L1 cache in the Message Service avoids most network calls.
  • On a miss, load roles and overrides, compute, and store.

Invalidation

Editing a role or override publishes permission.changed to Kafka. For a 500K-member server, deleting 500K keys is expensive, so each guild has a version number embedded in the key. Bumping guild_perm_version makes all old keys unreachable in one write, and they expire via TTL. (The same trick the Google Drive design uses per user.)

Building the key needs the current version, so each Message Service keeps the version numbers of its active guilds in memory and updates them from the permission.changed events, rather than reading the version from Redis before every check. The L1 cache is cleared for a guild at the same moment, so a change takes effect within the time it takes that event to arrive.

Security

The check runs server side in the Message Service and is never trusted from the client. VIEW_CHANNEL is also checked before delivering any event, so users never receive messages from channels they cannot see.

10. Notifications

Kafka fanout from the Message Service to the Notification Service, using the same APNs and FCM pattern as earlier designs, and as the Notification System design.

  • Mentions (@user, @role, @everyone) generate push notifications for users who are offline or not viewing the channel.
  • Unread counters are stored per user per channel in Redis and sent as lightweight events, not full messages.
  • Mute settings per server and channel are checked before sending.
  • @everyone in a huge server is rate limited and processed in batches to avoid a notification storm.
  • Online users receive events over their WebSocket, so push is sent only to offline devices.

Elasticsearch indexes messages for search within a server or DM.

json
{
  "message_id": "1234567890",
  "guild_id": "42",
  "channel_id": "7",
  "author_id": "99",
  "content": "deploy finished successfully",
  "has_attachment": false,
  "mentions": ["123"],
  "created_at": "2026-10-25T13:00:00Z"
}
  • Indexing is asynchronous. A Kafka consumer reads new messages and indexes them, so search lags by a few seconds, which is acceptable.
  • Index routing is by guild_id, so a search touches only that server's shard.
  • Filters such as from:user, in:channel, has:file, and before:date map to Elasticsearch filter clauses.
  • Security filter: results are limited to channels the user has VIEW_CHANNEL on.

12. Direct Messages

DMs reuse the same pipeline. A DM is a channel with two participants (or a few for group DMs), so it has a channel_id and the same Cassandra table, Snowflake IDs, and Kafka partitioning. Fan-out is trivial because there are only a couple of recipients. Permissions reduce to "are these users friends or share a server, and is neither blocking the other."

13. Full Data Flow Summary

Connect Flow:

text
Client -> Load balancer + service registry -> Gateway
    -> Authenticate token
    -> Register connection (user_id -> gateway_id)
    -> Heartbeat every ~30s refreshes presence TTL in Redis

Send Message Flow:

text
Client -> Gateway -> Message Service
    -> Permission check (Redis, L1 cache, bitmask AND)
    -> Assign Snowflake ID
    -> Write to Cassandra (channel_id, bucket)
    -> Publish to Kafka (keyed by channel_id)
    -> Publish once to channel pub/sub topic
        -> Each subscribed Gateway fans out to local WebSockets
    -> Kafka consumers:
        -> Search indexer -> Elasticsearch
        -> Notification Service -> mentions, unread counters, APNs/FCM
        -> Hot-channel cache -> Redis recent messages

Read History Flow:

text
Client -> Message Service
    -> Redis recent messages (hit for active channels)
    -> Cassandra (channel_id, bucket) stepping back through buckets
    -> Return 50 messages sorted by Snowflake ID

Typing Flow:

text
Client -> Gateway (throttled to 1 event per ~5s)
    -> Redis Pub/Sub on channel topic
    -> Subscribed Gateways -> clients (auto-hide after ~8s)

Presence Flow:

text
Heartbeat -> Gateway -> Redis SET presence:{user_id} TTL 60s
Status change -> publish once per interested Gateway -> local fan-out
Large servers: client requests presence for visible members only

Permission Change Flow:

text
Admin edits role or override -> Permission Service
    -> Update database
    -> Bump guild_perm_version
    -> Kafka (permission.changed) -> old cache keys become unreachable and expire

Attachment Flow:

text
Client -> Media Service -> presigned S3 URL
Client -> S3 (direct upload)
Message stores the URL, CDN serves the file

14. Resilience and Fault Tolerance

  • Stateless Message Service. Any instance handles any request, and failures are just retries.
  • Gateway failure. Clients detect the dropped WebSocket and reconnect through the load balancer to a new Gateway. They then fetch missed messages using their last seen message ID, which works because IDs are time-sortable.
  • Kafka durability. Replicated, keyed by channel, and consumers resume from offsets after crashes, so search indexing and notifications never lose events.
  • Cassandra replication. Replication factor 3 across availability zones means node loss does not lose messages.
  • Redis failure. Presence rebuilds from heartbeats. Permission and message caches fall back to the source of truth. Pub/Sub loss only drops live events, and clients backfill from history.
  • Hot-channel protection. Sharded pub/sub or relay tiers, lazy delivery, and throttled typing and presence in giant servers.
  • Backpressure. Per-user and per-channel rate limits at the Message Service stop spam from overwhelming a channel (backpressure).
  • Graceful degradation. If search or notifications are down, sending and receiving messages still work.

15. Technology Choices Summary

ComponentTechnologyWhy
API GatewayKong / AWS API GatewayAuth, rate limiting, REST routing
Real-time TransportWebSockets on a stateful Gateway tier100M concurrent connections, ~200 to 1,000 servers
Service Discoveryetcd / Consul plus load balancerGateway assignment, connection lookup
Message QueueApache Kafka keyed by channel_idOrdering, durability, fanout to search, notifications, cache
Live Fan-outRedis Pub/Sub with per-Gateway subscriptionsOne publish per Gateway, local in-memory fan-out
Message StoreCassandra / ScyllaDB with (channel_id, bucket) partition keyAppend-only, time-ordered, bounded partitions
Message IDsSnowflake64-bit, time-sortable, no coordination
Recent Message CacheRedisLatest ~50 messages per active channel
PresenceRedis with heartbeat TTLEphemeral, rebuilt from heartbeats
Typing IndicatorsRedis Pub/Sub, client throttledNever stored, never in Kafka
PermissionsBitmask roles and overrides in PostgreSQL, cached in Redis with guild version keysSub-millisecond checks, one-write invalidation
Guild, User, Role MetadataPostgreSQLLow write, relational, strongly consistent
SearchElasticsearch routed by guild_idFull text, filters, permission-aware results
MediaAWS S3 with presigned URLs plus CDNAttachments never touch the API layer
NotificationsKafka consumers plus APNs and FCMMentions, unread counters, mute rules