Design Google Docs
System design for a real-time collaborative platform like Google Docs

On this page
- 1. Requirements
- Functional Requirements
- Out of Scope
- Non-Functional Requirements
- 2. Capacity Estimation
- 3. API Gateway
- 4. Operational Transformation
- The Core Problem
- Operation Structure
- Why Not Timestamps
- OT Algorithm (Simplified)
- The Client Side of OT
- OT vs CRDTs
- 5. User and Auth Service
- Data Model
- 6. Document Service
- Document Storage Architecture
- Data Model
- Permission Caching in Redis
- Permission Invalidation
- Document Content in MongoDB
- Bottlenecks in the Document Service
- 7. Collaboration Service
- WebSocket Connection Management
- Multi-Server WebSocket Problem
- Full Operation Flow: User A Presses "X"
- Cursor Presence
- Initial Document State for Joining Users
- Bottlenecks in the Collaboration Service
- 8. Operation Log Service
- Operation Storage in Cassandra
- Operations Table (Cassandra)
- Snapshot Strategy
- Operation Retention Policy
- 9. Version History Service
- Point-in-Time Reconstruction
- Named Versions
- 10. Offline Editing and Sync
- Offline Flow
- Reconnection Flow
- 11. Comment Service
- Data Model
- 12. Notification Service
- 13. Search Service
- 14. Full Data Flow Summary
- 15. Resilience and Fault Tolerance
- 16. Technology Choices Summary
1. Requirements
Functional Requirements
- Create, read, update, and delete documents
- Real-time collaborative editing (multiple users editing simultaneously)
- Cursor presence (see where other users are typing with their avatar)
- Rich text formatting (bold, italic, headings, lists, code blocks)
- Document sharing and permissions (view only, comment, edit, public link)
- Comment threads on specific text selections
- Version history (see and restore previous versions of a document)
- Offline editing with automatic sync on reconnect
Out of Scope
- Google Sheets (spreadsheet logic)
- Google Slides (presentation logic)
- Google Drive (file system management)
- Export to PDF or Word
- AI writing suggestions
- Voice typing
Non-Functional Requirements
- Strong consistency for document state: all users must end up seeing the same document content
- Real-time operation delivery under 100ms for users in the same region
- No data loss for offline edits: operations queued locally must survive reconnection
- Conflict-free convergence: concurrent edits from multiple users must always produce the same final document state
- Horizontal scalability for the collaboration layer, a single popular document can have 100 concurrent editors
- Eventual consistency acceptable for version history reconstruction and search indexing
2. Capacity Estimation
| Metric | Value |
|---|---|
| DAU | 50 million |
| Average document size | 1 to 5 MB including formatting metadata and embedded images (the text and formatting alone are usually tens of KB, images live in S3) |
| Documents created per day | 10 million (many users create multiple documents daily) |
| Average concurrent editors per document | 2 to 5 (design for up to 100 for shared class or team documents) |
| Operations per user per session | 100 keystrokes per minute × 30 minute session = 3000 operations per session |
| Total operations per day | 50M DAU × 3000 operations = 150 billion operations per day (~1.7 million per second on average) |
| Operation size | 50 bytes each |
| Raw operation log per day | 150B × 50 bytes = 7.5 TB/day |
| Operation log kept (30-day retention) | 7.5 TB × 30 = ~225 TB, before replication |
| Snapshot storage | Snapshots store text and formatting only (~50 KB). With a few snapshots for each actively edited document per day, roughly 30M snapshots × 50 KB ≈ 1.5 TB/day |
1.7 million operations per second is a lot of messages, so clients don't send one message per keystroke. They batch everything typed in the last ~100-200 ms into a single message, which cuts the message count by an order of magnitude without any visible delay.
The dominant challenge is not storage but correctness under concurrency. Ensuring all users converge to the same document state despite network delays, concurrent edits, and offline editing is the hardest distributed systems problem in this entire series.
3. API Gateway
All client requests enter through the API Gateway which handles authentication, rate limiting, and routing. The API Gateway handles standard REST requests for document CRUD, sharing, and version history. Real-time collaboration traffic bypasses the REST layer entirely and connects directly to the Collaboration Service via WebSocket after initial authentication.
The API Gateway enforces permission checks on document open requests before a WebSocket connection is established. Once the WebSocket session is active, permissions are checked via Redis cache on every operation rather than via the API Gateway, keeping the collaboration hot path off the Gateway entirely.
4. Operational Transformation
Operational Transformation (OT) is the conflict resolution algorithm at the heart of Google Docs. It is the core concept that makes real-time collaborative editing possible.
The Core Problem
When two users edit the same document simultaneously, their operations can conflict. Without conflict resolution, different users end up with different document states.
Example:
Document: "Hello"
User A: INSERT(2, "X") -> intends "HeXllo"
User B: INSERT(2, "Y") -> intends "HeYllo"
If applied naively in different orders, User A gets "HeXYllo" and User B gets "HeYXllo". The document has diverged.
OT solves this by transforming one operation against the other before applying it. Say A's insert was processed first. B's insert is then shifted past it to position 3: INSERT(3, "Y"). Both users end up with "HeXYllo". Consistent.
Two inserts at exactly the same position need a fixed tie-break rule (for example, the operation the server ordered first goes first, or the lower user ID wins). Every client must apply the same rule, otherwise they would make different choices and diverge again.
Server order: A's INSERT(2,"X") is version 9, B's INSERT(2,"Y") is version 10
B's op was written against version 8, so it is transformed past version 9:
INSERT(2,"Y") → INSERT(3,"Y")
Everyone applies: "Hello" → "HeXllo" → "HeXYllo"
Operation Structure
Every keystroke generates an operation with the following structure:
Operation {
op_id UUID
doc_id UUID
user_id UUID
type ENUM('insert', 'delete', 'format')
position INTEGER
character VARCHAR (null for delete and format)
format_type VARCHAR (null for insert and delete)
client_version INTEGER (document version the client had when generating this op)
server_version INTEGER (assigned by server after transformation, null until processed)
created_at TIMESTAMP
}
client_version is the most critical field. It tells the OT worker exactly which version of the document the client was looking at when they generated the operation. The OT worker transforms the incoming operation against all operations that happened between client_version and the current server_version. This is called the revision number approach.
Why Not Timestamps
Timestamps are unreliable for ordering in distributed systems. Two operations can have the same millisecond timestamp across different servers, and clocks drift. The monotonically incrementing server_version integer is the correct ordering mechanism. Every operation processed by the server increments the version by exactly 1. (One server deciding the order for each document is the same idea as a single leader deciding the order of writes.)
OT Algorithm (Simplified)
When the OT worker receives an operation from User B with client_version = 5 and the server is currently at server_version = 8:
- Fetch all operations with
server_versionbetween 6 and 8 from the operation log - Transform User B's operation against each of those 3 operations in order
- Each transformation adjusts positions to account for earlier inserts and deletes
- The final transformed operation is applied at
server_version = 9 - The transformed operation is broadcast to all other connected users
All clients converge to the same document state regardless of the order operations were received.
The Client Side of OT
The server is only half of the algorithm. While a user's own operation is on its way to the server, operations from other users keep arriving, and they were written against a document that doesn't contain the user's pending edit yet. So each client keeps its unacknowledged operations aside and transforms every incoming server operation against them before applying it. Google Docs keeps this manageable by allowing only one operation (or batch) in flight per client at a time: anything typed meanwhile is buffered and sent once the server acknowledges the previous one.
OT vs CRDTs
The main alternative to OT is a CRDT (Conflict-free Replicated Data Type), used by libraries like Yjs and Automerge. Instead of transforming positions, a CRDT gives every character a permanent unique ID, so concurrent edits can be merged in any order without a central server deciding the order. CRDTs are a natural fit for offline-first and peer-to-peer editing. OT needs a central server per document but keeps documents smaller and simpler, which is why Google Docs, with its always-present server, uses OT.
5. User and Auth Service
Manages user identity, authentication, and session management.
Data Model
Users Table (PostgreSQL)
Users
user_id UUID, primary key
email VARCHAR, unique
display_name VARCHAR
profile_pic_url TEXT
created_at TIMESTAMP
last_active TIMESTAMP
User sessions are managed via JWT tokens with short expiry (1 hour) and refresh tokens. The Collaboration Service validates the JWT on WebSocket connection establishment. Subsequent WebSocket messages do not re-validate the JWT, the connection itself is the authenticated session. Editing sessions often last longer than an hour, so when the token expires the client sends a refreshed token over the open connection, and the server closes connections that don't.
6. Document Service
The Document Service manages document metadata, creation, deletion, and sharing permissions. It handles standard REST operations separate from real-time collaboration.
Document Storage Architecture
Google Docs documents are significantly more complex than Medium articles. They contain nested rich text blocks, inline comments, suggestion mode edits, and formatting spans. This content is stored as a structured JSON document in MongoDB.
PostgreSQL stores document metadata. MongoDB stores document content. S3 stores embedded images referenced within documents.
Data Model
Documents Table (PostgreSQL)
Documents
doc_id UUID, primary key
owner_id UUID, foreign key -> Users
title VARCHAR
current_version INTEGER (monotonically incrementing server version)
content_mongo_id TEXT (MongoDB document ID for current content)
is_deleted BOOLEAN
created_at TIMESTAMP
updated_at TIMESTAMP
last_edited_by UUID, foreign key -> Users
DocumentPermissions Table (PostgreSQL)
DocumentPermissions
permission_id UUID, primary key
doc_id UUID, foreign key -> Documents
user_id UUID, foreign key -> Users (null for link-based sharing)
permission ENUM('owner', 'editor', 'commenter', 'viewer')
share_type ENUM('specific_user', 'link_anyone')
created_at TIMESTAMP
UNIQUE (doc_id, user_id)
(A primary key can't include a column that may be null, so link-based shares get their own permission_id, the same fix as in the Google Drive permission model.)
Permission Caching in Redis
Checking permissions on every WebSocket operation from the database would generate 1000 database hits per second per document with 100 concurrent editors each sending 10 operations per second. Instead permissions are cached in Redis:
Key: doc:permissions:{doc_id}:{user_id}
Value: ENUM('owner', 'editor', 'commenter', 'viewer')
TTL: 30 minutes
When a user opens a document, their permission is fetched from PostgreSQL and cached in Redis. Every subsequent WebSocket operation checks Redis only, sub-millisecond with no database involvement.
Permission Invalidation
If a document owner revokes a user's access mid-session, a permission.revoked event is published to Kafka. The Collaboration Service consumes this event and immediately closes the affected user's WebSocket connection. Redis cache is invalidated simultaneously. The revoked user loses access within seconds.
Document Content in MongoDB
{
"doc_id": "uuid",
"version": 1847,
"blocks": [
{ "id": "b1", "type": "heading", "level": 1, "text": "Project Proposal" },
{ "id": "b2", "type": "paragraph", "text": "This document outlines...",
"formats": [{ "start": 5, "end": 13, "type": "bold" }] },
{ "id": "b3", "type": "image", "url": "https://storage.googleapis.com/...",
"caption": "Architecture diagram" },
{ "id": "b4", "type": "paragraph", "text": "The system consists of..." }
],
"updated_at": "2026-08-11T10:00:00Z"
}
Each block has a unique ID that never changes even as content shifts. This makes partial document updates efficient since only modified blocks need to be rewritten.
Bottlenecks in the Document Service
Document metadata reads are cached in Redis since title and permission checks happen on every document open. MongoDB handles document content reads for the initial load and snapshot saves. The Document Service itself is not on the hot path during active editing, the Collaboration Service owns that.
7. Collaboration Service
The Collaboration Service is the most unique and complex service in this design. It manages all real-time editing activity: WebSocket connections, OT processing, operation broadcasting, cursor presence, and the snapshot plus operation log architecture.
WebSocket Connection Management
When a user opens a document for editing:
- Client authenticates via API Gateway (REST) and receives a session token
- Client establishes a WebSocket connection to the Collaboration Service:
WSS /docs/{doc_id}/collaborate - Collaboration Service validates the session token and checks permission from Redis
- Client joins the document's collaboration room
Each document has a collaboration room, a logical grouping of all WebSocket connections for that document. A document with 50 concurrent editors has 50 WebSocket connections all in the same room.
Multi-Server WebSocket Problem
WebSocket connections are stateful. User A is connected to Collaboration Server 1. User B is connected to Collaboration Server 3. When User A types a character, Collaboration Server 1 must broadcast the transformed operation to User B on Collaboration Server 3. The servers do not share memory.
The solution is Redis Pub/Sub, the same pattern used in chat services and LeetCode's verdict delivery:
Channel: doc:{doc_id}:operations
Every Collaboration Server subscribes to the Redis Pub/Sub channel for every active document on that server. When a transformed operation is ready to broadcast, it is published to the Redis channel. Every Collaboration Server subscribed to that channel receives it and pushes it to all connected users for that document.
Full Operation Flow: User A Presses "X"
sequenceDiagram
participant A as User A
participant CS as Collaboration Server
participant K as Kafka (by doc_id)
participant OT as OT Worker
participant P as Redis Pub/Sub
participant B as User B
A->>CS: INSERT(47, "X") @ v1523
CS->>CS: permission check (Redis)
CS->>K: doc.operations
K->>OT: consume (one worker per doc)
OT->>OT: transform against ops after v1523, assign v1524
OT->>P: publish op v1524
P->>CS: op v1524
CS-->>A: ack v1524
CS-->>B: apply op v1524
Step 1 - Operation Generated
Client generates operation:
{
"op_id": "uuid",
"doc_id": "doc_123",
"type": "insert",
"position": 47,
"character": "X",
"client_version": 1523
}
Step 2 - Operation Sent via WebSocket
Operation sent to Collaboration Server 1 via the open WebSocket connection.
Step 3 - Permission Check
Collaboration Service checks doc:permissions:{doc_id}:{user_id} in Redis. If not editor or owner, reject operation.
Step 4 - Operation Published to Kafka
Operation published to Kafka topic doc.operations, keyed by doc_id, for durability. Even if the Collaboration Service crashes at this point, the operation is not lost.
Step 5 - OT Worker Picks Up Operation
OT Worker consumes from Kafka. It needs every operation with server_version > client_version and transforms the incoming operation against each of those operations in order.
Because Kafka is partitioned by doc_id, exactly one OT Worker handles any given document at a time. That worker keeps the recent history of each of its documents in memory, so it reads these operations from memory rather than from Cassandra. This matters for correctness, not just speed: Cassandra writes happen asynchronously (Step 7), so the newest operations might not be in Cassandra yet, and transforming against an incomplete list would make documents diverge. Cassandra is only read for older history, for example when a worker starts up or takes over a document after a failover.
Step 6 - Server Version Assigned
Transformed operation is assigned the next server_version atomically using a Redis counter:
INCR doc:version:{doc_id}
This Redis INCR is atomic and monotonically incrementing. No two operations ever get the same server_version. (If the counter is ever lost, for example after a Redis failure, it must be restored from the highest version stored in the operation log before any new operation is numbered.)
Step 7 - Operation Persisted
Transformed operation written to Cassandra Operation Log asynchronously. The OT Worker applies it to its in-memory copy of the document and refreshes the document snapshot in Redis every few seconds rather than on every keystroke (rewriting a multi-KB document a million times a second would be wasteful):
Key: doc:snapshot:{doc_id}
Value: { content: current_state, server_version: 1524 }
Step 8 - Operation Broadcast
OT Worker publishes transformed operation to Redis Pub/Sub channel doc:{doc_id}:operations. All Collaboration Servers subscribed to this channel receive it and push it to all connected users for this document via their WebSocket connections.
Step 9 - Client Acknowledgement
User A receives their own operation back with server_version: 1524, which acts as the acknowledgement: it updates its local version and may now send its next buffered operation. User B receives the same operation, transforms it against any pending operation of their own (see the client side of OT above), and applies it. User B sees "X" appear in the document within milliseconds.
Cursor Presence
Cursor presence is simpler than OT, it is a pure broadcast problem with no conflict resolution needed.
When User A moves their cursor:
- Client sends
{ user_id, doc_id, position, color }to Collaboration Service - Collaboration Service publishes to Redis Pub/Sub:
doc:{doc_id}:cursors - All other users in the document receive the cursor update and render User A's cursor avatar at the correct position
Cursor positions are not persisted. They exist only in Redis Pub/Sub and client memory. If a user disconnects, their cursor disappears immediately. Each client also shifts the other users' cursors as operations arrive (an insert before someone's cursor moves it right), the same way it transforms text operations, so cursors stay on the right character between updates.
Key: doc:cursors:{doc_id}
Value: Hash { user_id: { position, color, display_name, last_seen } }
TTL: refreshed on each cursor update, expires 5 seconds after last update
Initial Document State for Joining Users
When a new user opens a document that 5 people are already editing:
- Collaboration Service subscribes to Redis Pub/Sub channel
doc:{doc_id}:operations - Fetches document snapshot from Redis:
doc:snapshot:{doc_id} - Serves snapshot to client (content +
server_version) - Any operations that arrive after the subscription are buffered
- Client applies buffered operations newer than the snapshot's version, in
server_versionorder, on top of the snapshot. Any version gap (an operation committed after the snapshot but published before the subscription) is fetched from the OT Worker's recent history or the operation log - Client is now fully synchronized with no gaps
The version number on every operation guarantees the client can detect and fill any gaps. Subscribing before reading the snapshot is what makes this safe: an operation can't slip through unseen between the two steps.
Bottlenecks in the Collaboration Service
The OT Worker is the primary bottleneck. Transforming an offline user's 300 queued operations against 300 server operations requires 300 × 300 = 90,000 transformation computations in the worst case. This is mitigated by horizontal scaling of the OT Worker pool via Kafka consumer groups. Each document's operations are partitioned by doc_id in Kafka, ensuring all operations for a single document are processed by the same OT Worker instance maintaining correct ordering.
8. Operation Log Service
The Operation Log Service manages the append-only log of all document operations and the periodic snapshot system.
Operation Storage in Cassandra
Cassandra is chosen for the operation log because:
- Extremely high write throughput (150 billion operations per day)
- Append-only write pattern maps perfectly to Cassandra's LSM tree storage
- Time-ordered retrieval by
server_versionis efficient with clustering keys - Horizontal scalability without complex sharding logic
Operations Table (Cassandra)
Operations
doc_id UUID, partition key
server_version INTEGER, clustering key (ascending)
op_id UUID
user_id UUID
type TEXT
position INTEGER
character TEXT
client_version INTEGER
created_at TIMESTAMP
Partitioning by doc_id means all operations for a document are stored together on the same Cassandra node, making version range queries extremely fast:
SELECT * FROM Operations
WHERE doc_id = 'doc_123'
AND server_version > 1500
ORDER BY server_version ASC;
A long-lived, heavily edited document can grow into a very large partition. Adding a version bucket to the partition key (for example, every 100,000 versions) keeps each partition a manageable size, the same hot partition fix as for big group chats.
Snapshot Strategy
A background Snapshot Worker triggers a full document snapshot:
- Every 30 minutes of active editing
- Every 1000 operations, whichever comes first
- When the last editor closes the document
Snapshot flow:
- Snapshot Worker reads current document state from Redis:
doc:snapshot:{doc_id} - Writes full document content to MongoDB Snapshots collection
- Records snapshot metadata in PostgreSQL:
Snapshots
snapshot_id UUID, primary key
doc_id UUID, foreign key -> Documents
mongo_content_id TEXT (MongoDB document ID)
server_version INTEGER
created_at TIMESTAMP
Operation Retention Policy
Raw operations are retained for 30 days in Cassandra. Beyond 30 days, only snapshots are kept. This caps the operation log at about 30 days of raw operations (~225 TB) instead of letting it grow by 7.5 TB every day forever. Version history older than 30 days is reconstructed from snapshots only, providing lower granularity but sufficient for most use cases.
9. Version History Service
The Version History Service allows users to view and restore previous versions of a document.
Point-in-Time Reconstruction
To reconstruct the document as it was on a specific date and time:
Step 1 - Find nearest snapshot before target time
SELECT * FROM Snapshots
WHERE doc_id = X
AND created_at <= target_timestamp
ORDER BY created_at DESC
LIMIT 1;
Step 2 - Fetch operations between snapshot and target time
SELECT * FROM Operations
WHERE doc_id = X
AND server_version > snapshot.server_version
AND created_at <= target_timestamp
ORDER BY server_version ASC;
Step 3 - Replay operations on snapshot
Start with the snapshot content and apply each operation in server_version order until the target timestamp is reached. The result is the exact document state at that point in time.
Step 4 - Cache reconstructed version
Key: doc:version:{doc_id}:{timestamp_bucket}
Value: reconstructed document content
TTL: 1 hour
Frequently viewed version history snapshots are cached in Redis so repeated requests for the same version do not require repeated Cassandra queries and operation replay.
Named Versions
Users can name specific versions ("Before client review", "Final draft"). Named versions store a direct pointer to a snapshot plus a specific server_version, bypassing the reconstruction step entirely for instant retrieval.
Restoring a version doesn't rewrite history. The service computes the operations that turn the current document into the old one and submits them like any other edit, so restoring is itself a new version that collaborators see live, and can be undone.
10. Offline Editing and Sync
Google Docs continues to work when the user loses internet connectivity. The client maintains a local operation queue in the browser's IndexedDB (persistent local storage available in all modern browsers).
Offline Flow
- User loses connectivity
- Client detects disconnection via WebSocket close event
- Client continues accepting user input and appending operations to the local IndexedDB queue
- Each offline operation is tagged with the
client_versionat the time of disconnection - User types for 5 minutes offline, generating 300 queued operations
Reconnection Flow
Step 1 - WebSocket Reconnects: Client re-establishes WebSocket connection to Collaboration Service and sends its last known client_version.
Step 2 - Server Sends Missed Operations: Collaboration Service fetches all operations with server_version > client_version (operations other users made while this user was offline). Sends them to the reconnecting client.
Step 3 - Client Rebases Its Offline Queue: Client applies the server's missed operations to its local state using OT, transforming them against the local offline queue. This also transforms the queued operations past the missed ones, so the queue now describes the user's edits as if they had been made on the latest server version.
Step 4 - Client Sends Offline Queue: Client sends the rebased offline operations to the server, tagged with the latest server_version it just received, not the version from the time of disconnection. (Sending the old version as well would make the server transform the same operations a second time against changes the client has already accounted for, corrupting the document.)
Step 5 - OT Worker Transforms Offline Operations: OT Worker transforms each offline operation against any server operations that arrived after that version, usually few or none. Same transformation process as real-time editing, just with a larger batch.
Step 6 - Transformed Operations Broadcast: Transformed offline operations are broadcast to all currently connected users. Document converges to the same state on all clients.
The version field on every operation makes offline sync mathematically equivalent to real-time sync. The server always knows exactly what state the client was in when each operation was generated.
11. Comment Service
Comments in Google Docs are anchored to specific text selections, not to fixed positions. If the commented text is edited, the comment moves with it.
Data Model
Comments Table (PostgreSQL)
Comments
comment_id UUID, primary key
doc_id UUID, foreign key -> Documents
author_id UUID, foreign key -> Users
anchor_start INTEGER (start position of the selection)
anchor_end INTEGER (end position of the selection)
anchor_version INTEGER (the server_version those positions refer to)
content TEXT
is_resolved BOOLEAN
created_at TIMESTAMP
A comment's anchor is treated exactly like a cursor or an operation: every edit that lands before or inside the selection transforms the anchor positions, so the range keeps covering the same text as the document changes around it. Storing the version the positions refer to means a client can always bring an old anchor up to date by transforming it through the operations since then. (Anchoring to the IDs of the operations that created the selected characters only works in a CRDT-style system where each character keeps a permanent ID. In OT, characters are identified by their position, so the anchor has to be transformed like everything else.)
CommentReplies Table (PostgreSQL)
CommentReplies
reply_id UUID, primary key
comment_id UUID, foreign key -> Comments
author_id UUID, foreign key -> Users
content TEXT
created_at TIMESTAMP
12. Notification Service
Same architecture as all previous systems, and as the Notification System design. Kafka fanout from upstream services, APNs for iOS, FCM for Android, email via SES.
The Notification Service subscribes to:
doc.shared-> "Harry shared a document with you" email + push notificationcomment.created-> comment notification to document owner and mentioned userscomment.replied-> reply notification to parent comment authorpermission.changed-> "Your access to this document has changed" notificationdoc.access_requested-> "Someone requested access to your document" to owner
13. Search Service
Google Docs search covers document titles and content. Unlike Medium where full content search was limited to a preview, Google Docs indexes full document content since users frequently search for specific phrases within their documents.
Document search index in Elasticsearch:
{
"doc_id": "123",
"title": "Project Proposal Q3 2026",
"content_text": "This document outlines the plan for...",
"owner_id": "456",
"owner_name": "Harry The Nerd",
"last_edited_at": "2026-08-11T10:00:00Z",
"shared_with": ["user_789", "user_101"]
}
Search results are filtered by shared_with to ensure users only see documents they have access to. The shared_with array is updated via Kafka consumer when permissions change.
Document content is synced to Elasticsearch from MongoDB snapshots via a background indexing worker, not on every keystroke. A document's search index is updated every time a new snapshot is taken (every 30 minutes of active editing or 1000 operations). Real-time keystroke-level search indexing is unnecessary and would overwhelm Elasticsearch.
14. Full Data Flow Summary
Document Open Flow:
Client -> API Gateway (auth) -> Document Service
-> Fetch document metadata from PostgreSQL
-> Check permission from Redis (doc:permissions:{doc_id}:{user_id})
-> Client establishes WebSocket to Collaboration Service
-> Subscribe to Redis Pub/Sub (doc:{doc_id}:operations)
-> Fetch document snapshot from Redis (doc:snapshot:{doc_id})
-> Return content + server_version to client
-> Buffer any operations arriving during connection setup
-> Apply buffered operations newer than the snapshot, in server_version order
Real-time Edit Flow:
User presses key -> client generates operation with client_version
-> WebSocket -> Collaboration Service
-> Redis permission check (sub-millisecond)
-> Kafka (doc.operations, partitioned by doc_id)
-> OT Worker (the only worker for this document):
-> Transform incoming op against ops after client_version (from its in-memory history)
-> Assign server_version via Redis INCR doc:version:{doc_id}
-> Write transformed op to Cassandra (async)
-> Refresh Redis snapshot every few seconds (doc:snapshot:{doc_id})
-> Publish to Redis Pub/Sub (doc:{doc_id}:operations)
-> All Collaboration Servers subscribed to channel receive op
-> Push to all connected users via WebSocket
-> All clients transform against their pending op, apply, and update their version
Cursor Presence Flow:
User moves cursor -> WebSocket -> Collaboration Service
-> Redis HSET doc:cursors:{doc_id} user_id {position, color}
-> Redis Pub/Sub publish (doc:{doc_id}:cursors)
-> All connected users receive cursor update
-> Render cursor avatar at new position
Offline Sync Flow:
User disconnects -> client queues ops in IndexedDB
User reconnects -> WebSocket re-established
-> Client sends last known client_version
-> Server sends missed ops (server_version > client_version)
-> Client applies missed ops and rebases its offline queue on them
-> Client sends rebased queue, tagged with the latest server_version
-> OT Worker transforms them against anything newer
-> Transformed ops broadcast to all connected users via Redis Pub/Sub
Snapshot Flow:
Every 30 mins or 1000 ops -> Snapshot Worker triggers
-> Read current state from Redis (doc:snapshot:{doc_id})
-> Write to MongoDB Snapshots collection
-> Record in PostgreSQL Snapshots Table
-> Trigger Elasticsearch reindex of document content
Version History Flow:
User requests document at target_timestamp
-> Version History Service
-> PostgreSQL: find nearest snapshot before target_timestamp
-> Cassandra: fetch ops between snapshot.server_version and target_timestamp
-> Replay ops on snapshot content
-> Cache result in Redis (doc:version:{doc_id}:{timestamp_bucket})
-> Return reconstructed document to client
Permission Revocation Flow:
Owner revokes access -> Document Service
-> DELETE doc:permissions:{doc_id}:{user_id} from Redis
-> Publish permission.revoked to Kafka
-> Collaboration Service closes affected user's WebSocket
-> User loses access within seconds
15. Resilience and Fault Tolerance
- OT algorithm correctness is the primary correctness guarantee. The version number on every operation ensures the OT worker always knows exactly which operations to transform against, regardless of network delays or out-of-order delivery, and one worker per document means there is always exactly one authority on the order.
- Kafka durability for all operations means no keystroke is ever lost even if the OT Worker or Collaboration Service crashes mid-processing. Operations are replayed from Kafka on worker restart.
- Redis snapshot as fast initial state means a user joining an active document never waits for a database query. The snapshot is served from in-memory Redis in sub-millisecond time.
- Cassandra operation log provides durable, ordered storage for all operations with high write throughput. Cassandra's replication factor of 3 ensures no operation is lost under node failures.
- MongoDB snapshot storage provides durable version history with flexible schema. Snapshot writes happen asynchronously so MongoDB slowdowns do not affect real-time editing performance.
- Offline editing with IndexedDB queue ensures no work is lost during connectivity loss. The version on every queued operation makes reconnection sync mathematically equivalent to real-time sync.
- Redis Pub/Sub for operation broadcast decouples Collaboration Servers from each other. A server crash only affects users connected to that server. They reconnect to another server within seconds and receive missed operations from their last known version.
- Permission caching with event-driven invalidation ensures permission checks are sub-millisecond in the hot path while access revocation propagates within seconds via Kafka events.
- Snapshot plus operation retention policy keeps storage manageable. Raw operations are retained for 30 days and then evicted. Snapshots are kept indefinitely providing coarser-grained version history beyond 30 days.
16. Technology Choices Summary
| Component | Technology | Why |
|---|---|---|
| API Gateway | Kong / AWS API Gateway | Auth, rate limiting, REST request routing |
| Document Metadata DB | PostgreSQL | Document metadata, permissions, snapshot references, strong consistency |
| Document Content Store | MongoDB | Rich text JSON blocks, flexible schema, large nested documents |
| Operation Log | Apache Cassandra | Append-only high write throughput, partition by doc_id, cluster by server_version |
| Snapshot Cache | Redis | Current document state + server_version, sub-millisecond initial load for joining users |
| Version Cache | Redis with TTL | Cached reconstructed point-in-time versions, 1-hour TTL |
| Real-time Transport | WebSockets | Persistent bidirectional connection, sub-100ms operation delivery |
| Operation Broadcast | Redis Pub/Sub | Cross-server operation and cursor broadcasting to all connected clients |
| Conflict Resolution | Operational Transformation | One OT Worker per document via Kafka partitioning, recent history in memory |
| Version Counter | Redis INCR | Atomic monotonically incrementing server_version per document |
| Permission Cache | Redis with TTL + event-driven invalidation | Sub-millisecond permission checks, Kafka-based revocation |
| Cursor Presence | Redis Hash with TTL | Per-document cursor state, auto-expires 5 seconds after last update |
| Offline Queue | Browser IndexedDB | Persistent local operation queue, survives tab close and connectivity loss |
| Search Engine | Elasticsearch | Full document content search, updated from MongoDB snapshots |
| Image Storage | AWS S3 | Embedded document images, CDN-native |
| CDN | CloudFront / Akamai | Edge caching for document images |
| Decoupling | Apache Kafka | Operation durability, permission events, snapshot triggers, notification fanout |
| Notification Delivery | APNs + FCM + AWS SES | Share notifications, comment alerts, access requests |