HLD · 24 min read

Design Google Docs

System design for a real-time collaborative platform like Google Docs

On this page
  1. 1. Requirements
  2. Functional Requirements
  3. Out of Scope
  4. Non-Functional Requirements
  5. 2. Capacity Estimation
  6. 3. API Gateway
  7. 4. Operational Transformation
  8. The Core Problem
  9. Operation Structure
  10. Why Not Timestamps
  11. OT Algorithm (Simplified)
  12. The Client Side of OT
  13. OT vs CRDTs
  14. 5. User and Auth Service
  15. Data Model
  16. 6. Document Service
  17. Document Storage Architecture
  18. Data Model
  19. Permission Caching in Redis
  20. Permission Invalidation
  21. Document Content in MongoDB
  22. Bottlenecks in the Document Service
  23. 7. Collaboration Service
  24. WebSocket Connection Management
  25. Multi-Server WebSocket Problem
  26. Full Operation Flow: User A Presses "X"
  27. Cursor Presence
  28. Initial Document State for Joining Users
  29. Bottlenecks in the Collaboration Service
  30. 8. Operation Log Service
  31. Operation Storage in Cassandra
  32. Operations Table (Cassandra)
  33. Snapshot Strategy
  34. Operation Retention Policy
  35. 9. Version History Service
  36. Point-in-Time Reconstruction
  37. Named Versions
  38. 10. Offline Editing and Sync
  39. Offline Flow
  40. Reconnection Flow
  41. 11. Comment Service
  42. Data Model
  43. 12. Notification Service
  44. 13. Search Service
  45. 14. Full Data Flow Summary
  46. 15. Resilience and Fault Tolerance
  47. 16. Technology Choices Summary

1. Requirements

Functional Requirements

  • Create, read, update, and delete documents
  • Real-time collaborative editing (multiple users editing simultaneously)
  • Cursor presence (see where other users are typing with their avatar)
  • Rich text formatting (bold, italic, headings, lists, code blocks)
  • Document sharing and permissions (view only, comment, edit, public link)
  • Comment threads on specific text selections
  • Version history (see and restore previous versions of a document)
  • Offline editing with automatic sync on reconnect

Out of Scope

  • Google Sheets (spreadsheet logic)
  • Google Slides (presentation logic)
  • Google Drive (file system management)
  • Export to PDF or Word
  • AI writing suggestions
  • Voice typing

Non-Functional Requirements

  • Strong consistency for document state: all users must end up seeing the same document content
  • Real-time operation delivery under 100ms for users in the same region
  • No data loss for offline edits: operations queued locally must survive reconnection
  • Conflict-free convergence: concurrent edits from multiple users must always produce the same final document state
  • Horizontal scalability for the collaboration layer, a single popular document can have 100 concurrent editors
  • Eventual consistency acceptable for version history reconstruction and search indexing

2. Capacity Estimation

MetricValue
DAU50 million
Average document size1 to 5 MB including formatting metadata and embedded images (the text and formatting alone are usually tens of KB, images live in S3)
Documents created per day10 million (many users create multiple documents daily)
Average concurrent editors per document2 to 5 (design for up to 100 for shared class or team documents)
Operations per user per session100 keystrokes per minute × 30 minute session = 3000 operations per session
Total operations per day50M DAU × 3000 operations = 150 billion operations per day (~1.7 million per second on average)
Operation size50 bytes each
Raw operation log per day150B × 50 bytes = 7.5 TB/day
Operation log kept (30-day retention)7.5 TB × 30 = ~225 TB, before replication
Snapshot storageSnapshots store text and formatting only (~50 KB). With a few snapshots for each actively edited document per day, roughly 30M snapshots × 50 KB ≈ 1.5 TB/day

1.7 million operations per second is a lot of messages, so clients don't send one message per keystroke. They batch everything typed in the last ~100-200 ms into a single message, which cuts the message count by an order of magnitude without any visible delay.

The dominant challenge is not storage but correctness under concurrency. Ensuring all users converge to the same document state despite network delays, concurrent edits, and offline editing is the hardest distributed systems problem in this entire series.

3. API Gateway

All client requests enter through the API Gateway which handles authentication, rate limiting, and routing. The API Gateway handles standard REST requests for document CRUD, sharing, and version history. Real-time collaboration traffic bypasses the REST layer entirely and connects directly to the Collaboration Service via WebSocket after initial authentication.

The API Gateway enforces permission checks on document open requests before a WebSocket connection is established. Once the WebSocket session is active, permissions are checked via Redis cache on every operation rather than via the API Gateway, keeping the collaboration hot path off the Gateway entirely.

4. Operational Transformation

Operational Transformation (OT) is the conflict resolution algorithm at the heart of Google Docs. It is the core concept that makes real-time collaborative editing possible.

The Core Problem

When two users edit the same document simultaneously, their operations can conflict. Without conflict resolution, different users end up with different document states.

Example:

text
Document: "Hello"
User A: INSERT(2, "X") -> intends "HeXllo"
User B: INSERT(2, "Y") -> intends "HeYllo"

If applied naively in different orders, User A gets "HeXYllo" and User B gets "HeYXllo". The document has diverged.

OT solves this by transforming one operation against the other before applying it. Say A's insert was processed first. B's insert is then shifted past it to position 3: INSERT(3, "Y"). Both users end up with "HeXYllo". Consistent.

Two inserts at exactly the same position need a fixed tie-break rule (for example, the operation the server ordered first goes first, or the lower user ID wins). Every client must apply the same rule, otherwise they would make different choices and diverge again.

text
Server order:  A's INSERT(2,"X") is version 9, B's INSERT(2,"Y") is version 10
B's op was written against version 8, so it is transformed past version 9:
    INSERT(2,"Y")  →  INSERT(3,"Y")
Everyone applies: "Hello" → "HeXllo" → "HeXYllo"

Operation Structure

Every keystroke generates an operation with the following structure:

text
Operation {
  op_id           UUID
  doc_id          UUID
  user_id         UUID
  type            ENUM('insert', 'delete', 'format')
  position        INTEGER
  character       VARCHAR (null for delete and format)
  format_type     VARCHAR (null for insert and delete)
  client_version  INTEGER (document version the client had when generating this op)
  server_version  INTEGER (assigned by server after transformation, null until processed)
  created_at      TIMESTAMP
}

client_version is the most critical field. It tells the OT worker exactly which version of the document the client was looking at when they generated the operation. The OT worker transforms the incoming operation against all operations that happened between client_version and the current server_version. This is called the revision number approach.

Why Not Timestamps

Timestamps are unreliable for ordering in distributed systems. Two operations can have the same millisecond timestamp across different servers, and clocks drift. The monotonically incrementing server_version integer is the correct ordering mechanism. Every operation processed by the server increments the version by exactly 1. (One server deciding the order for each document is the same idea as a single leader deciding the order of writes.)

OT Algorithm (Simplified)

When the OT worker receives an operation from User B with client_version = 5 and the server is currently at server_version = 8:

  1. Fetch all operations with server_version between 6 and 8 from the operation log
  2. Transform User B's operation against each of those 3 operations in order
  3. Each transformation adjusts positions to account for earlier inserts and deletes
  4. The final transformed operation is applied at server_version = 9
  5. The transformed operation is broadcast to all other connected users

All clients converge to the same document state regardless of the order operations were received.

The Client Side of OT

The server is only half of the algorithm. While a user's own operation is on its way to the server, operations from other users keep arriving, and they were written against a document that doesn't contain the user's pending edit yet. So each client keeps its unacknowledged operations aside and transforms every incoming server operation against them before applying it. Google Docs keeps this manageable by allowing only one operation (or batch) in flight per client at a time: anything typed meanwhile is buffered and sent once the server acknowledges the previous one.

OT vs CRDTs

The main alternative to OT is a CRDT (Conflict-free Replicated Data Type), used by libraries like Yjs and Automerge. Instead of transforming positions, a CRDT gives every character a permanent unique ID, so concurrent edits can be merged in any order without a central server deciding the order. CRDTs are a natural fit for offline-first and peer-to-peer editing. OT needs a central server per document but keeps documents smaller and simpler, which is why Google Docs, with its always-present server, uses OT.

5. User and Auth Service

Manages user identity, authentication, and session management.

Data Model

Users Table (PostgreSQL)

text
Users
user_id           UUID, primary key
email             VARCHAR, unique
display_name      VARCHAR
profile_pic_url   TEXT
created_at        TIMESTAMP
last_active       TIMESTAMP

User sessions are managed via JWT tokens with short expiry (1 hour) and refresh tokens. The Collaboration Service validates the JWT on WebSocket connection establishment. Subsequent WebSocket messages do not re-validate the JWT, the connection itself is the authenticated session. Editing sessions often last longer than an hour, so when the token expires the client sends a refreshed token over the open connection, and the server closes connections that don't.

6. Document Service

The Document Service manages document metadata, creation, deletion, and sharing permissions. It handles standard REST operations separate from real-time collaboration.

Document Storage Architecture

Google Docs documents are significantly more complex than Medium articles. They contain nested rich text blocks, inline comments, suggestion mode edits, and formatting spans. This content is stored as a structured JSON document in MongoDB.

PostgreSQL stores document metadata. MongoDB stores document content. S3 stores embedded images referenced within documents.

Data Model

Documents Table (PostgreSQL)

text
Documents
doc_id            UUID, primary key
owner_id          UUID, foreign key -> Users
title             VARCHAR
current_version   INTEGER (monotonically incrementing server version)
content_mongo_id  TEXT (MongoDB document ID for current content)
is_deleted        BOOLEAN
created_at        TIMESTAMP
updated_at        TIMESTAMP
last_edited_by    UUID, foreign key -> Users

DocumentPermissions Table (PostgreSQL)

text
DocumentPermissions
permission_id UUID, primary key
doc_id        UUID, foreign key -> Documents
user_id       UUID, foreign key -> Users (null for link-based sharing)
permission    ENUM('owner', 'editor', 'commenter', 'viewer')
share_type    ENUM('specific_user', 'link_anyone')
created_at    TIMESTAMP
UNIQUE (doc_id, user_id)

(A primary key can't include a column that may be null, so link-based shares get their own permission_id, the same fix as in the Google Drive permission model.)

Permission Caching in Redis

Checking permissions on every WebSocket operation from the database would generate 1000 database hits per second per document with 100 concurrent editors each sending 10 operations per second. Instead permissions are cached in Redis:

text
Key:   doc:permissions:{doc_id}:{user_id}
Value: ENUM('owner', 'editor', 'commenter', 'viewer')
TTL:   30 minutes

When a user opens a document, their permission is fetched from PostgreSQL and cached in Redis. Every subsequent WebSocket operation checks Redis only, sub-millisecond with no database involvement.

Permission Invalidation

If a document owner revokes a user's access mid-session, a permission.revoked event is published to Kafka. The Collaboration Service consumes this event and immediately closes the affected user's WebSocket connection. Redis cache is invalidated simultaneously. The revoked user loses access within seconds.

Document Content in MongoDB

json
{
  "doc_id": "uuid",
  "version": 1847,
  "blocks": [
    { "id": "b1", "type": "heading", "level": 1, "text": "Project Proposal" },
    { "id": "b2", "type": "paragraph", "text": "This document outlines...",
      "formats": [{ "start": 5, "end": 13, "type": "bold" }] },
    { "id": "b3", "type": "image", "url": "https://storage.googleapis.com/...",
      "caption": "Architecture diagram" },
    { "id": "b4", "type": "paragraph", "text": "The system consists of..." }
  ],
  "updated_at": "2026-08-11T10:00:00Z"
}

Each block has a unique ID that never changes even as content shifts. This makes partial document updates efficient since only modified blocks need to be rewritten.

Bottlenecks in the Document Service

Document metadata reads are cached in Redis since title and permission checks happen on every document open. MongoDB handles document content reads for the initial load and snapshot saves. The Document Service itself is not on the hot path during active editing, the Collaboration Service owns that.

7. Collaboration Service

The Collaboration Service is the most unique and complex service in this design. It manages all real-time editing activity: WebSocket connections, OT processing, operation broadcasting, cursor presence, and the snapshot plus operation log architecture.

WebSocket Connection Management

When a user opens a document for editing:

  1. Client authenticates via API Gateway (REST) and receives a session token
  2. Client establishes a WebSocket connection to the Collaboration Service: WSS /docs/{doc_id}/collaborate
  3. Collaboration Service validates the session token and checks permission from Redis
  4. Client joins the document's collaboration room

Each document has a collaboration room, a logical grouping of all WebSocket connections for that document. A document with 50 concurrent editors has 50 WebSocket connections all in the same room.

Multi-Server WebSocket Problem

WebSocket connections are stateful. User A is connected to Collaboration Server 1. User B is connected to Collaboration Server 3. When User A types a character, Collaboration Server 1 must broadcast the transformed operation to User B on Collaboration Server 3. The servers do not share memory.

The solution is Redis Pub/Sub, the same pattern used in chat services and LeetCode's verdict delivery:

text
Channel: doc:{doc_id}:operations

Every Collaboration Server subscribes to the Redis Pub/Sub channel for every active document on that server. When a transformed operation is ready to broadcast, it is published to the Redis channel. Every Collaboration Server subscribed to that channel receives it and pushes it to all connected users for that document.

Full Operation Flow: User A Presses "X"

mermaid
sequenceDiagram
    participant A as User A
    participant CS as Collaboration Server
    participant K as Kafka (by doc_id)
    participant OT as OT Worker
    participant P as Redis Pub/Sub
    participant B as User B
    A->>CS: INSERT(47, "X") @ v1523
    CS->>CS: permission check (Redis)
    CS->>K: doc.operations
    K->>OT: consume (one worker per doc)
    OT->>OT: transform against ops after v1523, assign v1524
    OT->>P: publish op v1524
    P->>CS: op v1524
    CS-->>A: ack v1524
    CS-->>B: apply op v1524

Step 1 - Operation Generated

Client generates operation:

json
{
  "op_id": "uuid",
  "doc_id": "doc_123",
  "type": "insert",
  "position": 47,
  "character": "X",
  "client_version": 1523
}

Step 2 - Operation Sent via WebSocket

Operation sent to Collaboration Server 1 via the open WebSocket connection.

Step 3 - Permission Check

Collaboration Service checks doc:permissions:{doc_id}:{user_id} in Redis. If not editor or owner, reject operation.

Step 4 - Operation Published to Kafka

Operation published to Kafka topic doc.operations, keyed by doc_id, for durability. Even if the Collaboration Service crashes at this point, the operation is not lost.

Step 5 - OT Worker Picks Up Operation

OT Worker consumes from Kafka. It needs every operation with server_version > client_version and transforms the incoming operation against each of those operations in order.

Because Kafka is partitioned by doc_id, exactly one OT Worker handles any given document at a time. That worker keeps the recent history of each of its documents in memory, so it reads these operations from memory rather than from Cassandra. This matters for correctness, not just speed: Cassandra writes happen asynchronously (Step 7), so the newest operations might not be in Cassandra yet, and transforming against an incomplete list would make documents diverge. Cassandra is only read for older history, for example when a worker starts up or takes over a document after a failover.

Step 6 - Server Version Assigned

Transformed operation is assigned the next server_version atomically using a Redis counter:

redis
INCR doc:version:{doc_id}

This Redis INCR is atomic and monotonically incrementing. No two operations ever get the same server_version. (If the counter is ever lost, for example after a Redis failure, it must be restored from the highest version stored in the operation log before any new operation is numbered.)

Step 7 - Operation Persisted

Transformed operation written to Cassandra Operation Log asynchronously. The OT Worker applies it to its in-memory copy of the document and refreshes the document snapshot in Redis every few seconds rather than on every keystroke (rewriting a multi-KB document a million times a second would be wasteful):

text
Key:   doc:snapshot:{doc_id}
Value: { content: current_state, server_version: 1524 }

Step 8 - Operation Broadcast

OT Worker publishes transformed operation to Redis Pub/Sub channel doc:{doc_id}:operations. All Collaboration Servers subscribed to this channel receive it and push it to all connected users for this document via their WebSocket connections.

Step 9 - Client Acknowledgement

User A receives their own operation back with server_version: 1524, which acts as the acknowledgement: it updates its local version and may now send its next buffered operation. User B receives the same operation, transforms it against any pending operation of their own (see the client side of OT above), and applies it. User B sees "X" appear in the document within milliseconds.

Cursor Presence

Cursor presence is simpler than OT, it is a pure broadcast problem with no conflict resolution needed.

When User A moves their cursor:

  1. Client sends { user_id, doc_id, position, color } to Collaboration Service
  2. Collaboration Service publishes to Redis Pub/Sub: doc:{doc_id}:cursors
  3. All other users in the document receive the cursor update and render User A's cursor avatar at the correct position

Cursor positions are not persisted. They exist only in Redis Pub/Sub and client memory. If a user disconnects, their cursor disappears immediately. Each client also shifts the other users' cursors as operations arrive (an insert before someone's cursor moves it right), the same way it transforms text operations, so cursors stay on the right character between updates.

text
Key:   doc:cursors:{doc_id}
Value: Hash { user_id: { position, color, display_name, last_seen } }
TTL:   refreshed on each cursor update, expires 5 seconds after last update

Initial Document State for Joining Users

When a new user opens a document that 5 people are already editing:

  1. Collaboration Service subscribes to Redis Pub/Sub channel doc:{doc_id}:operations
  2. Fetches document snapshot from Redis: doc:snapshot:{doc_id}
  3. Serves snapshot to client (content + server_version)
  4. Any operations that arrive after the subscription are buffered
  5. Client applies buffered operations newer than the snapshot's version, in server_version order, on top of the snapshot. Any version gap (an operation committed after the snapshot but published before the subscription) is fetched from the OT Worker's recent history or the operation log
  6. Client is now fully synchronized with no gaps

The version number on every operation guarantees the client can detect and fill any gaps. Subscribing before reading the snapshot is what makes this safe: an operation can't slip through unseen between the two steps.

Bottlenecks in the Collaboration Service

The OT Worker is the primary bottleneck. Transforming an offline user's 300 queued operations against 300 server operations requires 300 × 300 = 90,000 transformation computations in the worst case. This is mitigated by horizontal scaling of the OT Worker pool via Kafka consumer groups. Each document's operations are partitioned by doc_id in Kafka, ensuring all operations for a single document are processed by the same OT Worker instance maintaining correct ordering.

8. Operation Log Service

The Operation Log Service manages the append-only log of all document operations and the periodic snapshot system.

Operation Storage in Cassandra

Cassandra is chosen for the operation log because:

  • Extremely high write throughput (150 billion operations per day)
  • Append-only write pattern maps perfectly to Cassandra's LSM tree storage
  • Time-ordered retrieval by server_version is efficient with clustering keys
  • Horizontal scalability without complex sharding logic

Operations Table (Cassandra)

text
Operations
doc_id          UUID, partition key
server_version  INTEGER, clustering key (ascending)
op_id           UUID
user_id         UUID
type            TEXT
position        INTEGER
character       TEXT
client_version  INTEGER
created_at      TIMESTAMP

Partitioning by doc_id means all operations for a document are stored together on the same Cassandra node, making version range queries extremely fast:

sql
SELECT * FROM Operations
WHERE doc_id = 'doc_123'
  AND server_version > 1500
ORDER BY server_version ASC;

A long-lived, heavily edited document can grow into a very large partition. Adding a version bucket to the partition key (for example, every 100,000 versions) keeps each partition a manageable size, the same hot partition fix as for big group chats.

Snapshot Strategy

A background Snapshot Worker triggers a full document snapshot:

  • Every 30 minutes of active editing
  • Every 1000 operations, whichever comes first
  • When the last editor closes the document

Snapshot flow:

  1. Snapshot Worker reads current document state from Redis: doc:snapshot:{doc_id}
  2. Writes full document content to MongoDB Snapshots collection
  3. Records snapshot metadata in PostgreSQL:
text
Snapshots
snapshot_id       UUID, primary key
doc_id            UUID, foreign key -> Documents
mongo_content_id  TEXT (MongoDB document ID)
server_version    INTEGER
created_at        TIMESTAMP

Operation Retention Policy

Raw operations are retained for 30 days in Cassandra. Beyond 30 days, only snapshots are kept. This caps the operation log at about 30 days of raw operations (~225 TB) instead of letting it grow by 7.5 TB every day forever. Version history older than 30 days is reconstructed from snapshots only, providing lower granularity but sufficient for most use cases.

9. Version History Service

The Version History Service allows users to view and restore previous versions of a document.

Point-in-Time Reconstruction

To reconstruct the document as it was on a specific date and time:

Step 1 - Find nearest snapshot before target time

sql
SELECT * FROM Snapshots
WHERE doc_id = X
  AND created_at <= target_timestamp
ORDER BY created_at DESC
LIMIT 1;

Step 2 - Fetch operations between snapshot and target time

sql
SELECT * FROM Operations
WHERE doc_id = X
  AND server_version > snapshot.server_version
  AND created_at <= target_timestamp
ORDER BY server_version ASC;

Step 3 - Replay operations on snapshot

Start with the snapshot content and apply each operation in server_version order until the target timestamp is reached. The result is the exact document state at that point in time.

Step 4 - Cache reconstructed version

text
Key:   doc:version:{doc_id}:{timestamp_bucket}
Value: reconstructed document content
TTL:   1 hour

Frequently viewed version history snapshots are cached in Redis so repeated requests for the same version do not require repeated Cassandra queries and operation replay.

Named Versions

Users can name specific versions ("Before client review", "Final draft"). Named versions store a direct pointer to a snapshot plus a specific server_version, bypassing the reconstruction step entirely for instant retrieval.

Restoring a version doesn't rewrite history. The service computes the operations that turn the current document into the old one and submits them like any other edit, so restoring is itself a new version that collaborators see live, and can be undone.

10. Offline Editing and Sync

Google Docs continues to work when the user loses internet connectivity. The client maintains a local operation queue in the browser's IndexedDB (persistent local storage available in all modern browsers).

Offline Flow

  1. User loses connectivity
  2. Client detects disconnection via WebSocket close event
  3. Client continues accepting user input and appending operations to the local IndexedDB queue
  4. Each offline operation is tagged with the client_version at the time of disconnection
  5. User types for 5 minutes offline, generating 300 queued operations

Reconnection Flow

Step 1 - WebSocket Reconnects: Client re-establishes WebSocket connection to Collaboration Service and sends its last known client_version.

Step 2 - Server Sends Missed Operations: Collaboration Service fetches all operations with server_version > client_version (operations other users made while this user was offline). Sends them to the reconnecting client.

Step 3 - Client Rebases Its Offline Queue: Client applies the server's missed operations to its local state using OT, transforming them against the local offline queue. This also transforms the queued operations past the missed ones, so the queue now describes the user's edits as if they had been made on the latest server version.

Step 4 - Client Sends Offline Queue: Client sends the rebased offline operations to the server, tagged with the latest server_version it just received, not the version from the time of disconnection. (Sending the old version as well would make the server transform the same operations a second time against changes the client has already accounted for, corrupting the document.)

Step 5 - OT Worker Transforms Offline Operations: OT Worker transforms each offline operation against any server operations that arrived after that version, usually few or none. Same transformation process as real-time editing, just with a larger batch.

Step 6 - Transformed Operations Broadcast: Transformed offline operations are broadcast to all currently connected users. Document converges to the same state on all clients.

The version field on every operation makes offline sync mathematically equivalent to real-time sync. The server always knows exactly what state the client was in when each operation was generated.

11. Comment Service

Comments in Google Docs are anchored to specific text selections, not to fixed positions. If the commented text is edited, the comment moves with it.

Data Model

Comments Table (PostgreSQL)

text
Comments
comment_id        UUID, primary key
doc_id            UUID, foreign key -> Documents
author_id         UUID, foreign key -> Users
anchor_start      INTEGER (start position of the selection)
anchor_end        INTEGER (end position of the selection)
anchor_version    INTEGER (the server_version those positions refer to)
content           TEXT
is_resolved       BOOLEAN
created_at        TIMESTAMP

A comment's anchor is treated exactly like a cursor or an operation: every edit that lands before or inside the selection transforms the anchor positions, so the range keeps covering the same text as the document changes around it. Storing the version the positions refer to means a client can always bring an old anchor up to date by transforming it through the operations since then. (Anchoring to the IDs of the operations that created the selected characters only works in a CRDT-style system where each character keeps a permanent ID. In OT, characters are identified by their position, so the anchor has to be transformed like everything else.)

CommentReplies Table (PostgreSQL)

text
CommentReplies
reply_id      UUID, primary key
comment_id    UUID, foreign key -> Comments
author_id     UUID, foreign key -> Users
content       TEXT
created_at    TIMESTAMP

12. Notification Service

Same architecture as all previous systems, and as the Notification System design. Kafka fanout from upstream services, APNs for iOS, FCM for Android, email via SES.

The Notification Service subscribes to:

  • doc.shared -> "Harry shared a document with you" email + push notification
  • comment.created -> comment notification to document owner and mentioned users
  • comment.replied -> reply notification to parent comment author
  • permission.changed -> "Your access to this document has changed" notification
  • doc.access_requested -> "Someone requested access to your document" to owner

13. Search Service

Google Docs search covers document titles and content. Unlike Medium where full content search was limited to a preview, Google Docs indexes full document content since users frequently search for specific phrases within their documents.

Document search index in Elasticsearch:

json
{
  "doc_id": "123",
  "title": "Project Proposal Q3 2026",
  "content_text": "This document outlines the plan for...",
  "owner_id": "456",
  "owner_name": "Harry The Nerd",
  "last_edited_at": "2026-08-11T10:00:00Z",
  "shared_with": ["user_789", "user_101"]
}

Search results are filtered by shared_with to ensure users only see documents they have access to. The shared_with array is updated via Kafka consumer when permissions change.

Document content is synced to Elasticsearch from MongoDB snapshots via a background indexing worker, not on every keystroke. A document's search index is updated every time a new snapshot is taken (every 30 minutes of active editing or 1000 operations). Real-time keystroke-level search indexing is unnecessary and would overwhelm Elasticsearch.

14. Full Data Flow Summary

Document Open Flow:

text
Client -> API Gateway (auth) -> Document Service
    -> Fetch document metadata from PostgreSQL
    -> Check permission from Redis (doc:permissions:{doc_id}:{user_id})
    -> Client establishes WebSocket to Collaboration Service
    -> Subscribe to Redis Pub/Sub (doc:{doc_id}:operations)
    -> Fetch document snapshot from Redis (doc:snapshot:{doc_id})
    -> Return content + server_version to client
    -> Buffer any operations arriving during connection setup
    -> Apply buffered operations newer than the snapshot, in server_version order

Real-time Edit Flow:

text
User presses key -> client generates operation with client_version
    -> WebSocket -> Collaboration Service
    -> Redis permission check (sub-millisecond)
    -> Kafka (doc.operations, partitioned by doc_id)
    -> OT Worker (the only worker for this document):
        -> Transform incoming op against ops after client_version (from its in-memory history)
        -> Assign server_version via Redis INCR doc:version:{doc_id}
        -> Write transformed op to Cassandra (async)
        -> Refresh Redis snapshot every few seconds (doc:snapshot:{doc_id})
        -> Publish to Redis Pub/Sub (doc:{doc_id}:operations)
    -> All Collaboration Servers subscribed to channel receive op
    -> Push to all connected users via WebSocket
    -> All clients transform against their pending op, apply, and update their version

Cursor Presence Flow:

text
User moves cursor -> WebSocket -> Collaboration Service
    -> Redis HSET doc:cursors:{doc_id} user_id {position, color}
    -> Redis Pub/Sub publish (doc:{doc_id}:cursors)
    -> All connected users receive cursor update
    -> Render cursor avatar at new position

Offline Sync Flow:

text
User disconnects -> client queues ops in IndexedDB
User reconnects -> WebSocket re-established
    -> Client sends last known client_version
    -> Server sends missed ops (server_version > client_version)
    -> Client applies missed ops and rebases its offline queue on them
    -> Client sends rebased queue, tagged with the latest server_version
    -> OT Worker transforms them against anything newer
    -> Transformed ops broadcast to all connected users via Redis Pub/Sub

Snapshot Flow:

text
Every 30 mins or 1000 ops -> Snapshot Worker triggers
    -> Read current state from Redis (doc:snapshot:{doc_id})
    -> Write to MongoDB Snapshots collection
    -> Record in PostgreSQL Snapshots Table
    -> Trigger Elasticsearch reindex of document content

Version History Flow:

text
User requests document at target_timestamp
    -> Version History Service
    -> PostgreSQL: find nearest snapshot before target_timestamp
    -> Cassandra: fetch ops between snapshot.server_version and target_timestamp
    -> Replay ops on snapshot content
    -> Cache result in Redis (doc:version:{doc_id}:{timestamp_bucket})
    -> Return reconstructed document to client

Permission Revocation Flow:

text
Owner revokes access -> Document Service
    -> DELETE doc:permissions:{doc_id}:{user_id} from Redis
    -> Publish permission.revoked to Kafka
    -> Collaboration Service closes affected user's WebSocket
    -> User loses access within seconds

15. Resilience and Fault Tolerance

  • OT algorithm correctness is the primary correctness guarantee. The version number on every operation ensures the OT worker always knows exactly which operations to transform against, regardless of network delays or out-of-order delivery, and one worker per document means there is always exactly one authority on the order.
  • Kafka durability for all operations means no keystroke is ever lost even if the OT Worker or Collaboration Service crashes mid-processing. Operations are replayed from Kafka on worker restart.
  • Redis snapshot as fast initial state means a user joining an active document never waits for a database query. The snapshot is served from in-memory Redis in sub-millisecond time.
  • Cassandra operation log provides durable, ordered storage for all operations with high write throughput. Cassandra's replication factor of 3 ensures no operation is lost under node failures.
  • MongoDB snapshot storage provides durable version history with flexible schema. Snapshot writes happen asynchronously so MongoDB slowdowns do not affect real-time editing performance.
  • Offline editing with IndexedDB queue ensures no work is lost during connectivity loss. The version on every queued operation makes reconnection sync mathematically equivalent to real-time sync.
  • Redis Pub/Sub for operation broadcast decouples Collaboration Servers from each other. A server crash only affects users connected to that server. They reconnect to another server within seconds and receive missed operations from their last known version.
  • Permission caching with event-driven invalidation ensures permission checks are sub-millisecond in the hot path while access revocation propagates within seconds via Kafka events.
  • Snapshot plus operation retention policy keeps storage manageable. Raw operations are retained for 30 days and then evicted. Snapshots are kept indefinitely providing coarser-grained version history beyond 30 days.

16. Technology Choices Summary

ComponentTechnologyWhy
API GatewayKong / AWS API GatewayAuth, rate limiting, REST request routing
Document Metadata DBPostgreSQLDocument metadata, permissions, snapshot references, strong consistency
Document Content StoreMongoDBRich text JSON blocks, flexible schema, large nested documents
Operation LogApache CassandraAppend-only high write throughput, partition by doc_id, cluster by server_version
Snapshot CacheRedisCurrent document state + server_version, sub-millisecond initial load for joining users
Version CacheRedis with TTLCached reconstructed point-in-time versions, 1-hour TTL
Real-time TransportWebSocketsPersistent bidirectional connection, sub-100ms operation delivery
Operation BroadcastRedis Pub/SubCross-server operation and cursor broadcasting to all connected clients
Conflict ResolutionOperational TransformationOne OT Worker per document via Kafka partitioning, recent history in memory
Version CounterRedis INCRAtomic monotonically incrementing server_version per document
Permission CacheRedis with TTL + event-driven invalidationSub-millisecond permission checks, Kafka-based revocation
Cursor PresenceRedis Hash with TTLPer-document cursor state, auto-expires 5 seconds after last update
Offline QueueBrowser IndexedDBPersistent local operation queue, survives tab close and connectivity loss
Search EngineElasticsearchFull document content search, updated from MongoDB snapshots
Image StorageAWS S3Embedded document images, CDN-native
CDNCloudFront / AkamaiEdge caching for document images
DecouplingApache KafkaOperation durability, permission events, snapshot triggers, notification fanout
Notification DeliveryAPNs + FCM + AWS SESShare notifications, comment alerts, access requests