HLD · 26 min read

Design Google Drive

System design of a cloud storage service like Google Drive

On this page
  1. 1. Requirements
  2. Functional Requirements
  3. Out of Scope
  4. Non-Functional Requirements
  5. 2. Capacity Estimation
  6. 3. API Gateway
  7. 4. Upload Service and Block Service
  8. Small File Upload (under 5 MB)
  9. Large File Upload (over 5 MB) - Resumable Upload
  10. Encryption
  11. Bottlenecks in the Upload Pipeline
  12. 5. Deduplication Service
  13. Hash the Plaintext, Not the Ciphertext
  14. Content-Based Deduplication via SHA-256 Hashing
  15. S3Objects Table (PostgreSQL)
  16. Reference Counting for Safe Deletion
  17. Security and Privacy
  18. Bottlenecks in the Deduplication Service
  19. 6. File Metadata Service
  20. Data Model
  21. Folder Hierarchy - Multi-Parent Adjacency List
  22. Caching Strategy
  23. Bottlenecks in the File Metadata Service
  24. 7. Permission Service
  25. Permission Model
  26. Permission Inheritance
  27. Permission Revocation
  28. Bottlenecks in the Permission Service
  29. 8. Download Service
  30. Download Flow
  31. CDN Layer
  32. Small vs Large File Handling
  33. Bottlenecks in the Download Service
  34. 9. Version History Service
  35. Data Model
  36. Storage Tiering for Old Versions
  37. Version Restoration
  38. Deduplication and Versioning Interaction
  39. 10. Quota Service
  40. Quota Tracking
  41. Quota Enforcement
  42. Quota and Deduplication
  43. Quota and Deletion
  44. 11. Search Service
  45. Text Extraction Pipeline
  46. Elasticsearch Document Model
  47. Filter Support
  48. Security Filtering
  49. Keeping Elasticsearch in Sync
  50. 12. Notification Service
  51. 13. Full Data Flow Summary
  52. 14. Resilience and Fault Tolerance
  53. 15. Technology Choices Summary

1. Requirements

Functional Requirements

  • File upload and download (any file type: documents, images, videos, audio, PDFs, zip files)
  • Folder hierarchy and organization
  • File and folder sharing with role-based access (viewer, commenter, editor, owner)
  • File versioning (see and restore previous versions)
  • Search for files and folders (by name and by content)
  • Notifications (share alerts, file update notifications)
  • Trash and restore deleted files
  • Storage quota management per user

Out of Scope

  • Google Docs real-time collaborative editing (separate system, already designed)
  • Google Photos (separate product)
  • Third-party app integrations
  • Team Drives and Shared Drives (enterprise feature)
  • Offline sync client

Non-Functional Requirements

  • High availability for file upload and download
  • Resumable uploads, interrupted uploads must resume without restarting
  • No data loss, every uploaded file must be durable across hardware failures
  • Strong consistency for file metadata and permissions
  • Eventual consistency acceptable for search indexing and notifications
  • Storage deduplication to avoid storing identical files multiple times
  • Storage tiering, old file versions move to cheaper cold storage automatically
  • Permission changes must propagate immediately, revoked access must stop working within seconds

2. Capacity Estimation

MetricValue
DAU10 million
Files uploaded per user per day2
Average file size500 KB
Raw storage per day10M × 2 × 500 KB = 10 TB/day
After 40% deduplication60% × 10 TB = 6 TB/day effective new storage
Read/write ratio10:1, files are uploaded once and accessed many times across devices and shared users
Storage tieringCurrent versions in S3 Standard, versions older than 30 days in S3 Infrequent Access, versions older than 90 days in S3 Glacier
Quota per user15 GB free, paid tiers for more

The dominant design challenge is correctness across three dimensions: resumable uploads that survive network failures, deduplication that saves storage without leaking information between users, and permission enforcement that propagates instantly when access is revoked.

3. API Gateway

All client requests enter through the API Gateway which handles authentication, rate limiting, and routing. The API Gateway enforces authentication on every request before any file operation reaches downstream services.

One important distinction from previous systems: the API Gateway handles metadata operations (create folder, rename file, share file) but never handles binary file payloads directly. File uploads go through a presigned URL pattern for small files and a resumable chunked upload pattern for large files. The API Gateway only orchestrates the session creation and tracks upload state, never touching the raw bytes.

4. Upload Service and Block Service

Small File Upload (under 5 MB)

For small files, the standard presigned S3 URL pattern applies:

  1. Client sends POST /api/v1/files/upload with file metadata (name, size, MIME type) to the Upload Service via API Gateway
  2. Upload Service generates a presigned S3 URL
  3. Client uploads file directly to S3 (S3 encrypts it at rest with a KMS-managed key, and can also record a SHA-256 checksum of the content as it arrives)
  4. S3 triggers an upload event, bridged into Kafka
  5. File Assembly Service picks up the event and triggers deduplication

Large File Upload (over 5 MB) - Resumable Upload

Large files are split into chunks on the client side before upload. This enables resumable uploads where interrupted transfers can continue from the last successful chunk.

mermaid
sequenceDiagram
    participant C as Client
    participant U as Upload Service
    participant B as Block Service
    participant R as Redis
    participant S as S3
    C->>U: create session
    U-->>C: session_id
    loop each 5 MB chunk
        C->>B: PUT chunk N
        B->>S: upload part N (encrypted)
        B->>R: SADD completed N
    end
    Note over C,B: connection drops, then...
    C->>U: GET status
    U->>R: SMEMBERS completed
    U-->>C: [0, 1, 2, 4]
    C->>B: resend missing chunks only
    C->>U: complete
    U->>S: CompleteMultipartUpload

Step 1 - Create Upload Session

http
POST /api/v1/files/upload?uploadType=resumable
Body: { file_name, file_size, mime_type, parent_folder_id }

Upload Service creates an upload session record (and starts an S3 multipart upload for it) and returns a session_id:

text
UploadSessions
session_id        UUID, primary key
user_id           UUID, foreign key -> Users
file_name         VARCHAR
file_size         BIGINT
mime_type         VARCHAR
parent_folder_id  UUID
total_chunks      INTEGER
chunk_size        INTEGER (5 MB default)
s3_upload_id      VARCHAR (the S3 multipart upload this session writes into)
status            ENUM('in_progress', 'completed', 'expired')
expires_at        TIMESTAMP (24 hours from last activity)
created_at        TIMESTAMP

Uploaded chunk tracking is stored in Redis for fast access:

text
Key:   upload:{session_id}:chunks
Value: SET of completed chunk indices
TTL:   24 hours

Step 2 - Upload Chunks

Client splits file into 5 MB chunks and uploads each independently:

http
PUT /api/v1/files/upload/{session_id}/chunk/{chunk_index}
Content-Range: bytes 0-5242879/10737418240
Body: raw chunk bytes

The Block Service receives each chunk and:

  1. Computes the SHA-256 hash of this chunk's plaintext and stores it with the session (chunks can arrive in any order, so each is hashed on its own and the file's hash is built from the chunk hashes, see Section 5)
  2. Encrypts the chunk using AES-256 with the file's encryption key
  3. Uploads the encrypted chunk to S3 as part number chunk_index + 1 of the session's multipart upload
  4. Records completion: SADD upload:{session_id}:chunks {chunk_index}
  5. Returns success to client

(S3 multipart uploads require every part except the last to be at least 5 MB, which is why 5 MB is the chunk size.)

Step 3 - Resume After Interruption

If the upload is interrupted, the client queries upload status:

http
GET /api/v1/files/upload/{session_id}/status

Response: { completed_chunks: [0, 1, 2, 4], total_chunks: 10 }

Client identifies missing chunks (3, 5, 6, 7, 8, 9) and resumes uploading from the first missing chunk. No previously uploaded chunks need to be re-sent.

Step 4 - Complete Upload

When all chunks are uploaded, client sends:

http
POST /api/v1/files/upload/{session_id}/complete

File Assembly Service picks this up and calls S3's CompleteMultipartUpload, which stitches all the uploaded parts into a single S3 object in order. The upload session is marked complete. (The same multipart approach powers large uploads in the YouTube design.)

Encryption

Every file is encrypted at rest. The Block Service uses envelope encryption:

  1. A unique Data Encryption Key (DEK) is generated per stored object
  2. File chunks are encrypted with the DEK using AES-256
  3. The DEK itself is encrypted with a master Key Encryption Key (KEK) managed by AWS KMS
  4. Only the encrypted DEK is stored in the database, never the plaintext DEK

When a file is downloaded, the Download Service decrypts the DEK using KMS and uses it to decrypt the file chunks before serving them to the client.

text
File chunk ──AES-256 with DEK──► encrypted chunk (stored in S3)
DEK        ──encrypted by KEK in KMS──► encrypted DEK (stored in the DB)

Bottlenecks in the Upload Pipeline

The Block Service is the primary bottleneck since it handles encryption for every chunk of every file. Encryption is CPU-intensive. This is mitigated by horizontal scaling of Block Service instances via Kubernetes autoscaling based on CPU utilization. The upload session state in Redis is fast and not a concern. S3 multipart upload handles chunk assembly natively without additional bottlenecks.

5. Deduplication Service

The Deduplication Service prevents storing identical file content multiple times in S3, saving significant storage costs at scale.

Hash the Plaintext, Not the Ciphertext

Deduplication and encryption pull in opposite directions. Encryption with a random key makes two identical files look completely different once encrypted, so hashing the stored (encrypted) object would never find a match. That's why the hash is computed on the plaintext, as the chunks pass through the Block Service and before they are encrypted. The hash identifies the content, and the encryption key belongs to the stored object (the S3Objects row below), so every user file that points at that object can be decrypted with the same key.

Content-Based Deduplication via SHA-256 Hashing

When the File Assembly Service completes chunk assembly:

  1. Deduplication Service builds the file's content hash from the plaintext hashes of its chunks (for small files uploaded straight to S3, from the checksum S3 recorded on arrival)

  2. Queries the S3Objects table:

    sql
    SELECT s3_url, reference_count FROM S3Objects
    WHERE content_hash = 'abc123...'
    FOR UPDATE;
    
  3. If hash exists: the file is a duplicate. Create a new file metadata record pointing to the existing S3 object. Increment reference_count. Delete the newly assembled object from S3 (that content was already stored).

  4. If hash does not exist: the file is new. Move it from the temporary upload path to the permanent path. Create a new S3Objects record with reference_count = 1.

S3Objects Table (PostgreSQL)

text
S3Objects
content_hash      VARCHAR(64), primary key (SHA-256 hex of the plaintext)
s3_url            TEXT (permanent S3 path)
encrypted_dek     BYTEA (this object's data key, encrypted by the KMS master key)
file_size         BIGINT
reference_count   INTEGER
created_at        TIMESTAMP

Reference Counting for Safe Deletion

The reference_count tracks how many user files (and file versions) point to this S3 object:

text
User A uploads file X                       → reference_count = 1
User B uploads same file X (deduplicated)   → reference_count = 2
User A deletes their file                   → reference_count = 1 (S3 object stays)
User B deletes their file                   → reference_count = 0 (S3 object safe to delete)

Reference count updates are done inside PostgreSQL transactions to prevent race conditions:

sql
BEGIN;
UPDATE S3Objects SET reference_count = reference_count - 1
WHERE content_hash = 'abc123...'
RETURNING reference_count;
-- If reference_count = 0, record an s3.object.orphaned event to publish to Kafka
COMMIT;

The S3 Cleanup Worker consumes s3.object.orphaned events and deletes the actual S3 object.

One race needs care. Between the count reaching 0 and the cleanup worker running, someone could upload the same file again, find the hash, and bump the count back to 1. If the worker then deleted the S3 object anyway, that new upload would point at nothing. So the worker deletes the database row first, and only if the count is still zero, and deletes the S3 object only if that row delete succeeded:

sql
DELETE FROM S3Objects WHERE content_hash = 'abc123...' AND reference_count = 0;
-- 1 row deleted: safe to delete the S3 object. 0 rows: it was reused, keep it.

Security and Privacy

Deduplication must never leak information between users. User B cannot infer that User A has a file by noticing their identical upload completed instantly. The deduplication is completely transparent to users. Each user sees their own file in their own Drive with no indication of shared underlying storage.

This is why the deduplication runs on the server, after the whole file has been uploaded. If the client sent the hash first and skipped the upload whenever the server already had that content, the instant "upload complete" would reveal that someone else already stores that exact file. Uploading the full bytes every time costs bandwidth but keeps that signal hidden.

Bottlenecks in the Deduplication Service

SHA-256 hashing of large files (10 GB video) is CPU-intensive and takes several seconds. This happens as the chunks stream through the Block Service, and the dedup decision happens asynchronously after upload completes, so users are not blocked. The hash computation can be distributed by hashing each chunk independently and combining chunk hashes into a final file hash (a Merkle tree approach), reducing single-threaded hashing time significantly.

6. File Metadata Service

The File Metadata Service manages all metadata about files and folders: names, types, sizes, locations in the folder hierarchy, and ownership.

Data Model

Files Table (PostgreSQL)

text
Files
file_id           UUID, primary key
owner_id          UUID, foreign key -> Users
file_name         VARCHAR
file_extension    VARCHAR
mime_type         VARCHAR
file_size         BIGINT
content_hash      VARCHAR(64), foreign key -> S3Objects
is_folder         BOOLEAN
is_starred        BOOLEAN
is_trashed        BOOLEAN
trashed_at        TIMESTAMP (null if not trashed)
version           INTEGER (current version number)
created_at        TIMESTAMP
modified_at       TIMESTAMP
last_modified_by  UUID, foreign key -> Users

(The S3 location comes from the content_hash via S3Objects, so it isn't duplicated on every file row.)

Folder Hierarchy - Multi-Parent Adjacency List

Google Drive uses a multi-parent adjacency list to model the folder hierarchy. Files and folders are stored in the same Files table (distinguished by is_folder). The parent-child relationship is stored in a separate join table to support files appearing in multiple folders simultaneously via shortcuts:

FileParents Table (PostgreSQL)

text
FileParents
file_id       UUID, foreign key -> Files
parent_id     UUID, foreign key -> Files
is_shortcut   BOOLEAN (true if this is a shortcut, not the original location)
PRIMARY KEY (file_id, parent_id)

Key queries:

Give me all contents of folder X (one level deep):

sql
SELECT f.* FROM Files f
JOIN FileParents fp ON f.file_id = fp.file_id
WHERE fp.parent_id = 'folder_x'
  AND f.is_trashed = false
ORDER BY f.is_folder DESC, f.file_name ASC;

Give me the full path of file X (breadcrumb navigation):

sql
WITH RECURSIVE path AS (
  SELECT file_id, parent_id, 1 AS depth FROM FileParents
  WHERE file_id = 'file_x' AND is_shortcut = false
  UNION ALL
  SELECT fp.file_id, fp.parent_id, p.depth + 1 FROM FileParents fp
  JOIN path p ON fp.file_id = p.parent_id
  WHERE fp.is_shortcut = false
)
SELECT * FROM path ORDER BY depth;   -- file -> parent -> parent -> ... -> root

Typically 5-6 levels deep, fast with indexed lookups.

The adjacency list tradeoff is acceptable for Google Drive because:

  • One-level-deep queries (listing folder contents) are the overwhelming majority of queries
  • Full path reconstruction (breadcrumbs) only goes 5-6 levels deep in practice
  • Moving a file is a single UPDATE FileParents SET parent_id = new_folder operation

(It's the same pattern Reddit uses for comment threads.) One check is needed when moving a folder: the destination must not be the folder itself or one of its own descendants, otherwise the tree would get a loop. The service walks up from the destination with the same recursive query and rejects the move if it reaches the folder being moved.

Caching Strategy

Hot folders (My Drive root, frequently accessed project folders) are cached in Redis with a short TTL. File metadata for recently accessed files is cached per user. Cache is invalidated on file rename, move, or delete via Kafka events.

Bottlenecks in the File Metadata Service

Folder listing queries with large folders (thousands of files) can be slow without proper indexing. An index can only cover columns of one table, so the listing query is served by an index on FileParents (parent_id, file_id), which finds the children of a folder instantly, plus the primary key lookup on Files for each child. If very large folders still need sorting help, is_trashed, is_folder and file_name can be copied onto the FileParents row so a single index (parent_id, is_trashed, is_folder, file_name) serves the whole listing in order. The File Metadata Service is read-heavy and benefits significantly from Redis caching of hot directories.

7. Permission Service

The Permission Service manages role-based access control for all files and folders. It is the most security-critical service in the system.

Permission Model

text
FilePermissions
permission_id   UUID, primary key
file_id         UUID, foreign key -> Files
user_id         UUID, foreign key -> Users (null for link-based sharing)
permission      ENUM('owner', 'editor', 'commenter', 'viewer')
share_type      ENUM('specific_user', 'link_anyone', 'link_org')
created_at      TIMESTAMP
granted_by      UUID, foreign key -> Users
UNIQUE (file_id, user_id)

(A primary key can't contain a null, and link-based shares have no user_id, so the table gets its own permission_id and a unique constraint on (file_id, user_id) for user shares instead.)

Permission Inheritance

Permissions in Google Drive inherit down the folder hierarchy. If User A has editor access to a folder, they have editor access to all files and subfolders within it. Permission inheritance is resolved at query time by walking up the FileParents tree until a permission record is found.

Inheritance resolution is cached in Redis per user per file to avoid repeated tree traversals:

text
Key:   perm:{user_id}:v{perm_version}:{file_id}
Value: ENUM('owner', 'editor', 'commenter', 'viewer', 'none')
TTL:   15 minutes

The perm_version in the key matters because of inheritance. Revoking a user's access to a folder changes their effective permission on every file inside it, which could be thousands of cached entries. Deleting them one by one is slow and easy to get wrong. Instead, each user has a small permission version counter (perm_version:{user_id} in Redis). Any permission change that affects the user bumps the counter, every old cache key for that user instantly stops being read, and the stale entries simply expire.

Permission Revocation

When a file owner revokes a user's access:

  1. Delete permission record from FilePermissions table in PostgreSQL
  2. Bump the user's permission version in Redis (INCR perm_version:{user_id}), which invalidates all of their cached permissions at once
  3. Publish permission.revoked event to Kafka
  4. Any new request from that user (listing, opening, generating a download link) is now checked against the fresh permission and refused, within seconds

Download links that were already handed out are the one gap. A signed URL is self-contained: whoever holds it can use it until it expires, without the server being asked again, so it can't be "invalidated" individually. That's why Google Drive's signed URLs are kept short, and why the TTL should match how strict the revocation requirement is. To make revocation effective within seconds even for files already being downloaded, large or sensitive downloads go through a proxy (or a CDN with a revocation check) that checks permission on each request, instead of a long-lived URL.

Bottlenecks in the Permission Service

Permission checks happen on every file access, download, and metadata operation. The Redis permission cache absorbs the vast majority of checks. The inheritance resolution walk is the bottleneck for deeply nested folders, mitigated by caching the resolved permission rather than the raw inheritance chain.

8. Download Service

The Download Service generates signed URLs for file access and handles both small file downloads and large file streaming.

Download Flow

  1. Client requests file: GET /api/v1/files/{file_id}/download
  2. Download Service checks permission via Redis cache
  3. If permitted: fetches file metadata (S3 URL, encryption key reference) from File Metadata Service
  4. Generates a short-lived signed CDN URL (a few minutes, at most 15-30) for the file
  5. Returns signed URL to client
  6. Client fetches file directly from CDN using signed URL

CDN Layer

All files are served through a CDN (CloudFront or Akamai) in front of S3. Popular shared files (a PDF shared with 10,000 colleagues for a product launch) are cached at CDN edge nodes. The first download triggers a cache fill from S3. All subsequent downloads are served from the CDN edge with no S3 involvement.

For large files (videos, large archives), the CDN supports byte-range requests which allow clients to fetch specific portions of a file. This enables:

  • Video seeking without downloading the entire file
  • Resumable downloads that continue from where they left off
  • Parallel chunk downloads for faster transfer speeds

Small vs Large File Handling

Small files (under 10 MB)Large files (over 10 MB)
How it's fetchedSingle CDN request, served in one responseClient issues byte-range requests, CDN serves chunks sequentially or in parallel

The Block Service's encryption per chunk aligns naturally with byte-range requests. Each chunk can be decrypted independently since it was encrypted independently during upload, so a byte range only needs the chunks it touches.

Bottlenecks in the Download Service

The CDN absorbs the vast majority of download load. The Download Service itself only generates signed URLs and checks permissions, both sub-millisecond operations. The bottleneck is CDN cache fill for newly uploaded files that are immediately shared with thousands of users simultaneously, mitigated by CDN warming for anticipated high-traffic files.

9. Version History Service

Google Drive retains previous versions of files when they are modified and re-uploaded. Users can view and restore any previous version.

Data Model

FileVersions Table (PostgreSQL)

text
FileVersions
version_id      UUID, primary key
file_id         UUID, foreign key -> Files
version_number  INTEGER
content_hash    VARCHAR(64), foreign key -> S3Objects
file_size       BIGINT
created_at      TIMESTAMP
created_by      UUID, foreign key -> Users
comment         VARCHAR (optional version label)

Each time a file is re-uploaded, a new FileVersions record is created pointing to the new S3 object. The previous version record is retained. The Files table version column points to the current version number.

Storage Tiering for Old Versions

Version storage follows a tiered approach to control costs:

TierWhat lives thereTrade-off
S3 StandardCurrent versionsFast access, full price
S3 Infrequent AccessVersions older than 30 daysSlightly slower, roughly 40% cheaper
S3 GlacierVersions older than 90 daysMinutes to hours to retrieve, roughly 80% cheaper

A background Version Tiering Worker runs nightly and moves eligible versions to cheaper storage tiers based on age. S3 Object Lifecycle Policies automate the actual storage class transitions.

Deduplication adds one rule here: tiering is decided per stored object, not per version. An S3 object that is an old version for one user can be the current version for another user (the same content uploaded twice), so it is only moved to a colder tier once no current version anywhere still points at it.

Version Restoration

When a user restores a previous version:

  1. Version History Service fetches the target version record from FileVersions table
  2. If version is in Glacier: initiates restore request (1-5 minutes with expedited retrieval, a few hours with standard retrieval)
  3. Once restored: creates a new version record pointing to the old content (does not delete any existing versions)
  4. Updates Files table version to the new version number
  5. Publishes file.version_restored event to Kafka for search reindexing and notifications

Deduplication and Versioning Interaction

If a user uploads a file, edits it, then reverts to the original content and re-uploads, the re-upload produces the same SHA-256 hash as the original. Deduplication detects this and the new version points to the same S3 object as the original version. No duplicate storage. Reference count on the S3 object increments to account for the new version reference.

10. Quota Service

Every Google Drive user has a storage quota. The Quota Service tracks usage and enforces limits.

Quota Tracking

UserQuotas Table (PostgreSQL)

text
UserQuotas
user_id           UUID, primary key, foreign key -> Users
quota_bytes       BIGINT (15 GB free = 16,106,127,360 bytes)
used_bytes        BIGINT
last_updated      TIMESTAMP

used_bytes is a denormalized counter updated asynchronously via Kafka consumers when files are uploaded or deleted. It reflects the sum of all current file versions owned by the user, regardless of deduplication savings.

Quota Enforcement

Quota is checked before every upload:

  1. GET quota:{user_id} from Redis
  2. If used_bytes + new_file_size > quota_bytes: reject upload with 507 Insufficient Storage

Quota balance is cached in Redis per user with a short TTL for fast checks:

text
Key:   quota:{user_id}
Value: { quota_bytes, used_bytes }
TTL:   5 minutes

Quota and Deduplication

Each user's quota is charged against their own files regardless of deduplication:

text
User A uploads 500 MB file                          → User A's used_bytes += 500 MB
User B uploads identical 500 MB file (deduplicated) → User B's used_bytes += 500 MB

Google saves the 500 MB S3 storage cost internally but both users are charged fairly. (It also keeps deduplication invisible: if User B's quota didn't move, that would reveal the file already existed.)

Quota and Deletion

When a user deletes a file it moves to Trash. Quota is not freed until the file is permanently deleted (user empties Trash or 30-day auto-purge):

  1. File permanently deleted -> Kafka (file.permanently_deleted)
  2. Quota Service consumer: UPDATE UserQuotas SET used_bytes = used_bytes - file_size WHERE user_id = X
  3. Redis quota cache invalidated: DEL quota:{user_id}
  4. Reference count on S3 object decremented

11. Search Service

Google Drive search covers file names, folder names, and full text content within supported file types.

Text Extraction Pipeline

Before files can be searched by content, their text must be extracted. This happens asynchronously after upload:

text
file.assembled event -> Kafka -> Text Extraction Worker
    -> Determine file type from MIME type
    -> Run appropriate parser:
        PDF    -> Apache PDFBox extracts text
        DOCX   -> Apache POI extracts text from XML
        XLSX   -> Apache POI extracts cell values
        Images -> OCR via Tesseract (makes whiteboard photos searchable)
        TXT    -> direct indexing, no extraction needed
        ZIP    -> extract and parse contents recursively (with size and depth limits, so a "zip bomb" can't explode on the server)
    -> Publish file.text_extracted to Kafka
    -> Search Indexing Worker indexes into Elasticsearch

Extracted text is indexed into Elasticsearch only, not stored permanently in the database. If the search index needs rebuilding, text extraction runs again from the original S3 file.

Elasticsearch Document Model

json
{
  "file_id": "123",
  "owner_id": "456",
  "file_name": "Q3 Product Roadmap.pdf",
  "file_type": "application/pdf",
  "file_extension": "pdf",
  "file_size_bytes": 2048000,
  "content_text": "This document outlines the product strategy for Q3 2026...",
  "parent_folder_id": "789",
  "shared_with": ["user_101", "user_202", "user_303"],
  "is_starred": false,
  "is_trashed": false,
  "created_at": "2026-06-01T10:00:00Z",
  "modified_at": "2026-08-10T14:30:00Z",
  "last_modified_by": "user_456"
}

Filter Support

All common Drive filters map to Elasticsearch structured queries:

text
Files shared with me:     { "term":  { "shared_with": "current_user_id" } }
Files modified last week: { "range": { "modified_at": { "gte": "now-7d" } } }
Files larger than 10 MB:  { "range": { "file_size_bytes": { "gte": 10485760 } } }
Files of type PDF:        { "term":  { "file_extension": "pdf" } }

Combined search with filters:

json
{
  "query": {
    "bool": {
      "must": [
        { "match": { "content_text": "product strategy" } }
      ],
      "filter": [
        { "term": { "shared_with": "current_user_id" } },
        { "term": { "file_extension": "pdf" } },
        { "range": { "modified_at": { "gte": "now-7d" } } },
        { "term": { "is_trashed": false } }
      ]
    }
  }
}

Security Filtering

Every search query automatically includes a security filter:

json
{
  "filter": {
    "bool": {
      "should": [
        { "term": { "owner_id": "current_user_id" } },
        { "term": { "shared_with": "current_user_id" } }
      ]
    }
  }
}

A user can never see files they do not have access to in search results, even if they know the exact filename. The shared_with array in the Elasticsearch document is updated via Kafka consumer whenever permissions change.

Because permissions inherit, shared_with has to hold everyone with effective access, not only people the file was shared with directly. Sharing a folder therefore fans out an update to every file inside it. As a second line of defence, the Search Service re-checks each result against the Permission Service before returning it, so a search index that is a few seconds behind a revocation still can't show a file the user just lost access to.

Keeping Elasticsearch in Sync

The Search Indexing Worker consumes from multiple Kafka topics:

TopicUpdate
file.assembledInitial index after upload
file.text_extractedUpdate with extracted content
file.renamedUpdate file_name
file.movedUpdate parent_folder_id
file.sharedUpdate shared_with array
file.trashedUpdate is_trashed flag
file.permanently_deletedDelete from index

12. Notification Service

Same architecture as all previous systems, and as the Notification System design. Kafka fanout from upstream services, APNs for iOS, FCM for Android, email via SES.

The Notification Service subscribes to:

  • file.shared -> "Harry shared a file with you" email + push notification
  • file.permission_changed -> "Your access to a file has changed" notification
  • file.comment_added -> comment notification to file owner and mentioned users
  • file.version_restored -> "A file was restored to a previous version" notification
  • quota.warning -> "You are approaching your storage limit" email when usage exceeds 80% and 95%
  • upload.completed -> "Your upload is complete" notification for large file uploads

13. Full Data Flow Summary

Small File Upload Flow:

text
Client -> API Gateway (auth) -> Upload Service -> presigned S3 URL
Client -> S3 (direct upload)
S3 -> Kafka (file.uploaded)
    -> Deduplication Service:
        -> Take the SHA-256 hash of the plaintext
        -> Check S3Objects table (duplicate or new)
        -> Increment reference_count if duplicate
        -> Create new S3Objects record if new
    -> File Metadata Service: create Files record
    -> Quota Service: UPDATE used_bytes
    -> Text Extraction Worker -> Kafka (file.text_extracted)
    -> Search Indexing Worker -> Elasticsearch

Large File Upload Flow (Resumable):

text
Client -> POST /upload?uploadType=resumable -> Upload Service -> session_id
Client -> PUT /upload/{session_id}/chunk/{index} (repeat per chunk)
    -> Block Service: hash + encrypt chunk -> S3 multipart part {index + 1}
    -> Redis SADD upload:{session_id}:chunks {index}
Client -> POST /upload/{session_id}/complete
    -> File Assembly Service: S3 CompleteMultipartUpload
    -> Deduplication Service (same as small file flow)

Interrupted Upload Resume Flow:

text
Client reconnects -> GET /upload/{session_id}/status
    -> Redis SMEMBERS upload:{session_id}:chunks -> completed chunk list
    -> Client resumes from first missing chunk index

Download Flow:

text
Client -> GET /files/{file_id}/download -> Download Service
    -> Redis permission check
    -> Fetch file metadata + S3 URL
    -> Generate short-lived signed CDN URL
    -> Client fetches from CDN (cache hit ~90% for popular shared files)
    -> CDN cache miss -> S3 -> CDN caches -> serves file

Permission Revocation Flow:

text
Owner revokes access -> Permission Service
    -> DELETE from FilePermissions (PostgreSQL)
    -> INCR perm_version:{user_id} (Redis, invalidates all cached permissions)
    -> Kafka (permission.revoked)
    -> New requests refused within seconds; outstanding signed URLs expire on their short TTL

File Deletion Flow:

text
User deletes file -> File Metadata Service -> is_trashed = true
User empties trash (or 30-day auto-purge) -> hard delete triggered
    -> Quota Service: used_bytes decremented
    -> S3Objects: reference_count decremented via PostgreSQL transaction
    -> If reference_count = 0: Kafka (s3.object.orphaned) -> S3 Cleanup Worker
       -> deletes the row only if still at 0, then deletes the S3 object
    -> Search Indexing Worker: delete from Elasticsearch index

Version History Flow:

text
User re-uploads file -> new FileVersions record created
    -> Previous version retained
    -> Files.version incremented
Version Tiering Worker (nightly):
    -> Move versions older than 30 days to S3 Infrequent Access
    -> Move versions older than 90 days to S3 Glacier
    (only objects no current version still points to)
User restores version:
    -> If Glacier: initiate restore (minutes with expedited retrieval)
    -> Create new version record pointing to old content
    -> Update Files.version

Search Flow:

text
Client searches "product strategy pdf" -> Search Service
    -> Elasticsearch bool query:
        must: match content_text
        filter: file_extension, is_trashed = false
        security filter: owner_id OR shared_with = current_user
    -> Re-check permissions for the results
    -> Redis MGET (hydrate file metadata for results)

14. Resilience and Fault Tolerance

  • Resumable uploads with Redis chunk tracking ensure no upload progress is lost on network failure. The client always knows exactly which chunks succeeded and resumes from the first failure. Upload sessions expire after 24 hours of inactivity preventing orphaned session state (an S3 lifecycle rule also aborts multipart uploads that were never completed, so their parts don't linger as hidden storage).
  • Content-addressable deduplication with reference counting prevents both storage waste and premature S3 deletion. PostgreSQL transactions ensure reference count updates are atomic, and the cleanup worker's "delete only if still zero" check means a file that is re-uploaded at the last moment is never deleted out from under it.
  • Short-lived signed URLs limit how long an already-issued link keeps working after revocation, while new requests are refused within seconds by the permission check. Downloads that need instant revocation go through a permission-checking proxy instead.
  • Kafka durability throughout all pipelines means no file event is lost even if downstream services crash. Text extraction, search indexing, quota updates, and notification delivery all retry from Kafka offsets on recovery.
  • Storage tiering with S3 Glacier reduces long-term storage costs for old file versions without deleting them. Restoration from Glacier takes minutes with expedited retrieval, which is acceptable for version history retrieval.
  • CDN caching for all file downloads absorbs the vast majority of download traffic. Popular shared files are served from CDN edge nodes with no S3 origin involvement after the initial cache fill.
  • Envelope encryption with AWS KMS ensures files are encrypted at rest with per-object keys. A compromised S3 bucket cannot be read without the encryption keys, which are managed separately in KMS.
  • Redis quota caching with short TTL ensures quota checks are fast while remaining reasonably accurate. The 5-minute TTL means a user could theoretically upload slightly over their quota in a race condition, but the Kafka consumer reconciles the true used_bytes within minutes.

15. Technology Choices Summary

ComponentTechnologyWhy
API GatewayKong / AWS API GatewayAuth, rate limiting, routing for metadata operations
Block ServiceCustom encryption service + AWS S3 Multipart UploadPlaintext hashing, AES-256 encryption per chunk, 5 MB parts
Upload Session StoreRedisChunk completion tracking, resumable upload state, 24-hour TTL
File Content StoreAWS S3Encrypted file content, lifecycle policies for tiering
CDNCloudFront / AkamaiEdge caching for all file downloads, byte-range request support
Storage TieringS3 Standard + S3 Infrequent Access + S3 GlacierAutomated via S3 Object Lifecycle Policies
DeduplicationSHA-256 of plaintext + PostgreSQL S3Objects tableReference counting, per-object encryption key
File Metadata DBPostgreSQL sharded + replicasFiles, FileParents, FileVersions, FilePermissions, strong consistency
Folder HierarchyMulti-parent adjacency list in FileParents tableSupports shortcuts via multiple parent rows
Permission CacheRedis with per-user version keysSub-millisecond permission checks, instant bulk invalidation
Quota StorePostgreSQL UserQuotas + Redis cache with 5-minute TTLFast quota checks, async counter updates
Encryption Key ManagementAWS KMS with envelope encryptionPer-object DEK encrypted by master KEK
Search EngineElasticsearchFilename search, full-text content search, rich filter support
Text ExtractionApache PDFBox + Apache POI + Tesseract OCRAsync pipeline via Kafka
DecouplingApache KafkaUpload events, deduplication triggers, permission changes, quota updates, search indexing
Version Tiering WorkerBackground job + S3 Lifecycle PoliciesNightly tiering of old versions to cheaper storage
S3 Cleanup WorkerKafka consumerDeletes orphaned S3 objects once their reference count is confirmed zero
Notification DeliveryAPNs + FCM + AWS SESShare notifications, quota warnings, upload completion alerts