Design Cloud File Storage (Dropbox/Drive)

1Requirements

2Back-of-envelope estimation

3API design

4High-level design

The desktop client is a significant system component in its own right: a watcher detects local file changes, a chunker splits files into blocks and hashes each (SHA-256), a local SQLite index maps files to chunk hashes, and a sync engine reconciles local state against the server journal. Because chunks are content-addressed, 'what changed' is computable entirely from hashes.

On upload, the client sends the chunk-hash manifest to the metadata service first. The server diffs it against known chunks and returns only the missing ones, so unchanged chunks (the common case for edits) and chunks any user already uploaded (dedup) are never transferred. Missing chunks go straight to block storage (S3-style) via presigned URLs, keeping bulk bytes off the application tier.

Once all chunks are durable, the client commits the version. The metadata service transactionally writes the new file version, its ordered chunk list, and a journal entry, with an atomic parent-version check that is the linchpin of conflict detection. Metadata lives in a sharded relational database (sharded by user/namespace) because sync correctness leans hard on transactions.

The journal entry triggers the notification service, which pings the user's other online devices over long-lived connections. Devices respond by pulling changes from their cursor, computing which chunks they lack, and downloading just those, assembling the new version locally. Shared folders work the same way with the namespace's journal fanned out to all members.

Cold chunks tier from hot object storage to cheaper archival classes based on access recency, and block storage runs erasure coding across failure domains for durability at ~1.5x overhead. A garbage collector deletes chunks only when reference counts from all live versions and the retention window reach zero.

5Data model

6Deep dives

7Rapid implementation: build the MVP

Theory is table stakes. Here is how you would stand up a working version fast, on a budget, with the core algorithm in real code.

1. 01Run MinIO locally with a chunks bucket; create SQLite tables files, file_versions, version_chunks, and chunks.
2. 02Write the client chunker: fixed 4 MB blocks, SHA-256 each, producing an ordered hash manifest per file.
3. 03Build POST /files/precommit that returns which manifest hashes the server does not yet have, and PUT /chunks/{hash} that stores a block in MinIO keyed by its hash.
4. 04Build POST /files/commit that inserts the version row and manifest atomically, conditional on the parent version still being head; reject stale parents.
5. 05Write the client sync loop: watchdog detects a changed file, re-chunk it, upload only the missing hashes, then commit.
6. 06Implement download as the mirror image: fetch the manifest, pull only chunks absent from the local chunk cache, reassemble in order.
7. 07Handle commit rejection by saving the local file as 'name (conflicted copy)' and committing it as a new file.
8. 08Verify dedup end to end: copy a 1 GB file to a second name and confirm the second upload transfers zero chunks.

import hashlib

CHUNK = 4 * 1024 * 1024  # 4 MB fixed blocks

def chunk_manifest(path):
    hashes = []
    with open(path, "rb") as f:
        while True:
            block = f.read(CHUNK)
            if not block:
                break
            hashes.append(hashlib.sha256(block).hexdigest())
    return hashes

def sync_file(api, path, parent_version):
    manifest = chunk_manifest(path)
    # server answers with only the hashes it has never seen
    missing = set(api.post("/files/precommit", hashes=manifest))
    with open(path, "rb") as f:
        for seq, h in enumerate(manifest):
            if h not in missing:
                continue  # dedup: server already has this block
            f.seek(seq * CHUNK)
            api.put_chunk(h, f.read(CHUNK))  # content-addressed PUT
    # blocks are durable first; metadata commit comes second
    return api.post("/files/commit", path=path,
                    hashes=manifest, parent_version=parent_version)

-- precommit: which of the client's hashes are new to us?
SELECT h.hash
FROM unnest($1::text[]) AS h(hash)
WHERE NOT EXISTS (SELECT 1 FROM chunks c WHERE c.hash = h.hash);

-- commit: one transaction, conditional on parent still being head
BEGIN;
INSERT INTO file_versions (file_id, version, file_hash)
SELECT $1, $2 + 1, $3
WHERE COALESCE(
  (SELECT MAX(version) FROM file_versions WHERE file_id = $1), 0
) = $2;
-- 0 rows inserted: another device committed first, so the client
-- must create a conflicted copy instead of overwriting head.

INSERT INTO version_chunks (version_id, seq, chunk_hash)
SELECT currval('file_versions_id_seq'), s.ord - 1, s.hash
FROM unnest($4::text[]) WITH ORDINALITY AS s(hash, ord);

INSERT INTO journal (namespace_id, file_id, op)
VALUES ($5, $1, 'commit');
COMMIT;