Design Dropbox (File Sync)
Chunk files by content hash so an edit uploads one chunk, keep metadata separate from bytes, and handle conflicting edits without losing work.
- 0.4%of a 1 GiB file uploaded for one edit
- ~47.7B chunksfor 200 PB at 4 MiB
- ~5,800/sfile changes from 50M daily users
Show what to say
“I'd keep bytes and metadata apart. Files are cut into 4 MiB chunks named by SHA-256 and stored once in object storage; the client asks which hashes are new and uploads only those. A sharded SQL service stores each file as a revision plus chunk list, commits against a base revision, and appends to a per-user journal that devices read with a cursor.”
Trade-offs
- Fixed vs content-defined chunks: fixed is trivial, but one inserted byte shifts every chunk; content-defined keeps most hashes at a CPU cost.
- Conflicted copies vs last write wins: nobody loses work, but the user must merge by hand.
- Cross-user dedupe vs privacy: large storage savings, but it can reveal that someone else holds the same file.
Bring it up before they ask
Garbage collection racing a commit: use reference counts and a grace period so a chunk an in-flight commit needs is never deleted.
Split every file into chunks named by the hash of their bytes. Upload only chunks the server lacks, and keep a small metadata service that says which chunks make up each file. Sync devices with a change cursor, and never overwrite silently: a stale commit becomes a conflicted copy.
Functional
- Put files in a folder that syncs across a user's devices
- Upload and download large files reliably
- Pick up edits quickly on other devices
- Keep version history and restore old versions
- Share a file or folder with other people
Non-functional
- Edits appear on other devices within seconds
- Uploading a small edit to a big file costs a small upload
- Never lose or silently overwrite a user's work
- About 200 PB stored, strongly consistent file metadata
- Works offline and catches up later
Out of scope
- Real-time collaborative editing (see the Docs design)
- Search inside files and previews
- Billing and storage plans
Clarify and estimate
Ask first:
Read more · 218 words
- Scale? Users, files per user, largest file.
- Conflicts? Do two people edit one file at once? What wins?
- Sharing and history? How many versions, for how long?
Capacity estimate
| Quantity | Calculation | Result |
|---|---|---|
| Stored data | 100M users × 2 GB average | 200 PB |
| Chunks | 200 PB ÷ 4 MiB | ≈ 47.7 billion |
| Chunk index | 47.7B × 64 B (hash, location, count) | ≈ 3 TB |
| Changes | 50M daily users × 10 changes ÷ 86,400 s | ≈ 5,800/s, ~17K/s at peak |
| Upload bytes | 500M changes × at most 1 chunk of 4 MiB | ≤ 2 PB/day ≈ 24 GB/s |
| Connections | 50M online devices ÷ 50K per server | ≈ 1,000 notifier servers |
| Edit of a 1 GiB file | 1 chunk ÷ 256 chunks | 0.4% of the file |
Metadata is small but must be right. Chunk bytes are huge but easy to spread. That split drives the design: for bytes, a store for metadata.
API and data model
API
Read more · 204 words
| Method | Path | Notes |
|---|---|---|
| POST | /chunks/unknown | { hashes[] } → the ones the server lacks |
| PUT | /chunks/:hash | One chunk; the server verifies the hash |
| POST | /files/commit | { path, baseRev, chunks[] } → { rev, cursor } or 409 |
| GET | /changes?cursor= | Everything after the cursor, plus the new cursor |
| GET | /files/:id?rev= | Ordered chunk list for a revision |
| POST | /shares | Grant another user a folder |
Devices authenticate with a token. The notifier connection is a or long poll that carries only a poke.
Data model
All of one user's rows share a , so a commit is one single-shard . The chunks table is a separate, much larger key-value store by hash.
CREATE TABLE files (
user_id bigint,
path text,
rev int NOT NULL, -- bumped on every accepted commit
chunks text[] NOT NULL, -- ordered chunk hashes
deleted boolean NOT NULL DEFAULT false,
PRIMARY KEY (user_id, path)
);
CREATE TABLE journal (
user_id bigint,
cursor bigint, -- 1, 2, 3 ... per user
path text, rev int, chunks text[], deleted boolean,
PRIMARY KEY (user_id, cursor)
);
CREATE TABLE chunks (
hash bytea PRIMARY KEY, -- SHA-256 of the bytes
location text NOT NULL, -- key in the block store
refs int NOT NULL -- how many revisions use it
);Architecture
Bytes and metadata take different paths.
Read more · 36 words
The client works out what is new, uploads only that to the block store, and then commits a small list of hashes. Play Save a new file, Edit one paragraph, Another device syncs and Edit conflict.
Tap any box to see what it is and what it talks to.
- Client
- Service
- Object storage
- Database
- Queue
- Pub/Sub
Text description
- A 12 MiB report is saved. The client cuts it into three 4 MiB chunks and hashes each one.
- It asks: which of these 3 hashes do you not have?
- All 3 are new. (Had a colleague already stored the same file, the answer would be none, and nothing would upload.)
- The 3 chunks are uploaded and stored by hash. A broken connection costs at most one chunk.
- Commit: path /report.doc, base revision 0, chunks [h1, h2, h3]. The metadata service checks that every chunk exists.
- One transaction on the user's shard: revision 1 saved, chunk reference counts bumped.
- The change is appended to the user's feed as cursor 41. The file is now safe.
Chunking and dedupe
Cut each file into chunks of about 4 MiB and name each by its SHA-256.
Read more · 69 words
Same bytes, same name. An edit uploads one chunk, a broken upload resumes at the last good chunk, and identical chunks are stored once. The client sends hashes first and uploads only the unknown ones.
Fixed chunks break when you insert one byte at the front: every boundary shifts. Content-defined chunking cuts where a rolling hash matches a pattern, so cuts re-sync after an edit and most hashes survive.
Sync, conflicts and notifications
Conflicts and the journal. A file is a path, a revision and an ordered list of chunk hashes.
Read more · 201 words
A commit says "I started from revision N". If the file is still at N, it moves to N+1. If not, someone got there first: conflict. The loser is saved as "report (Maya's conflicted copy)", and nothing is lost.
Each commit is appended to a per-user journal. A device keeps a cursor and asks "what changed since 40?". Asking twice is harmless, and a device offline for a week catches up the same way.
The same push as a sequence. Pick a branch: a normal edit, a hit, or a conflict.
Telling other devices. every device every few seconds would cost millions of requests for nothing. Instead each online device holds a connection to a notifier. After a commit, the notifier sends a tiny poke, and the device pulls the journal.
A poke carries no data, so a lost one only delays sync. Devices also poll slowly (every few minutes) as a backstop. Offline devices simply use their cursor when they return.
- Client
- Service
- Database
- Object storage
Text description
- Client → Sync API: unknown(hashes)?
- Sync API → Metadata: which are new?
- Metadata replies to Client: new: [h2]
- Client → Block store: put chunk h2
- Client → Sync API: commit(path, base 7, chunks)
- Sync API → Metadata: base == current? yes
- Metadata replies to Client: rev 8, cursor 41
Step-by-step answer
Order: clarify, estimate, separate bytes from metadata, deep dive on chunking then versions and conflicts, then notifications and failure.
Read more · 325 words
Split bytes from metadata
Bytes are huge, immutable and content-addressed: . Metadata is small, mutable and needs : a store keyed by user.
Chunk and hash on the client
4 MiB chunks, SHA-256 names. The client hashes locally, so unchanged data never leaves the machine.
Ask before sending
"Which of these hashes are new?" Upload only those. This is , delta sync and resumable upload in one move.
Commit with a base revision
The metadata service refuses a commit if any chunk is missing or the base revision is stale. A file never points at bytes that do not exist.
Journal the change
Append to the user's journal in the same transaction. The journal is the sync protocol: cursors, not timestamps, so clock skew cannot lose a change.
Poke, then pull
The notifier tells online devices something changed. They fetch the journal from their cursor and download the missing chunks.
Keep versions and clean up
Old revisions keep their chunk lists. A chunk's count drops when a revision expires; at zero, a delayed garbage collector deletes it, after a grace period so an in-flight commit cannot reference it.
Share safely
A shared folder is one namespace many users can read. Permissions are checked on the metadata call; chunk URLs are short-lived and signed.
Implementation
Tabs: fixed chunking with hashes, , the metadata service with revisions and the journal, and the client's push and pull loops.
Read more · 52 words
- The chunker test proves one changed byte makes one new chunk, and that an inserted byte changes all fixed chunks.
- The CDC test keeps over 95% of chunks after a 100-byte insertion; fixed chunks keep under 5%.
- The metadata test runs the conflict: first commit wins, the second gets a conflicted copy.
// Split a file into fixed-size chunks and name each one by the hash of its bytes.
// Same bytes, same name: that single idea gives dedupe, resumable upload and delta sync.
export const CHUNK = 4 * 1024 * 1024; // 4 MiB
export async function sha256Hex(bytes) {
const digest = await globalThis.crypto.subtle.digest('SHA-256', bytes); // Web Crypto: browser, Node 20+, Deno
return [...new Uint8Array(digest)].map((b) => b.toString(16).padStart(2, '0')).join('');
}
export async function fixedChunks(bytes, size = CHUNK) {
const out = [];
for (let start = 0; start < bytes.length; start += size) {
const part = bytes.subarray(start, Math.min(start + size, bytes.length));
out.push({ hash: await sha256Hex(part), start, length: part.length });
}
return out;
}
// What the client must upload: chunks the server does not already have.
export const missing = (chunks, serverHas) => chunks.filter((c) => !serverHas.has(c.hash));Failure modes and trade-offs
Read more · 175 words
| What breaks | Effect | Mitigation |
|---|---|---|
| Upload dies mid-file | Wasted bandwidth | Per-chunk upload; resume by asking which hashes are unknown |
| Commit before chunks exist | File points at nothing | Metadata verifies every chunk before accepting |
| Two devices edit offline | Lost update | Base revision check; conflicted copy |
| Garbage collection races a commit | A live file loses a chunk | Reference counts and a grace period before deleting |
| Notifier restarts | Pokes lost | Devices also poll slowly and use cursors |
| Hash collision or corrupt chunk | Wrong bytes served | SHA-256 verified on write and read; checksums at rest |
| Metadata shard fails | Users on it cannot sync | Replicas with failover; clients retry |
Trade-offs
| Choice | Gain | Cost |
|---|---|---|
| Fixed chunks | Trivial and fast | Insertions re-upload the file |
| Content-defined chunks | Small deltas on inserts | CPU, variable sizes, more metadata |
| Conflicted copies | No lost work | The user merges by hand |
| Cross-user dedupe | Big storage savings | Leaks "someone has this file" unless handled |
Follow-up questions
Is cross-user dedupe safe?
It can reveal that someone stores a file. Make the client prove it holds the bytes, or within an account only.
How would you support large video files?
Bigger chunks, parallel uploads and per-chunk retries, using multipart upload in .
How do you delete a file for good?
Mark it deleted, drop chunk counts after the retention window, and let garbage collection remove chunks that hit zero.
Check yourself
Question 1 of 3Score 2 out of 3 or better to mark this page complete.