Design a Distributed Key-Value Store
A Dynamo-style store: consistent hashing places keys, quorums balance freshness against availability, and vector clocks, hinted handoff and Merkle trees handle the failures.
- ~44K ops/sper node across 50 nodes
- ~2%of keys move when a 51st node joins
- R + W > Nquorum rule: 2 + 2 > 3
Show what to say
“I'd build it Dynamo-style. Keys and nodes are hashed onto a ring with 128 virtual nodes each, and every key lives on the next three distinct nodes. Any node coordinates: a write waits for W = 2 acks, a read asks R = 2, so R + W above N gives read-your-latest. Vector clocks keep conflicting writes as siblings, and hinted handoff, read repair and Merkle trees repair drift.”
Trade-offs
- Leaderless vs leader per shard: always writable with no failover, but concurrent writes create conflicts to resolve.
- Vector clocks vs last-write-wins: no lost updates, but clients must merge siblings; LWW is simple but clock skew drops writes.
- Sloppy quorum vs strict: higher availability during failures, but a read can miss a very recent write.
Bring it up before they ask
Tombstones: a delete must replicate as a record and be kept past the repair interval, or anti-entropy resurrects the deleted value.
Spread keys over a consistent-hash ring, keep each key on N = 3 nodes, and let every node coordinate. A write waits for W and a read asks R ; R + W greater than N makes reads see the latest write. , and Merkle-tree repair deal with the failures.
Functional
- put(key, value), get(key) and delete(key)
- Values up to a few KB, keys up to a few hundred bytes
- Tunable consistency per request
- Add and remove nodes without downtime
- Optional expiry (TTL)
Non-functional
- About 1M operations a second at peak, 80% reads
- p99 latency under 10 ms
- Highly available: writes still work when a node or a zone is down
- Scales to 100 TB of data by adding nodes
- No single point of failure: no leader to lose
Out of scope
- Multi-key transactions and joins
- Secondary indexes and range scans across the cluster
- Cross-region replication (a follow-up)
Clarifying questions, scale and interface
Read more · 351 words
- Consistency or availability? If a node is down, may a read be stale, or must it fail?
- Data and access? Value size, volume, read or write heavy, .
- Conflicts? Can two clients write one key at once? Who merges?
| Quantity | Calculation | Result |
|---|---|---|
| Logical data | given | 100 TB |
| With replicas | 100 TB × 3 copies | 300 TB |
| Nodes | 300 TB ÷ 6 TB usable per node | 50 nodes |
| Writes | 20% of 1M ops/s, × 3 replicas | 200K/s → 600K replica writes/s |
| Reads | 80% of 1M ops/s, × R = 2 | 800K/s → 1.6M replica reads/s |
| Per node | (600K + 1.6M) ÷ 50 | ≈ 44K ops/s |
| Ring points | 50 nodes × 128 virtual nodes | 6,400 |
Forty-four thousand small operations a second is comfortable for a node with an on SSD. Capacity comes from adding nodes, not from a bigger machine.
| Call | Notes |
|---|---|
PUT /kv/:key | Body is the value. Optional context (the clock from a previous get), w, ttl |
GET /kv/:key | Returns one value or several siblings, plus a context. Optional r |
DELETE /kv/:key | Writes a tombstone, removed after a grace period |
The context is the a get returned. Sending it back on the next put says "I have seen these versions", so the new value replaces them.
Each node stores records in an LSM tree: writes go to a commit log and an in-memory table, which is flushed to sorted files and merged in the background.
| Field | Meaning |
|---|---|
key | The partition key, hashed onto the ring |
value | Opaque bytes |
clock | Vector clock, for example {X:2, Y:1} |
tombstone | True for a delete, so the delete replicates and repair does not resurrect the value |
expires_at | Optional TTL |
Architecture
Every node runs the same code: any node looks up the key's and fans the request out.
Read more · 57 words
Play Write, Read and , Replica down and repair.
Tap any box to see what it is and what it talks to.
- Client
- Service
- Cache
- Database
- Worker
Text description
- put('cart:42', value). The SDK sends it to any node; this one becomes the coordinator.
- The coordinator hashes the key onto the ring. The next three distinct nodes clockwise are the replicas.
- It sends the write to all three replicas at once, stamped with a version (a vector clock).
- Replica 1 appends to its log and acknowledges.
- Replica 2 acknowledges. That is W = 2 acks, so the coordinator can answer the client now.
- The client gets success. Replica 3 is slow, and will catch up on its own.
Partitioning
Hash nodes and keys onto a circle.
Read more · 101 words
A key lives on the first node clockwise, and the next two distinct nodes hold its copies. Add a node and about 1 in 51 keys move, all onto it. Plain hash mod N would move over 90%.
Give each machine many . One point each leaves some nodes with three times their share; 128 keep the largest within about 20% of average. Try it: add and remove nodes, then change the virtual nodes.
Replication and quorums
Each key lives on N = 3 nodes.
Read more · 116 words
A write waits for W ; a read asks R and takes the newest. If R + W is above N, every read overlaps every write, so a read sees the latest acknowledged write.
| Setting | Behaviour |
|---|---|
| W = 2, R = 2 | Balanced; survives one dead replica |
| W = 1, R = 3 | Fast writes, slow reads |
| W = 3, R = 1 | Fast reads; one dead node blocks writes |
Try values and kill replicas.
Conflicts and repair
With no leader, two writes can meet.
Read more · 215 words
A records who has seen what. If one clock includes the other, keep the newer. If neither does, keep both as siblings for the client to merge. Last-write-wins is simpler, but clock skew can drop the newer write.
Drift is repaired three ways. fixes a read touched. has a stand-in hold writes for a down replica. compares and copies only mismatching ranges.
- Client
- Database
Text description
- Client A → Node X: put cart=[pen], clock {X:1}
- Client B → Node Y: put cart=[mug], clock {X:1}
- Client A → Node X: get cart
- Node X replies to Client A: siblings: [pen], [mug]
- Client A → Node X: put [pen, mug], context {X:2,Y:1}
Step-by-step answer
Five moves, in order: clarify, estimate (50 nodes, 44K operations per node, 6,400 ring points), partition, replicate, then repair and fail.
Read more · 470 words
Choose the model first
"I'll build a Dynamo-style store: always writable, by default, tunable per request." Then say what you give up.
Place keys with a ring
Hash keys and nodes onto a ring. are the next three distinct nodes. even out load and let big machines take more.
Make every node a coordinator
Nodes learn the ring by , so any node can route a request. No leader means nothing to fail over.
Set N, W and R
N = 3, W = 2, R = 2 by default. Lower W for write speed, raise R for freshness. State R + W greater than N.
Version values
Attach a . Return siblings on conflict, or use last-write-wins where overwrites are acceptable and say that out loud.
Survive failures
Sloppy with keeps writes flowing when replicas are down. and Merkle-tree bring replicas back in line.
Detect failure with gossip
Each node tracks counters from peers. A silent peer is marked down by many nodes, not just one, which avoids flapping on a single slow link.
Store it on disk
Commit log first, then a memtable, flushed to sorted files that are merged in the background. skip files that cannot hold a key.
Implementation
Tabs: the ring with and a preference list, , a small cluster with and , and a diff.
Read more · 65 words
- The ring test shows 1 point per node is uneven (some node over 3x), 128 is within 1.4x, and adding the 51st node moves about 2%.
- The quorum tests show W = 1, R = 1 reading stale data and W = 2, R = 2 never doing so.
- The Merkle test finds one stale key among 10,000 in 21 comparisons instead of scanning 1,024 leaves.
// A consistent-hash ring with virtual nodes and a preference list.
// - Keys and nodes are hashed onto the same circle of 2^32 positions.
// - A key belongs to the first node clockwise from its position; the next distinct nodes hold its replicas.
// - Each physical node owns many small arcs (virtual nodes), so load is even and a join or leave moves only ~1/N of the keys.
// 32-bit FNV-1a followed by a finaliser, so similar strings (node-1#1, node-1#2) land far apart.
export function hash32(str) {
let h = 0x811c9dc5;
for (let i = 0; i < str.length; i++) {
h ^= str.charCodeAt(i);
h = Math.imul(h, 0x01000193);
}
h ^= h >>> 16; h = Math.imul(h, 0x85ebca6b);
h ^= h >>> 13; h = Math.imul(h, 0xc2b2ae35);
h ^= h >>> 16;
return h >>> 0;
}
export class Ring {
#points = []; // sorted [{ pos, node }]
constructor(nodes = [], vnodes = 128) {
this.vnodes = vnodes;
this.nodes = new Set();
nodes.forEach((n) => this.add(n));
}
add(node) {
this.nodes.add(node);
for (let i = 0; i < this.vnodes; i++) this.#points.push({ pos: hash32(`${node}#${i}`), node });
this.#points.sort((a, b) => a.pos - b.pos);
}
remove(node) {
this.nodes.delete(node);
this.#points = this.#points.filter((p) => p.node !== node);
}
// Index of the first point at or after the key's position (binary search), wrapping around the circle.
#start(key) {
const h = hash32(key);
let lo = 0, hi = this.#points.length;
while (lo < hi) {
const mid = (lo + hi) >> 1;
if (this.#points[mid].pos < h) lo = mid + 1; else hi = mid;
}
return lo % this.#points.length;
}
owner(key) {
return this.#points[this.#start(key)].node;
}
// The n distinct physical nodes responsible for a key: the coordinator first, then its replicas.
preferenceList(key, n = 3) {
const out = [];
const total = this.#points.length;
for (let i = this.#start(key), seen = 0; out.length < Math.min(n, this.nodes.size) && seen < total; i = (i + 1) % total, seen++) {
const node = this.#points[i].node;
if (!out.includes(node)) out.push(node); // skip further vnodes of a node we already have
}
return out;
}
}Failure modes and trade-offs
Read more · 287 words
| What breaks | Effect | Mitigation |
|---|---|---|
| A replica goes down | Fewer acks available | W = 2 of 3 still works; hinted handoff |
| Two replicas down | Writes fail at W = 2 | Sloppy quorum, or lower W and accept risk |
| Network partition | Both sides accept writes | Vector clocks keep siblings; merge on read |
| A node loses disk | Missing keys | Anti-entropy rebuilds from the other replicas |
| Hot key | One replica set overloaded | Cache in front; split the key; more replicas for that key |
| Tombstones removed too early | Deleted values come back | Keep tombstones past the longest repair interval |
| Clock skew with LWW | Newer write dropped | Use vector clocks or logical timestamps |
Trade-offs
| Choice | Gain | Cost |
|---|---|---|
| Leaderless (Dynamo style) | Always writable, no failover | Conflicts to resolve |
| Leader per shard (Raft) | Simple strong consistency | Writes stall during an election |
| Vector clocks | No lost updates | Siblings; clients must merge |
| Last-write-wins | Simple | Silent data loss with skew |
| Sloppy quorum | Higher availability | Reads can miss recent writes |
Follow-up questions
How would you make it strongly consistent?
Use a group per key range, so one leader orders writes. You lose some availability during elections.
How do you add a node safely?
Join the ring, stream the arcs it owns, then serve. Old copies go after the stream ends.
How do you handle a hot key?
in front, replicate that key more widely, or split it into sub-keys.
Check yourself
Question 1 of 3Score 2 out of 3 or better to mark this page complete.