BackendHard50 min

Design a Distributed Key-Value Store

A Dynamo-style store: consistent hashing places keys, quorums balance freshness against availability, and vector clocks, hinted handoff and Merkle trees handle the failures.

Say it in 30 seconds
  • ~44K ops/sper node across 50 nodes
  • ~2%of keys move when a 51st node joins
  • R + W > Nquorum rule: 2 + 2 > 3
Show what to say

“I'd build it Dynamo-style. Keys and nodes are hashed onto a ring with 128 virtual nodes each, and every key lives on the next three distinct nodes. Any node coordinates: a write waits for W = 2 acks, a read asks R = 2, so R + W above N gives read-your-latest. Vector clocks keep conflicting writes as siblings, and hinted handoff, read repair and Merkle trees repair drift.”

Trade-offs

  • Leaderless vs leader per shard: always writable with no failover, but concurrent writes create conflicts to resolve.
  • Vector clocks vs last-write-wins: no lost updates, but clients must merge siblings; LWW is simple but clock skew drops writes.
  • Sloppy quorum vs strict: higher availability during failures, but a read can miss a very recent write.

Bring it up before they ask

Tombstones: a delete must replicate as a record and be kept past the repair interval, or anti-entropy resurrects the deleted value.

0 of 0 sections ticked
30-second answer

Spread keys over a consistent-hash ring, keep each key on N = 3 nodes, and let every node coordinate. A write waits for W and a read asks R ; R + W greater than N makes reads see the latest write. , and Merkle-tree repair deal with the failures.

Functional

  • put(key, value), get(key) and delete(key)
  • Values up to a few KB, keys up to a few hundred bytes
  • Tunable consistency per request
  • Add and remove nodes without downtime
  • Optional expiry (TTL)

Non-functional

  • About 1M operations a second at peak, 80% reads
  • p99 latency under 10 ms
  • Highly available: writes still work when a node or a zone is down
  • Scales to 100 TB of data by adding nodes
  • No single point of failure: no leader to lose

Out of scope

  • Multi-key transactions and joins
  • Secondary indexes and range scans across the cluster
  • Cross-region replication (a follow-up)

Clarifying questions, scale and interface

Read more · 351 words
  • Consistency or availability? If a node is down, may a read be stale, or must it fail?
  • Data and access? Value size, volume, read or write heavy, .
  • Conflicts? Can two clients write one key at once? Who merges?
QuantityCalculationResult
Logical datagiven100 TB
With replicas100 TB × 3 copies300 TB
Nodes300 TB ÷ 6 TB usable per node50 nodes
Writes20% of 1M ops/s, × 3 replicas200K/s → 600K replica writes/s
Reads80% of 1M ops/s, × R = 2800K/s → 1.6M replica reads/s
Per node(600K + 1.6M) ÷ 50≈ 44K ops/s
Ring points50 nodes × 128 virtual nodes6,400

Forty-four thousand small operations a second is comfortable for a node with an on SSD. Capacity comes from adding nodes, not from a bigger machine.

CallNotes
PUT /kv/:keyBody is the value. Optional context (the clock from a previous get), w, ttl
GET /kv/:keyReturns one value or several siblings, plus a context. Optional r
DELETE /kv/:keyWrites a tombstone, removed after a grace period

The context is the a get returned. Sending it back on the next put says "I have seen these versions", so the new value replaces them.

Each node stores records in an LSM tree: writes go to a commit log and an in-memory table, which is flushed to sorted files and merged in the background.

FieldMeaning
keyThe partition key, hashed onto the ring
valueOpaque bytes
clockVector clock, for example {X:2, Y:1}
tombstoneTrue for a delete, so the delete replicates and repair does not resurrect the value
expires_atOptional TTL

Architecture

Every node runs the same code: any node looks up the key's and fans the request out.

Read more · 57 words

Play Write, Read and , Replica down and repair.

Press Play to watch it run · 6 steps

Any node can coordinate a request. It hashes the key onto the ring, finds the three replicas in the key's preference list, and sends the write or read to them. A write succeeds after W acknowledgements, a read after R responses. Down replicas get their writes through a stand-in (hinted handoff), and a background process compares Merkle trees to repair drift.

Distributed key-value storeAny node can coordinate a request. It hashes the key onto the ring, finds the three replicas in the key's preference list, and sends the write or read to them. A write succeeds after W acknowledgements, a read after R responses. Down replicas get their writes through a stand-in (hinted handoff), and a background process compares Merkle trees to repair drift.put / getpreference listreplicatereplicatereplicatehinthand backcompare treescompare treesClient SDKknows the ringWhichever node receives the request. It is not special: every node can compute the key's replicas, so there is no leader and no single point of failure.Coordinatorany nodeEvery node keeps the full ring in memory, learned by gossip: each node tells a few random peers what it knows every second.Ring + gossipmembershipReplica 1LSM storageReplica 2LSM storageReplica 3LSM storageBackground job. Replicas of a key range exchange Merkle tree hashes and copy only the keys under mismatching branches.Anti-entropyMerkle treesThe next healthy node on the ring. It stores writes meant for a down replica and hands them over when it returns.Stand-in nodeholds hints

Tap any box to see what it is and what it talks to.

  • Client
  • Service
  • Cache
  • Database
  • Worker
Open in playground
Text description
  1. put('cart:42', value). The SDK sends it to any node; this one becomes the coordinator.
  2. The coordinator hashes the key onto the ring. The next three distinct nodes clockwise are the replicas.
  3. It sends the write to all three replicas at once, stamped with a version (a vector clock).
  4. Replica 1 appends to its log and acknowledges.
  5. Replica 2 acknowledges. That is W = 2 acks, so the coordinator can answer the client now.
  6. The client gets success. Replica 3 is slow, and will catch up on its own.

Partitioning

Hash nodes and keys onto a circle.

Read more · 101 words

A key lives on the first node clockwise, and the next two distinct nodes hold its copies. Add a node and about 1 in 51 keys move, all onto it. Plain hash mod N would move over 90%.

Give each machine many . One point each leaves some nodes with three times their share; 128 keep the largest within about 20% of average. Try it: add and remove nodes, then change the virtual nodes.

Interactive

Consistent hashing ring

Add or remove a node: only the keys next to it move. Raise virtual nodes to even out the load.

4 nodes20 vnodes each

Add or remove a node to see how many keys move. Outlined dots are keys that changed owner.

Keys per node

  • Node A30 keys
  • Node B21 keys
  • Node C31 keys
  • Node D38 keys

Ideal (dashed): 30 keys each. Spread (max ÷ min): 1.81×.

Replication and quorums

Each key lives on N = 3 nodes.

Read more · 116 words

A write waits for W ; a read asks R and takes the newest. If R + W is above N, every read overlaps every write, so a read sees the latest acknowledged write.

SettingBehaviour
W = 2, R = 2Balanced; survives one dead replica
W = 1, R = 3Fast writes, slow reads
W = 3, R = 1Fast reads; one dead node blocks writes

Try values and kill replicas.

Interactive

Quorum playground

Writes wait for W of N replicas; reads ask R and keep the newest version. When R + W > N the two sets always share a replica.

✓ guaranteed overlap R + W = 4 > N = 3. Every read shares at least 1 replica with every write. Writes survive 1 down; reads survive 1.

ClientAv1Bv1Cv1
  • W: acknowledged the last write
  • R: asked by the last read
  • holds the newest version

Press Write to send a new version to W random replicas, then Read from R random replicas.

Conflicts and repair

With no leader, two writes can meet.

Read more · 215 words

A records who has seen what. If one clock includes the other, keep the newer. If neither does, keep both as siblings for the client to merge. Last-write-wins is simpler, but clock skew can drop the newer write.

Drift is repaired three ways. fixes a read touched. has a stand-in hold writes for a down replica. compares and copies only mismatching ranges.

Press Play to watch it run · 5 steps

Two clients write the same key through different coordinators without seeing each other. Vector clocks detect that neither write supersedes the other and keep both as siblings. Last-write-wins instead keeps the one with the later timestamp and silently drops the other.

Two writes during a partitionTwo clients write the same key through different coordinators without seeing each other. Vector clocks detect that neither write supersedes the other and keep both as siblings. Last-write-wins instead keeps the one with the later timestamp and silently drops the other.Both clients read the cart at clock {X:1}put cart=[pen], clock {X:1}Network split: X and Y cannot talkput cart=[mug], clock {X:1}alt [vector clocks]X stores [pen] at {X:2}. Y stores [mug] at {X:1,Y:1}The split heals. Neither clock includes the other: concurrentget cartsiblings: [pen], [mug]put [pen, mug], context {X:2,Y:1}The merged write supersedes both siblings. No update was lost.
  • Client
  • Database
Text description
  1. Client A → Node X: put cart=[pen], clock {X:1}
  2. Client B → Node Y: put cart=[mug], clock {X:1}
  3. Client A → Node X: get cart
  4. Node X replies to Client A: siblings: [pen], [mug]
  5. Client A → Node X: put [pen, mug], context {X:2,Y:1}

Step-by-step answer

Five moves, in order: clarify, estimate (50 nodes, 44K operations per node, 6,400 ring points), partition, replicate, then repair and fail.

Read more · 470 words
  1. Choose the model first

    "I'll build a Dynamo-style store: always writable, by default, tunable per request." Then say what you give up.

  2. Place keys with a ring

    Hash keys and nodes onto a ring. are the next three distinct nodes. even out load and let big machines take more.

  3. Make every node a coordinator

    Nodes learn the ring by , so any node can route a request. No leader means nothing to fail over.

  4. Set N, W and R

    N = 3, W = 2, R = 2 by default. Lower W for write speed, raise R for freshness. State R + W greater than N.

  5. Version values

    Attach a . Return siblings on conflict, or use last-write-wins where overwrites are acceptable and say that out loud.

  6. Survive failures

    Sloppy with keeps writes flowing when replicas are down. and Merkle-tree bring replicas back in line.

  7. Detect failure with gossip

    Each node tracks counters from peers. A silent peer is marked down by many nodes, not just one, which avoids flapping on a single slow link.

  8. Store it on disk

    Commit log first, then a memtable, flushed to sorted files that are merged in the background. skip files that cannot hold a key.

Implementation

Tabs: the ring with and a preference list, , a small cluster with and , and a diff.

Read more · 65 words
  • The ring test shows 1 point per node is uneven (some node over 3x), 128 is within 1.4x, and adding the 51st node moves about 2%.
  • The quorum tests show W = 1, R = 1 reading stale data and W = 2, R = 2 never doing so.
  • The Merkle test finds one stale key among 10,000 in 21 comparisons instead of scanning 1,024 leaves.
// A consistent-hash ring with virtual nodes and a preference list.
//   - Keys and nodes are hashed onto the same circle of 2^32 positions.
//   - A key belongs to the first node clockwise from its position; the next distinct nodes hold its replicas.
//   - Each physical node owns many small arcs (virtual nodes), so load is even and a join or leave moves only ~1/N of the keys.

// 32-bit FNV-1a followed by a finaliser, so similar strings (node-1#1, node-1#2) land far apart.
export function hash32(str) {
  let h = 0x811c9dc5;
  for (let i = 0; i < str.length; i++) {
    h ^= str.charCodeAt(i);
    h = Math.imul(h, 0x01000193);
  }
  h ^= h >>> 16; h = Math.imul(h, 0x85ebca6b);
  h ^= h >>> 13; h = Math.imul(h, 0xc2b2ae35);
  h ^= h >>> 16;
  return h >>> 0;
}

export class Ring {
  #points = [];                                   // sorted [{ pos, node }]

  constructor(nodes = [], vnodes = 128) {
    this.vnodes = vnodes;
    this.nodes = new Set();
    nodes.forEach((n) => this.add(n));
  }

  add(node) {
    this.nodes.add(node);
    for (let i = 0; i < this.vnodes; i++) this.#points.push({ pos: hash32(`${node}#${i}`), node });
    this.#points.sort((a, b) => a.pos - b.pos);
  }

  remove(node) {
    this.nodes.delete(node);
    this.#points = this.#points.filter((p) => p.node !== node);
  }

  // Index of the first point at or after the key's position (binary search), wrapping around the circle.
  #start(key) {
    const h = hash32(key);
    let lo = 0, hi = this.#points.length;
    while (lo < hi) {
      const mid = (lo + hi) >> 1;
      if (this.#points[mid].pos < h) lo = mid + 1; else hi = mid;
    }
    return lo % this.#points.length;
  }

  owner(key) {
    return this.#points[this.#start(key)].node;
  }

  // The n distinct physical nodes responsible for a key: the coordinator first, then its replicas.
  preferenceList(key, n = 3) {
    const out = [];
    const total = this.#points.length;
    for (let i = this.#start(key), seen = 0; out.length < Math.min(n, this.nodes.size) && seen < total; i = (i + 1) % total, seen++) {
      const node = this.#points[i].node;
      if (!out.includes(node)) out.push(node);              // skip further vnodes of a node we already have
    }
    return out;
  }
}

Failure modes and trade-offs

Read more · 287 words
What breaksEffectMitigation
A replica goes downFewer acks availableW = 2 of 3 still works; hinted handoff
Two replicas downWrites fail at W = 2Sloppy quorum, or lower W and accept risk
Network partitionBoth sides accept writesVector clocks keep siblings; merge on read
A node loses diskMissing keysAnti-entropy rebuilds from the other replicas
Hot keyOne replica set overloadedCache in front; split the key; more replicas for that key
Tombstones removed too earlyDeleted values come backKeep tombstones past the longest repair interval
Clock skew with LWWNewer write droppedUse vector clocks or logical timestamps

Trade-offs

ChoiceGainCost
Leaderless (Dynamo style)Always writable, no failoverConflicts to resolve
Leader per shard (Raft)Simple strong consistencyWrites stall during an election
Vector clocksNo lost updatesSiblings; clients must merge
Last-write-winsSimpleSilent data loss with skew
Sloppy quorumHigher availabilityReads can miss recent writes

Follow-up questions

How would you make it strongly consistent?

Use a group per key range, so one leader orders writes. You lose some availability during elections.

How do you add a node safely?

Join the ring, stream the arcs it owns, then serve. Old copies go after the stream ends.

How do you handle a hot key?

in front, replicate that key more widely, or split it into sub-keys.

Check yourself

Question 1 of 3

Score 2 out of 3 or better to mark this page complete.

1. You grow a 50-node cluster by one node. Which placement scheme moves the least data?