Consistency and CAP
Copies of data on many machines can disagree, especially during network failures. You choose which guarantees to keep.
- R + W > Nquorum rule so reads overlap the latest write
- N=3, W=2, R=2balanced default, survives one node down
- 70-100 mscross-Atlantic round trip PACELC pays
Show what to say
“CAP is about the moment of a network partition: the minority side either refuses requests to stay consistent or answers possibly stale data to stay available. It's not pick two of three. So I choose per feature: strong for balances and checkout stock, read-your-writes for a user's own edits, eventual for likes and counters.”
Trade-offs
- CP vs AP: CP refuses some requests during a split but never double-sells; AP keeps answering but writes on both sides may conflict.
- Latency vs consistency (PACELC): waiting on another region costs a round trip; async replication skips it but replicas may be stale.
- Strict vs sloppy quorum: sloppy keeps writes up through failures, but R + W > N no longer guarantees reads overlap the latest write.
Bring it up before they ask
A lock with a lease needs a fencing token, because a paused client can't see its lease expired and may overwrite the new owner.
New to the words? Read the one-minute primer
When the network splits, a system must either refuse some requests (consistency) or answer with possibly stale data (availability). Pick per feature: strong for money, for your own edits, eventual for counters.
Why it matters
Your data lives on three machines. A switch dies. Now two copies disagree, and users may see either.
Read more · 156 words
Copies survive dead servers, but they lag, and a split lets them drift apart. Every choice in this page is an answer to one question: what should the system do when the copies disagree?
Play Partition: CP choice, then Partition: AP choice: one split, two opposite answers.
Skip all of this if you run one database in one region and read from its . A single node is consistent by default, and most products never need more. The trouble starts with , a second region, or a .
Tap any box to see what it is and what it talks to.
- Client
- Database
Text description
- Client A books seat 12A for Alice. The write goes to the leader.
- The leader appends it to its log and streams it to both followers.
- Followers acknowledge. A majority (2 of 3) has the write, so it is committed.
- The leader confirms to Client A.
- Client B reads seat 12A from Follower 2.
- It sees Alice: the replicas agree. (With async replication this read could lag by a few ms.)
CAP and PACELC
When the network splits, each side cannot hear the other, and each side must choose between consistency and availability.
Read more · 579 words
- Consistency: refuse or wait, so nobody reads old data.
- Availability: keep answering, even if the answer is stale.
A bank picks consistency. Two data centers lose their link while you and your partner each withdraw the last 100: if both sides pay, the bank loses 100. "Try again later" is cheaper. A like count picks availability: showing 1,204 instead of 1,207 hurts nobody.
So "pick 2 of 3" is wrong. You choose only while the network is split. When it is healthy, you get both. Partitions cannot be avoided in a real network, so the real choice is always between the other two.
| Choice | How it works | Use it for | Watch out for |
|---|---|---|---|
| CP | Stops answering on the minority side of a split, so no two sides disagree. (Play Partition: CP choice: Follower 2 holds 1 of 3 copies, cannot commit, and rejects Bob's booking.) | Seats, balances, inventory at checkout, unique usernames: a wrong answer costs more than no answer | Some users get an error until the split heals, so the client must retry safely |
| AP | Keeps every replica writable, so it always answers; replicas disagree until they merge. (Play Partition: AP choice: both sides accept a cart edit and the merge decides what survives.) | Carts, likes, feeds, presence: being down is worse than briefly wrong | Conflict handling (see Quorums and conflicts) |
In , C means linearizable (one copy, always current) and A means every non-failed node answers every request. Both are strict. Real systems sit between: a store stays "mostly available" and "mostly consistent" and lets you tune it per request. So the better interview question is not "CP or AP?" but "what does this feature do when the answer might be wrong?"
: the trade-off you pay every day. Splits are rare, but waiting on another region happens on every request: about 70 to 100 ms across the Atlantic. So even on a good day you choose a fast answer from a nearby copy that may be stale, or a slower answer that is always fresh. PACELC: if Partitioned, choose Availability or Consistency; Else, choose or Consistency.
Play Five reads in a row, then compare with Local read. A page that makes 5 reads from a far-away leader waits roughly 5 x 80 = 400 ms. A local copy costs a few ms each. Pay that wait for balances, not for like counts.
| System | During a split | Otherwise |
|---|---|---|
| Cassandra, Dynamo-style | Stays available | Fast |
| Spanner, etcd | Stays consistent | Consistent |
| Postgres + async replicas | Leader side works | Fast, may lag |
Many stores let you choose per request, which is how one system serves both a balance and a like count. DynamoDB reads are by default, and a strongly consistent read costs twice the read capacity. Cassandra lets each query pick ONE or QUORUM.
Tap any box to see what it is and what it talks to.
- Client
- Database
Text description
- The page reads the like count from the replica in Paris.
- Answer in a few ms. The replica is about 200 ms behind, so the count may be 3 short. Harmless for likes.
Consistency models
"Strong" is not one thing. Strongest first; each rung down is faster, with more surprises.
Read more · 731 words
Do not pick one level for the whole product: pick per feature.
| Model | Guarantee | Use it for | Cost / watch out for |
|---|---|---|---|
| Linearizable | Once a write is acknowledged, every later read from any client returns it, like one copy | Money, seat and stock counts, unique usernames, leader election | A round trip to the leader or a majority on every read (about 80 ms across the Atlantic); the minority side of a split refuses. Spanner and etcd give it by default; in DynamoDB you ask per read |
| Sequential | One write order for all, maybe late (ZooKeeper read without sync) | Rarely named out loud | Everyone sees one order, but it need not match real time |
| Causal | If B was written after seeing A, nobody sees B without A. Unrelated writes may show in any order | Chat threads, comments, collaborative editing | A little metadata per write (a dependency id or vector clock). The strongest model that still works on both sides of a partition, so it suits AP systems that cannot allow nonsense |
| Read-your-writes | You always see your own changes, even if others' lag | Profiles, settings, the comment you just posted | The cheapest guarantee that stops an app looking broken (recipes below) |
| Eventual | If writes stop, all replicas converge. (Play Stale, then fresh: the reader sees 1,204, then 1,207) | Likes, views, follower counts, DNS, search indexes: off by a few for seconds is harmless | Surprises and conflicts. Never for balances or stock |
Three ways to get :
- Pin to the leader. After a write, read that user from the leader for longer than the lag (5 s is a safe default). The arithmetic is mild: if 1,000 users save per second and each is pinned for 5 s, about 5,000 users read from the leader at any moment. This is the default.
- Pass a position. Return the write's log position, and read only from a replica that has reached it.
- Sticky replica. Send the user to one replica that you also write through.
The bug users see: you change your display name, the page reloads from a replica 200 ms behind, and it shows the old name. You type it again, certain the save failed. It is the commonest consistency bug users actually see. Drag up, switch read-your-writes on, then write and read. Do not fix it by making every read go to the leader, which just moves the load.
Pick per feature. Ask what a wrong answer costs, then whether users see their own edit, then whether order matters.
| Feature | Consistency | Why |
|---|---|---|
| Balance, payments, checkout stock | Strong | A stale read means double-spending |
| Your own profile edit | Read-your-writes | Seeing your old name after saving looks broken |
| Chat in one conversation | Same order for everyone | A reply before its question confuses |
| Likes and views | Eventual | Off by a few for seconds is harmless |
The movie booking question applies this to seats. Every rung up the ladder is paid for in and availability:
| Choice | What it costs | Rough figure |
|---|---|---|
| Read a local replica | A few ms, may be stale | A few ms |
| Read the far leader | A cross-region round trip | 70 to 100 ms |
| Majority write, one data center | Network plus disk flush | A few ms |
| Majority write, across the Atlantic | A round trip to the nearest peer in the majority | Tens of ms |
| DynamoDB strong read | Double the read capacity | 1 unit per 4 KB, against 0.5 |
Sum for a checkout page with 3 reads and 1 write across regions. All 3 reads to the far leader: 3 x 80 + 80 = 320 ms. If 2 are catalogue reads from a local replica at a few ms each: 80 + 80 + 2 x 3 = about 170 ms. Pay the cross-region trip for the balance and the stock check only.
- Client
- Service
- Database
Text description
- User → App: save name: Asha
- App → Leader: write
- Leader replies to App: ok
- Leader sends (async) to Replica: replicate (200 ms lag)
- User → App: reload profile
- App → Replica: read
- Replica replies to App: old name
Does a stale or conflicting read cost money or break a rule?
Quick checkWhy keep strong reads to the balance and stock check only?
Quorums and conflicts
Each fact is saved on N copies.
Read more · 809 words
How many must answer before you trust a read? A write waits for W copies. A read asks R copies and keeps the newest. If R + W > N, every read touches at least one copy with the latest write. Play the three tabs: Write, W = 2, Read, R = 2: overlap, then Read, R = 1: stale.
| N, W, R | Good for | Watch out for |
|---|---|---|
| 3, 2, 2 (the default to say first) | User data that must not be stale but need not be strictly linearizable. 2 + 2 = 4 is more than 3, so the read and write sets share a replica, and either survives one node down. Cassandra calls it QUORUM: floor(N / 2) + 1. Each read or write costs the second-fastest of three replicas | None: it balances both |
| 3, 3, 1 | Fast reads | No write survives a dead node |
| 3, 1, 1 | Speed (latency of the fastest replica), and it keeps working with 2 of 3 nodes down: view counts, feeds, sensor readings | 1 + 1 = 2 is not more than 3: a read can land on a replica that never saw the write |
| 5, 3, 2 | A trap | 3 + 2 is not more than 5, so the 2 replicas you read can both be the 2 that missed the write. Fix with R = 3 (3 + 3 = 6) or W = 4 |
- Sloppy and . A stand-in node takes a write when a key's home is down, and hands it back later. Writes stay up through failures (a shopping cart), but the overlap rule is gone: a read of the home replicas will not see the write until the handoff. and background Merkle-tree comparison close the gap afterwards.
- Availability cost. Say each node is up 99% of the time, failures independent. W = 3 needs all three up: 0.99 x 0.99 x 0.99 = 97.03%. W = 2 needs any two: 3 x 0.99^2 x 0.01 + 0.99^3 = 99.97%. W = 1 needs any one: 1 - 0.01^3 = 99.9999%. Real failures are correlated (one rack, one switch, one bad deploy), so treat these as best cases. Every extra you wait for buys consistency and sells availability.
Set R + W equal to N, write, then read a few times. The guarantee is gone.
Conflict resolution. Two replicas took writes to the same key. Who wins? Four answers, simplest to safest. The default: a or an app-level merge for user data, and last-writer-wins only where losing one write costs nothing.
| Option | How it works | Use it for | Avoid it for / watch out for |
|---|---|---|---|
| Last-writer-wins | Keeps the write with the highest timestamp and silently drops the other (the cart book disappears with no error) | Only the newest matters: last-seen time, a cached copy, a sensor reading | Carts, counters, documents. It trusts clocks: a laptop 3 s slow writes a new photo 2 s after the old one, stamps it lower, and LWW keeps the old photo. Use version numbers, vector clocks, a CRDT, or a single leader instead |
| Vector clocks | Each node counts its own writes; comparing clocks says whether one write came before the other or they were concurrent. They detect conflicts, they do not fix them | Dynamo-style stores where the app resolves conflicts (vector-clock.js below) | Metadata grows with the number of writers; someone still writes the merge |
| CRDTs | A data type designed so merging in any order gives the same result: counters, sets, carts, co-edited text. Nothing is lost; a repeated merge changes nothing | Both sides must keep accepting writes: like counts, carts, collaborative docs. Riak and Redis Enterprise active-active ship them | The type fixes the semantics (a removed item may reappear unless the set tracks removals), so match type to feature |
| App-level merge | The business rule decides: union the two carts, keep the larger quantity, or ask a human | No generic type captures the rule, such as merging two edits to an address | Code you must test, and a user-visible choice for cases you cannot automate |
Tap any box to see what it is and what it talks to.
- Client
- Service
- Database
Text description
- The client writes x = 2. The coordinator sends it to all 3 replicas.
- Replicas 1 and 2 store it. Replica 3 is slow.
- Two acks arrive: W = 2 is met, so the write succeeds. Replica 3 still has the old value.
- The client gets success after 2 acks, not 3.
Show data table
| minutes | |
|---|---|
| W = 3 | 260 h |
| W = 2 | 2.6 h |
| W = 1 | 32 s |
Consensus, transactions and locks
Some questions need exactly one answer: who leads, who holds the lock.
Read more · 705 words
picks a leader by majority vote and keeps one ordered log. An entry is safe once a majority stores it, so any minority can be lost. Play Commit on 2 of 3, One node down, then Leader cut off.
Any two majorities of the same cluster share at least one node. So two leaders can never both gather a majority, and a committed entry is always known to the next leader. That one overlap is the whole safety argument.
| Nodes | Majority | Failures tolerated |
|---|---|---|
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
| 7 | 4 | 3 |
- Use odd cluster sizes: a 4th node tolerates no more failures than 3, and every commit waits for one more . Run 3 nodes to survive 1 failure, 5 to survive 2. etcd's guidance is no more than 7.
- Elections and terms. Followers expect a . If none arrives before a random timeout (the Raft paper suggests 150 to 300 ms), a follower becomes a candidate, bumps the term and asks for votes. Press Kill leader and watch a new term begin. Then Partition a node: alone, it cannot win.
- Use it, do not build it. You use through etcd, ZooKeeper or Consul. Reach for it for , locks, configuration and cluster metadata: small data where a wrong answer is expensive. Keep it off the data path: each commit is a round trip to a majority, a few ms inside one data center and tens of ms if the majority spans regions.
Distributed . One order touches payments, stock and shipping, each somewhere else. The best answer is to avoid the problem: keep data that must change together in one database and use an ordinary transaction. When you cannot, choose:
| Two-phase commit (2PC) | Saga | |
|---|---|---|
| How it works | A coordinator asks everyone to prepare (lock rows, vote), then commits only if all said yes | Chains local transactions, each with an undo step (refund, release stock); on failure it runs the undos in reverse |
| Atomic | All or nothing | Eventually, via undo steps |
| If coordinator dies | Participants block, holding locks | Saga resumes from its log |
| Use it for | Inside one database cluster | Across services, as in Step 5 |
| Cost | Blocking between the phases | The customer briefly saw a charge; every step needs an undo and an idempotency key |
Distributed locks need . A lock with an expiry is not enough. Client 1 takes a 10 s lock, then freezes for 15 s (a garbage-collection pause) and the lock expires. Client 2 takes the lock and writes. Client 1 wakes, still thinks it holds the lock, and overwrites Client 2.
Client 1 cannot see its own pause, so storage checks. Each grant carries an ever-increasing fencing token, and storage rejects any older one. A write with token 33 after token 34 was seen updates zero rows.
etcd revisions and ZooKeeper's zxid work as tokens; the fenced-lock.js sample issues them from a counter. A Redis lock is fine to avoid duplicate work. For correctness, fence the storage, or let the database arbitrate with a conditional write. Answer the two questions in the tree.
Tap any box to see what it is and what it talks to.
- Client
- Database
Text description
- The client writes. The leader adds the entry to its own log: 1 of 3.
- It sends the entry to both followers.
- Follower 1 acknowledges. That is 2 of 3, a majority: the entry is committed.
- The leader answers the client without waiting for the slow Follower 2.
UPDATE files SET body = $1, fence = $2
WHERE id = $3 AND fence <= $2;
-- 0 rows updated: a newer holder already wrote, so rejectWould two holders at once corrupt data (not just waste work)?
Failure modes
Distributed systems fail in ways one machine cannot.
Read more · 227 words
At 2:14 on a Friday afternoon a switch between two data centers drops for 40 seconds; what happens next is decided by the choices above.
| Failure | What happens | Fix |
|---|---|---|
| Split brain | Two nodes both think they lead. A standby waits 30 s for a heartbeat, promotes itself, the old leader is still alive: Alice and Bob both book seat 12A | A majority: an odd number of voters, a node leads only while it reaches more than half, the old leader steps down when it cannot. Fencing tokens so a stale leader's writes are rejected downstream. Play Two leaders, then A majority decides |
| Stale read after write | You save, reload from a replica 200 ms behind, see the old value | Read-your-writes (pin to the leader for a few seconds), not every read to the leader |
| Clock skew | Machines disagree about the time, so last-writer-wins drops the newer write | Logical clocks, CRDTs, or no LWW |
| Ambiguous timeout | A timeout says the reply did not arrive, not that the write did not happen: a payment commits, the ack is lost, the client retries after 3 s, the customer is charged twice | An idempotency key on every write that must happen once, so a retry returns the first result. The same idea protects queue consumers, as in Step 5 |
| Lock expiry | A paused holder overwrites the new one | Fencing tokens |
Tap any box to see what it is and what it talks to.
- Client
- Database
- Service
Text description
- The link between the data centers fails. Both nodes are alive.
- The standby sees no heartbeat for 30 s and promotes itself. It never learns the old leader is alive.
- Alice books seat 12A on the old leader.
- Bob books the same seat 12A on the new leader. Both get success.
- The link heals. Two histories disagree about one seat, and one booking must be thrown away.
Quick checkTwo nodes both think they lead after a partition. What is it?
Code
In crdt.js, last-writer-wins loses data; the counter does not.
Read more · 42 words
fenced-lock.js issues tokens.
// N replicas; a write waits for W acks, a read asks R replicas and keeps the newest version
export const quorum = (N, W, R) => ({
strong: R + W > N, // every read set overlaps the latest write set
writeTolerates: N - W, // replicas that can be down while writes still succeed
readTolerates: N - R,
});
// quorum(3, 2, 2) → { strong: true, writeTolerates: 1, readTolerates: 1 }
// quorum(3, 1, 1) → strong: false (fast, but a read can be stale)
// quorum(5, 3, 2) → strong: false (3 + 2 is not greater than 5)
// Consensus (Raft, Paxos) needs a majority to elect a leader and commit
export const majority = (n) => Math.floor(n / 2) + 1;
export const failuresTolerated = (n) => n - majority(n);
// 3 nodes → majority 2, tolerates 1 · 4 nodes → majority 3, still tolerates 1 · 5 nodes → tolerates 2Consistency and CAP in the interview
Bring this up the moment your design has more than one copy of the data: , a second region, a , or a multi-service write.
Read more · 507 words
Then walk through five decisions in order.
Ask what a wrong answer costs
For each feature, ask: does a stale read lose money or break a rule? Seats and balances do; likes do not.
Say CAP precisely, then choose per feature
" only bites during a partition. Strong for money, for a user's own edits, eventual for counters."
Pick the replication and read path
A leader with async replicas by default. Pin a user to the leader for about 5 s after a write, and send everything else to local replicas.
Set the quorum or the consensus
N = 3, W = 2, R = 2 for a leaderless store. For a leader, a lock or a config, use etcd or ZooKeeper, with an odd cluster size.
Own the downsides
Name conflicts and how you merge them, , the ambiguous timeout, and on any lock that guards correctness.
| If the interviewer says | You raise |
|---|---|
| "Two data centers lose their link" | CP for money (the minority refuses), AP for likes (merge later) |
| "A user saved, then saw the old value" | Read-your-writes: pin to the leader for 5 s |
| "Two people edit the same cart" | A CRDT or app merge, not last-writer-wins |
| "Charge the card and reserve stock" | A saga with undo steps and idempotency keys |
| "Only one worker may run this job" | A lease lock; add a fencing token if it guards data |
Quick checkWhat is the first question to ask about each feature?
Interview questions
When would you choose eventual consistency?
When stale reads are harmless: likes, views, feeds. Never for money or checkout stock.
Explain CAP with an example.
Two lose their link. If both keep selling seats, one seat sells twice. So ticketing refuses writes on the minority side.
How do you get read-your-writes on an eventually consistent store?
For a few seconds after a write, read from the leader or a replica that has caught up to your write.
Why do distributed locks need fencing tokens?
A paused client cannot tell its lock expired. Increasing tokens let storage reject its stale write.
2PC or a saga?
is atomic but blocks if the coordinator dies, so keep it inside one database cluster. Across services, use .
Check yourself
Question 1 of 5Score 2 out of 3 or better to mark this page complete.