Reliability and Observability
At scale something is always broken. Reliable systems contain failures, degrade gracefully, and make problems visible.
- 27 callsfrom 3 layers each retrying 3 times
- 10,000failures allowed on 10M requests at 99.9%
- 43.2 mindowntime per 30 days at 99.9%
Show what to say
“I assume something is always failing, so I contain it. Every call gets a timeout, retries happen only for idempotent calls, with jittered backoff at a single layer, and circuit breakers and bulkheads fail fast to a fallback like cached bestsellers. I set an SLO such as 99.9% and use the error budget to decide whether to ship or freeze.”
Trade-offs
- Retries vs load: retries heal transient failures, but 3 layers of 3 attempts send 27 calls at a struggling service; retry in one layer.
- Circuit breaker vs always calling: failing fast frees threads and gives the dependency room, but users get the fallback while it is open.
- Four nines vs three: each extra nine cuts downtime 10x but costs far more and demands automated detection and failover.
Bring it up before they ask
Two app servers sharing one database, zone or deploy are not redundant, so check what fails together before counting nines.
New to the words? Read the one-minute primer
Something is always failing, so contain it: a timeout on every call, jittered for safe retries, and with fallbacks. Set and alert on what users feel, not CPU.
Why it matters
Chain 10 services at 99.9% each and only about 99% of requests succeed.
Read more · 66 words
At scale something is always broken. So contain failures, show something useful anyway, and notice before your users do.
Chains multiply their downtime; independent copies multiply their chance of all failing. That one idea explains most of this page: shorten the chain (fallbacks), and add copies that cannot fail together (redundancy).
Play Failure spike, then Half-open probe: the breaker trips, a fallback answers, then it heals.
Tap any box to see what it is and what it talks to.
- Client
- API gateway
- External API
- Cache
Text description
- A request for recommendations reaches the breaker. It is closed, so calls pass through.
- The call goes to the dependency with a 300 ms timeout.
- It answers in 80 ms. Each success resets the failure count to 0.
- The user gets personalised recommendations.
SLOs and error budgets
"Reliable enough" needs a number. Measure something users feel (an ), like requests answered in under 300 ms.
Read more · 385 words
Set a target (an SLO), like 99.9% over 30 days. The customer contract, the SLA, is looser.
An SLI is the share of good events out of valid events, and it must be something a user would complain about. Server CPU is not an SLI; a failed checkout is. Answer the questions in the decision tree to see which SLI fits a service.
Use availability plus for anything a person waits on. Count a request as good only if it is both successful and fast: a 12 s answer is an outage to the user even when it returns 200.
Pick the SLO from what users tolerate, not from what you hope to achieve, because every extra nine costs about ten times more.
| Availability | Downtime per 30 days |
|---|---|
| 99% | 7.2 h |
| 99.9% | 43.2 min |
| 99.99% | 4.32 min |
The arithmetic: 0.1% of a 30-day month is 0.001 x 43,200 minutes = 43.2 minutes. One 45-minute outage uses a whole 99.9% month. At 99.99% you get about 4 minutes, too fast for a human, so must be automatic.
Start with 99.9% for a normal product. Go to 99.99% only for money paths, and say what it costs: automatic failover, multiple zones and a tested runbook.
The is the failure the SLO allows, and it turns reliability into a shared number. At 99.9% and 10M requests a month, 10,000 may fail. Budget left: ship faster. Budget spent: freeze risky changes. The speed of spending is the burn rate: 1 means exactly on budget for the whole window. At 2x the budget is gone on day 15, at 10x on day 3.
The SLO is the internal target; the SLA is the contract, with refunds or penalties, set looser. If the SLA promises 99.9%, aim your SLO at 99.95% so you notice trouble before you owe money.
Does a user wait for each response (an API or a page)?
Show data table
| minutes | |
|---|---|
| 99% | 3.65 days |
| 99.9% | 8.77 h |
| 99.99% | 52.6 min |
| 99.999% | 5.26 min |
- Burn 1x
- Burn 2x
- Burn 10x
Show data table
| Day | Burn 1x (%) | Burn 2x (%) | Burn 10x (%) |
|---|---|---|---|
| 0 | 100 | 100 | 100 |
| 3 | 90 | 80 | 0 |
| 6 | 80 | 60 | 0 |
| 9 | 70 | 40 | 0 |
| 12 | 60 | 20 | 0 |
| 15 | 50 | 0 | 0 |
| 18 | 40 | 0 | 0 |
| 21 | 30 | 0 | 0 |
| 24 | 20 | 0 | 0 |
| 27 | 10 | 0 | 0 |
| 30 | 0 | 0 | 0 |
How failures spread
A failure rarely stays where it starts.
Read more · 278 words
Something slow or dead makes the healthy parts do more work, and they fall over too. Four scenes cover most outages.
| Scene | What happens | Fix |
|---|---|---|
| Retries multiply (retry amplification) | A client, a gateway and a service that each try 3 times send 3 x 3 x 3 = 27 calls to the database for one user request | Retry in one layer only, with a retry budget (next section) |
| A slow dependency eats your threads | A slow call holds its thread; a failure frees it at once. By Little's law, threads busy = requests per second x seconds per call: 500 per second at 80 ms is 40 threads; at 30 s it is 15,000, and a pool of 200 is empty in 0.4 s | A timeout near the dependency's slow case, a breaker, and a bulkhead. Optional calls must never starve the main flow |
| Overload cascades | One failure shifts its load onto the survivors and they fail too. Three servers at 70% busy carry 210 units; lose one and each of the other two needs 105% | With N servers, losing one raises each survivor's load by N / (N - 1). Run below (N - 1) / N: 66% for 3, 75% for 4, 80% for 5. Shed the rest |
| Redundancy that fails together (correlated failure) | Two app servers on one database, zone, deploy pipeline or config push are one failure waiting to happen | For each pair ask what they share. Separate the shared parts, or stagger the change |
Play Retry at every layer, then Retry in one layer.
Match the symptom to the defence before you reach for a pattern. The tree shows the order to say it in an interview.
Tap any box to see what it is and what it talks to.
- Client
- API gateway
- Service
- Database
Text description
- The database has slowed to 5 s. A user request starts, and the client library allows 3 tries.
- Each call that reaches the gateway is retried 3 times on its way to the service: 3 x 3 = 9 calls.
- The service retries the database 3 times for each call: 9 x 3 = 27 calls for one user request.
- At 1,000 requests a second that is 27,000 calls a second at a database that could not serve 1,000. The retries are now the outage.
Is a dependency slow or failing while you are healthy?
Quick checkThree layers each retry 3 times. Calls at the database?
Containing a bad dependency
A call that hangs holds a thread until the caller runs dry.
Read more · 727 words
So every call gets a timeout, set near the dependency's slow case (80 ms normally, so about 250 ms). A retry fixes a blip: one lost packet, and the second try works. It hurts when the service is overloaded: 1,000 requests a second with 3 tries each can become 3,000.
| Tool | How it works | Use it for | Watch out for |
|---|---|---|---|
| Timeout and deadline | A timeout limits one call; a deadline limits the whole request and travels with it. Each hop passes on only the time left, so work stops when the user has given up. Set a timeout from the dependency's p99 plus slack (p99 80 ms, so about 250 ms) | A short timeout everywhere, with the sum of timeouts on a path below the user's deadline | Some HTTP clients have no timeout until you set one |
| Retry | Helps when the failure was transient (dropped packet, restarting pod); hurts when it is load | Idempotent calls and retryable errors: a timeout, a connection reset, a 503. Keep tries to 2 or 3 in total | Never retry a 400 or a 404; the answer will not change. A card charge is not safe to retry without an idempotency key |
| Backoff and jitter | Exponential backoff doubles the wait after each failure; full jitter picks a random time within it: wait = random(0, min(cap, base x 2 to the power attempt)). With base 100 ms: 0 to 100, 0 to 200, 0 to 400 ms | Every retry. Without jitter, 1,000 clients that failed together retry at the same instant and hit the recovering service as a wall | Retry in one layer only |
| Retry budget | Caps retries at a share of normal traffic, typically 10%. Each first attempt earns a little credit and each retry spends ten times as much, so even if every call fails you add at most about 10% extra load | The default for services that call other services (retry-budget.js below) | |
| Idempotency key | The client sends a unique key with the request and the server saves the result under it, so a retry returns the first result. The first try may have worked with only the reply lost | Every POST that creates or charges something. Keep keys at least as long as clients may retry (often 24 hours) |
Switch Retry policy to Exponential + full and watch the peak after recovery drop.
A drowning service needs you to stop calling it. A is a fuse: it trips, calls fail fast, and a fallback answers. It has three states:
- Closed: calls pass; failures are counted.
- Open: after 5 failures, calls fail instantly for about 10 s.
- Half-open: one trial call. Success closes it; failure reopens it.
Play both scenarios: a successful probe, and Probe fails. Use a breaker on every call to a dependency that has a fallback or can be skipped. Trip on an error rate, such as 50% of at least 20 calls in the last 10 s, rather than a raw count, so a quiet service does not trip on one error. The cost: while it is open, users get the fallback even if the dependency has just recovered, for up to one cooldown.
Raise Dependency error rate until the breaker trips, then lower it and watch the half-open probe close it. Failure threshold sets how many failures in a row trip it.
A gives each dependency its own pool, like a ship's watertight rooms: if recommendations hangs, checkout still has its slots. Size each pool as x seconds per call, plus headroom of about 2 to 3 times. Use it when one optional dependency shares a process with a critical path. The cost is idle capacity: a pool that is not busy cannot lend threads to one that is.
Show data table
| calls | |
|---|---|
| Every layer retries | 27 calls |
| One layer retries | 3 calls |
| One layer plus 10% budget | 1.1 |
Tap any box to see what it is and what it talks to.
- API gateway
- External API
- Cache
Text description
- Closed: calls flow and failures are counted. The 5th consecutive failure trips the breaker to Open.
- Open: every call fails fast and the fallback answers. Once the 10 s cooldown has elapsed, the next call moves it to Half-open.
- Half-open: a single probe is let through. It succeeds, so the breaker closes and the failure count resets.
Surviving big failures
Overloaded, serving everyone means serving no one.
Read more · 991 words
Two ideas: refuse early, and decide the fallback before the outage. Then add copies that cannot fail together.
| Tool | How it works | Use it for | Watch out for |
|---|---|---|---|
| Load shedding | Refuse work you cannot finish, early and cheaply: before parsing or database work, reply 503 with Retry-After (429 for one client over its limit), check under a millisecond. Also drop abandoned work: a request queued past its deadline has no one waiting | Arrival rate above capacity for more than a few seconds, while autoscaling (minutes) cannot help yet | Per-client limits are a different tool: rate limiter |
| Priority shedding | Drop the least important work first: keep checkout and login; drop prefetch, analytics and batch jobs. With one shared queue the login codes wait behind a million marketing messages; with separate lanes they do not | Protecting the work that earns money. Give the critical class its own lane or reserved capacity (say 30% of workers nothing else may use) | The reserved capacity sits idle in quiet times |
| Bounded queue | An unbounded queue turns overload into a delay nobody wants. Wait at the back = backlog / throughput: at 1,000 req/s a cap of 100 is at most 0.1 s; a backlog of 30,000 is 30 s, long after the user left. When full, reject with 503: that is backpressure | Every queue. Set the cap so the wait stays under the request's deadline | |
| Fallback | For each dependency, write down what the user sees when it is down. Use a stale cache whenever slightly old beats nothing | Optional parts hidden or replaced with cached data; only the core flow protected | A little staleness: label it ("prices may be out of date") |
| Kill switch | A feature flag that turns an expensive extra off without a deploy: on-call flips recs.enabled and the page stops calling recommendations within seconds | Every optional feature that calls another service. Flip it first under load, ask questions later | |
| Read-only mode | With the primary database down, a banner keeps browsing, search and order history alive while writes are refused | Beats an error page for most products |
| When this is down | Show instead |
|---|---|
| Recommendations | Cached bestsellers |
| Reviews | Nothing; the page still sells |
| Search | Cached popular queries |
| Database writes | Read-only mode with a banner |
A fallback that never runs quietly breaks. Exercise each one on a schedule: turn the dependency off in staging, then in production for 1% of traffic, and check that the page still renders.
Redundancy. Two of everything helps only if they cannot fail together.
- N+1 across zones. One spare unit beyond what the load needs, so a failure or rolling deploy does not cut capacity. Across 3 zones, run each below about 66% busy so the two survivors carry everything. At 60%, 3 zones carry 180 units; 180 / 2 = 90% each. At 80% they carry 240; 240 / 2 = 120% each. The default: 3 zones, each below 66% busy. Play At 60% busy, then At 80% busy.
- Zones versus regions. Zones are separate buildings in one region, about 1 to 2 ms apart, so a there can be synchronous. Regions are far apart (tens of ms), so is asynchronous and the copy lags. Use a synchronous replica in another zone when you cannot lose committed writes; an asynchronous replica in another region to survive a disaster, accepting the loss of the last seconds. is how much data you can lose. is how long you can be down.
- Active-passive or active-active. Active-passive keeps a standby idle until the fails: simpler and cheaper, but slow to fail over. Active-active serves from several places at once: fast, but must handle writes in two places (Step 6). Start with active-passive. Go active-active only when RTO must be near zero, and then run each region below 50% busy so one can carry both.
- When goes wrong. Failover is detected, not known: a monitor only sees silence. Three missed 5 s apart means 15 s before it acts. If the old primary is only cut off, both accept writes: . The fix is a : every promotion gets a higher number and storage rejects writes with an older one. Rehearse failover on a schedule; the first real one should not be the first test.
Choose by RPO and RTO, two numbers the business can say out loud. A payments ledger needs RPO 0. A blog can live with 24 h.
| Setup | RPO | RTO |
|---|---|---|
| Nightly backups | Up to 24 h | Hours |
| Async replica, other region | Seconds to minutes | Tens of minutes |
| Sync replica, other zone | 0 | 1 to 2 min |
| Active-active regions | Near 0 | Near 0 |
Try the calculator: pick the weakest link in the request path, make it a redundant pair, and watch the total move.
Show data table
| seconds | |
|---|---|
| Cap of 100 | 0.1 s |
| Backlog 1,000 | 1 s |
| Backlog 30,000 | 30 s |
Tap any box to see what it is and what it talks to.
- Load balancer
- Service
Text description
- Three zones at 60% carry 180 units of work (100 is one zone's full capacity).
- Zone A loses power. Health checks remove it within seconds.
- 180 units over 2 zones is 90% each. Everyone is served.
Is losing one committed write unacceptable (money)?
Shipping and watching safely
Most outages start with a change. Keep changes small, visible and reversible.
Read more · 765 words
| Option | How it works | Use it for | Watch out for |
|---|---|---|---|
| Canary | Sends a small slice of traffic to the new version and compares it with the old. At 10,000 req/s, 1% is 100 a second: enough to spot a broken deploy in minutes while 99% of users never see it. Compare error rate and p99 against stable at the same time; step 1%, 10%, 50%, 100%; roll back automatically when the canary burns budget faster than v1 | The default for services | Slower rollout; two versions live at once |
| Blue/green | Runs the new version beside the old and switches at the load balancer, so rollback is instant | When you cannot run two versions on live traffic, or want a full-stack rehearsal before users arrive | Double capacity during the switch, and a shared database: use expand and contract (add the new column, deploy, backfill, switch reads, drop the old column in a later release) |
| Feature flag | Separates deploying code from releasing it: ship dark, enable for 1%, ramp, turn off without a deploy. Hash flag and user id so a user always gets the same answer and raising the percentage only adds users | Risky features and kill switches | Flag debt: delete a flag once the ramp is done |
Play Good , then Bad canary.
At 3 a.m. you need three answers, fast:
| Signal | Answers | Example |
|---|---|---|
| Metrics | Is something wrong? | Error ratio, p99 latency |
| Logs | What happened in this request? | JSON with reqId, status |
| Traces | Where did the time go? | API 320 ms, database 250 ms |
- Metrics are cheap numbers over time. For services watch rate, errors and duration (RED). Average hides the pain: take 100 requests, 98 at 100 ms and 2 at 10 s. The average is (98 x 100 + 2 x 10,000) / 100 = 298 ms, which looks fine. The is 10 s. Alert on p99 or on the share of requests over the threshold, never on the mean.
- Logs explain one request. Write JSON with a request id, one line per request at the edge. Use structured fields, not sentences, and never log passwords, tokens or card numbers. Sample the boring successes if volume is huge, but keep every error.
- Traces follow one request through every service as timed spans. A header (
traceparent) carries one id across services, so ten services join into one waterfall. Sample them (1% to 10%) and keep all traces for errors and slow requests.
Toggle Slow DB and Timeout + fallback and watch which span eats the 200 ms budget.
Alert on symptoms and burn rate, not CPU. On a 99.9% SLO, 1.44% errors is a burn of 14.4, which spends 2% of the month's budget in an hour: page. A slow burn over three days: ticket.
| Burn rate | Window | Budget spent | Action |
|---|---|---|---|
| 14.4x | 1 hour | 2% | Page |
| 6x | 6 hours | 5% | Page |
| 1x | 3 days | 10% | Ticket |
Every alert should name a user-visible symptom and link to a runbook of first safe steps. CPU, memory and disk go on dashboards.
When the pager fires, stop the bleeding first:
Detect
A burn-rate alert pages on-call.
Triage and communicate
Name one incident commander and update the status page.
Mitigate first
Roll back, fail over, shed load or flip a flag. Root cause can wait.
Resolve and learn
Write a blameless postmortem: timeline, causes, action items with owners.
Most incidents are fixed by undoing the last change, not by understanding it. Here the deploy is bad at minute 0, the page fires at minute 4, and the rollback finishes at minute 18. The cause is studied afterwards, with users already fine. Ask first: what changed in the last hour? Then roll back or flip the flag, and look at traces only once the graphs recover.
A postmortem records what happened so it does not happen again, and it blames the system, not a person. Write the timeline, the causes (usually several), what went well, and action items with an owner and a date. An incident without action items will repeat.
Tap any box to see what it is and what it talks to.
- Load balancer
- Service
- External API
Text description
- 1% of 10,000 requests a second is 100 a second to v2. In 5 minutes that is 30,000 requests.
- v2 shows 0.1% errors and the same p99 as v1. A real regression to 2% would show as 600 errors, not 30.
- Healthy, so the canary widens: 1%, then 10%, 50%, 100%, with the same check at each step.
{"ts":"2026-03-02T09:14:07Z","reqId":"r-81f3","route":"POST /order","status":503,"ms":312,"user":"u-42"}Show data table
| ms | |
|---|---|
| p50 | 100 ms |
| Average | 298 ms |
| p99 | 10 s |
Are users seeing errors or slowness against the SLO?
A bad deploy at minute 0 is detected by a burn-rate alert at minute 4, mitigated by rollback at minute 18, and only then investigated. The postmortem comes later.
Text description
- Detect: Burn-rate alert fires (0–4 min)
- Triage: Commander named, status page (4–10 min)
- Mitigate: Roll back the deploy (10–18 min)
- Learn: Confirm recovery, hand over (18–30 min)
Code
The breaker lets exactly one half-open probe through.
Read more · 58 words
availability.js and slo.js reproduce this page's numbers.
// Full-jitter exponential backoff, plus a "retryable" predicate
export async function retry(fn, { tries = 4, baseMs = 100, retryable = (e) => e.retryable ?? true } = {}) {
for (let i = 0; ; i++) {
try {
return await fn();
} catch (e) {
if (i >= tries - 1 || !retryable(e)) throw e; // never retry validation errors or non-idempotent calls
await new Promise((r) => setTimeout(r, Math.random() * baseMs * 2 ** i)); // full jitter: wait 0–100, 0–200, 0–400 ms
}
}
}export const serial = (...a) => a.reduce((x, y) => x * y, 1); // every dependency must be up
export const parallel = (...a) => 1 - a.reduce((x, y) => x * (1 - y), 1); // up if any copy is up (independent failures)
export const downtimeMinutes = (a, days) => (1 - a) * days * 24 * 60;
export const downtimeMinutesPerYear = (a) => downtimeMinutes(a, 365.25); // average year, leap days included
// serial(0.999, 0.999) → 0.998 (two 99.9% dependencies give ≈ 99.8%)
// serial(0.999, 0.999, 0.9995) → 0.9975 (~22 h down per year)
// parallel(0.99, 0.99) → 0.9999
// downtimeMinutes(0.999, 30) → 43.2 minutes per 30 days
// downtimeMinutesPerYear(0.999) → 526 minutes ≈ 8.77 hReliability in the interview
Bring reliability up as soon as the design has more than one service, or when the interviewer asks what happens when something fails.
Read more · 617 words
Then walk through five decisions in order.
Say the target as an SLO
"99.9% of checkouts succeed in under 300 ms over 30 days." That is 43 minutes of a month and it tells everyone how much to spend on resilience.
Find what fails together
Walk the diagram and name the single points of failure: one database, one zone, one deploy. Give each a copy in a separate failure domain.
Contain a bad dependency
A timeout on every call, one retrying layer with and , a breaker with a fallback, and a on optional calls.
Survive the big failures
3 zones each below 66% busy, a synchronous with automatic , and by priority for overload.
Ship and watch safely
at 1% with automatic rollback, , and burn-rate alerts on what users feel. Name and if data matters.
| If the interviewer says | You raise |
|---|---|
| "What if the payment service is slow?" | Timeout, one idempotent retry, breaker, queue the order |
| "What if a data center dies?" | 3 zones below 66% busy, sync replica, automatic failover |
| "How do you deploy safely?" | Canary at 1%, automatic rollback, feature flags |
| "How do you know it is healthy?" | RED metrics, p99, burn-rate alerts, traces |
| "We need five nines" | About 26 s a month: only automatic failover works; say the cost |
Quick checkWhat do you ask for before choosing resilience patterns?
Interview questions
Why add jitter to retries?
Clients would retry in lockstep and knock a recovering service back down.
SLO vs SLA?
The is your internal target, and its sets how fast you ship. The SLA is the looser contract with penalties.
How do you survive a zone failure?
3 zones with spare capacity, and synchronous database with automatic .
What do you alert on?
Burn rate on what users feel (errors, against the SLO). CPU and disk go on dashboards.
How can retries make an outage worse?
Retries at every layer multiply load (3 × 3 = 27 calls). Use jittered , breakers and one retrying layer.
How do you stop a slow dependency taking everything down?
A short timeout, a breaker with a fallback, and a so its pool is capped. Threads busy = rate x latency, so a hang drains a shared pool in under a second.
What is split brain and how do you prevent it?
Two nodes both think they are and accept writes. A makes storage reject the older one.
Check yourself
Question 1 of 5Score 2 out of 3 or better to mark this page complete.