Client, API and Load Balancing
One server is a limit and a single point of failure. Stateless servers plus a balancer let you add capacity by adding machines.
- 3 down, 2 uphealth-check failures and successes to flip state
- 2 random pickspower of two choices samples, takes the less loaded
- 30 spassive ejection after 5 consecutive 5xx
Show what to say
“I make the services stateless, with sessions in Redis and files in object storage, and put a load balancer in front, so adding capacity means adding a machine and a crash loses nothing. For uneven requests I use least connections or power of two choices, with health checks that need three failures to mark an instance down so a blip can't flap it.”
Trade-offs
- Stateless vs sticky sessions: sticky pins warm caches and WebSockets, but deploys, crashes and autoscaling break the pin, so state goes in Redis.
- L4 vs L7: L4 forwards fast at very low cost, while L7 reads URLs and headers for routing and retries at more CPU per request.
- Round robin vs least connections: round robin is simple but keeps feeding a slow server; least connections needs an accurate count per server.
Bring it up before they ask
Failing readiness on an optional dependency like recommendations takes the whole fleet out at once, so only fail it for what the instance truly needs.
New to the words? Read the one-minute primer
Make services and put a in front: capacity becomes "add a machine", and a crash loses nothing. Then give the API correct status codes, and .
Why it matters
One server is a ceiling and a .
Read more · 200 words
servers behind a balancer make crashes and spikes routine: the balancer routes around a dead instance. The API on top is every client's contract, so status codes and safe retries are design, not polish.
Do you need a balancer yet? At 200 req/s one server has room, but a second behind a balancer means deploys and crashes cost nothing. At 5,000 req/s peak you need it for capacity too.
Each hop before your code adds one thing:
| Hop | What it adds |
|---|---|
| DNS and CDN | The nearest region; static files from the edge |
| Load balancer | Spreads requests over healthy instances |
| API gateway | Auth, rate limits, routing to the right service |
| Services and data | Stateless logic; state lives in databases, caches and queues |
Play Happy path to follow one request, then Sticky-session pitfall, Fix: shared session store and Instance failure for the stories the next sections explain.
This page follows the order you would say it in an interview: where state lives, how requests are spread, what goes wrong when one server misbehaves, and how the API survives retries.
Tap any box to see what it is and what it talks to.
- Client
- API gateway
- Load balancer
- Service
- Cache
- Database
Text description
- Requests arrive at the API gateway, the single front door.
- The gateway checks the token and rate limit, then hands each request to the load balancer.
- The load balancer spreads requests over the healthy instances: one to A, the next to B, the next to C.
- The instances are identical and stateless: each reads what it needs from the shared database.
- Responses flow back. Need more capacity? Add another instance behind the load balancer.
- The client never knows which instance served it.
Stateless services
doesn't mean no state.
Read more · 283 words
It means no instance owns any, so any instance can serve any request, and any can die.
| State | Where it goes instead |
|---|---|
| Sessions, login | Redis with a TTL, or a signed token |
| Uploaded files | Object storage (S3) behind the CDN |
| Counters, locks | Redis |
| Business data | The database |
Long jobs go to a queue. The hardest piece to move is the login session, and there are three places for it:
| Option | How it works | Use it for | Watch out for |
|---|---|---|---|
| Sticky sessions | The balancer sends a user to the same instance every time, so it keeps the session in its own memory | A stopgap, or an instance that really holds a connection (a WebSocket) | Every deploy, crash and scale-in logs out everyone pinned to that instance. A hot user group cannot be rebalanced, because the pin overrides the algorithm |
| Shared session store (the default) | The session lives in Redis, keyed by a random id in a cookie. Any instance serves any user | Logged-in web apps. Lookup is about 1 ms; 1 million active sessions at about 1 KB each is about 1 GB of Redis | One more dependency on the hot path: if Redis is down, nobody can log in, so run it replicated |
| Signed token | The session sits inside the token the client carries; the server verifies the signature with a shared key, in microseconds, with no network call | APIs called by apps and other services, where a lookup per request is unwelcome | Revocation: a token stays valid until it expires. Keep lifetimes short (15 minutes) with a refresh token, or keep a small deny list in Redis for forced logouts |
Play Pinned, then B restarts to see why sticky is the weakest.
Tap any box to see what it is and what it talks to.
- Client
- Load balancer
- Service
Text description
- Alice logs in. The balancer picks Service B and sets an affinity cookie.
- B stores her session in its own memory.
- Every later request carries the cookie…
- …so it lands on B again. Alice sees nothing wrong. Service A stays idle for her.
L4 vs L7 and balancing algorithms
L4 sees connections; L7 reads requests. Most stacks use both: L4 spreads connections over L7 proxies.
Read more · 738 words
Pick by what you need to see.
| L4 (TCP/UDP) | L7 (HTTP) | |
|---|---|---|
| Sees | IP and port | URL, headers, cookies |
| Can do | Fast forwarding, any protocol; picks a server once per TCP connection | Path routing, TLS termination, retries, canaries; decides per request |
| Cost | Very low | More CPU per request (it parses every request and holds the TLS keys) |
| Use it for | Raw TCP or UDP, databases, game traffic, the outer tier in front of a proxy fleet | The default for HTTP services |
| Examples | AWS NLB, IPVS, Maglev | AWS ALB, NGINX, Envoy, HAProxy (HTTP mode) |
- The long-connection trap: because L4 chooses per connection, one or connection that carries 1,000 requests a second sends all 1,000 to one server.
- : with L7, a canary at 5% means 1 request in 20 reaches the new version, so a bad release hurts 5% of users for ten minutes instead of all of them.
- Both together: when you outgrow one proxy, a pair of L4 balancers gives clients one stable IP and spreads connections over a fleet of L7 proxies. Clients never notice when you add or remove a proxy. On a managed cloud, a single L7 balancer (an ALB) is enough until traffic or protocol needs say otherwise. Start with one box and name the reason before adding a second tier.
Which healthy instance gets the next request? Round robin is simple, until one server is slow: it stays on the rotation, its queue grows, timeouts trigger retries, and the retries bury it.
| Algorithm | How it works | Use it for | Watch out for |
|---|---|---|---|
| Round robin (default) | Hands out requests in turn: A, B, C, A, B, C. No state, perfectly even when servers and requests are alike | Stateless services with uniform calls (5 ms lookups on identical machines) | Ignores load: a slow server keeps a third of the traffic |
| Weighted round robin | Each server gets a share by weight: a 16-core machine with weight 5 beside two 4-core machines with weight 1. Weights 5, 1, 1 give a a b a c a a, not five a in a row, because the smooth variant spreads the turns | A migration between instance types, or a few much bigger machines | Still ignores live load; it only fixes a size mismatch |
| Least connections | Sends each request to the server with the fewest requests in flight, so slow servers and long requests balance themselves | Mixed costs: a 5-second CSV export beside 5 ms lookups | Needs an accurate count, which one balancer has but a fleet of balancers does not share. A new instance starts at zero and gets flooded (see failure modes) |
| Power of two choices (P2C) | Picks two servers at random and takes the less loaded. Almost as good as least connections, with no shared state and no scan of the fleet. Random assignment alone leaves the busiest server far above average; comparing two picks removes most of the gap | Many proxies or balancers that each see part of the traffic (Envoy, a service mesh) | Slightly coarser balance than a global count |
| Hashing | Picks the server from a key such as user id or client IP, so the same key reaches the same server and per-key state (local caches, shards, WebSocket sessions) stays warm | Per-key affinity, like cache servers | A hot key: one celebrity user overloads their server while the others idle |
Use , not hash mod N. Going from 4 nodes to 3 with plain mod moves 75% of keys, because only keys where k mod 4 equals k mod 3 stay put. Consistent hashing moves only the 25% that lived on the removed node. See the sharding strategies deep dive.
Not sure which fits? Walk the tree. Then, in the simulator, make server C slow under round robin and watch its queue grow, then switch to P2C.
One box is missing from the request path. Put it back.
Tap any box to see what it is and what it talks to.
- Client
- Load balancer
- API gateway
- Service
Text description
- Clients know one address. The L4 balancer is cheap, so one pair handles huge connection counts.
- It spreads connections over the L7 proxies.
- Each proxy reads requests and routes them to the right service, with retries.
Same-size servers, similar requests?
// Round robin that skips unhealthy servers (a health checker flips `up`)
export const servers = [{ url: 'a:3000', up: true }, { url: 'b:3000', up: true }, { url: 'c:3000', up: false }];
let i = 0;
export function pick() {
for (let n = 0; n < servers.length; n++) {
const s = servers[i++ % servers.length];
if (s.up) return s.url; // skip unhealthy nodes
}
throw new Error('no healthy servers');
}
// pick() × 4 → 'a:3000', 'b:3000', 'a:3000', 'b:3000' (c is down)Health checks and failure modes
The balancer can only route around failures it can see. Four mechanisms cover it, and each answers a different question.
Read more · 937 words
| Mechanism | How it works | Watch out for |
|---|---|---|
| Active checks (the baseline) | The balancer probes each instance on a schedule, usually /readyz every few seconds. 3 failures mark it down, 2 successes mark it up, so a blip cannot flap it | Detection time: with a 5 s interval and 3 failures, a dead instance takes about 15 s to be noticed. At 900 req/s over three instances that is 300 req/s sent to a corpse for 15 s, about 4,500 failed requests, unless the proxy retries them |
| Passive checks | Watch real traffic: an instance that returns 5 consecutive 5xx is ejected for 30 seconds. Catches an instance that passes /readyz but fails real requests | Use next to active checks, never instead. They only learn from real users failing, so pair them with a retry on another instance for idempotent requests |
| Liveness vs readiness | Liveness asks "is it alive?" (fail: restart the process). Readiness asks "should it get traffic?" (fail: stop routing, leave it running) | Keep readiness narrow: check only what this instance cannot work without, such as its own database connection. If /readyz also calls an optional recommendations service, then when it goes down every instance fails at once and the whole site returns 503 over a carousel. Stay ready, drop the feature, and put a short timeout and a circuit breaker on the call |
| Connection draining | On SIGTERM the instance fails readiness, the balancer stops sending new requests, and in-flight ones complete | Without it, every deploy cuts requests in half. Set the grace period above your slowest normal request, for example 30 s. Cloud balancers call this deregistration delay; AWS target groups default to 300 s |
Most outages are not the balancer dying, they are a small fault that the balancer spreads. Play The spiral, then The fix.
The slow-server spiral. 13:00, a sale starts at 900 req/s, and server C exhausts its database pool. Its calls go from 20 ms to 2 s. Round robin still sends C 300 req/s, and by (in flight = rate times ) that is about 600 requests in flight on C instead of about 6. Clients time out at 1 s and retry, which pushes more load onto A and B. Fix it in three layers: least connections or P2C so a slow server gets less, outlier ejection so a failing one gets none, and timeouts plus a retry budget so clients do not multiply the load.
| Failure | What happens | Fix |
|---|---|---|
| Retry storm | Retries add load to something already failing. Three retries per failed call turn 1,000 req/s into 4,000 req/s on a server that could not handle 1,000 | Exponential backoff with jitter; a retry budget (retries add at most about 10% extra requests); a circuit breaker. Retry only idempotent requests, on a different instance than the one that just failed |
| A new instance gets flooded | The autoscaler adds one during a spike, it reports ready with 0 open requests, and least connections sends it nearly everything while its caches, connection pools and runtime are cold | Slow start: ramp the instance from 10% to full weight over 30 to 60 s, and report ready only after a warm-up call. Round robin and P2C hit this far less |
| The balancer is a single point of failure | One balancer is one more machine that can fail, and everything sits behind it | Run at least two, in different zones. DNS with health checks can drop a dead address, but clients that cached it keep trying until the TTL (60 s here) runs out. Managed balancers (ALB, NLB) already spread across zones. On your own hardware, use a floating IP shared by a pair, or anycast |
Compare no retries, fixed retries and with full in the simulator.
Sizing the fleet. Size for the peak, not the average, and run well below full. A server at 90% busy has almost no room for a spike and answers slowly even before it fails. In the simplest queueing model (random arrivals, one queue), average response time is the service time divided by one minus utilisation, so running hot is steep.
Worked example: peak 5,000 req/s, and one instance does 500 req/s flat out.
| Step | Arithmetic | Result |
|---|---|---|
| Plan for 60% busy | 500 x 0.6 = 300 req/s each | 5,000 / 300 = 17 instances |
| Lose one of 3 zones | 2 zones must carry the peak | 7 per zone, 21 total |
| Check the survivors | 5,000 / 14 = 357 req/s | 71% busy, still fine |
| In flight per instance | 300 req/s x 20 ms | 6 requests |
Keep headroom instead of trusting for the first spike: a new instance needs a minute or two to boot and warm up. Use autoscaling for the slow growth, and headroom for the sudden jump.
// Liveness vs readiness, plus draining on deploy
let shuttingDown = false;
app.get('/livez', (_req, res) => res.sendStatus(200)); // the process is alive
app.get('/readyz', async (_req, res) => { // should the LB send traffic here?
if (shuttingDown) return res.sendStatus(503);
try { await pool.query('SELECT 1'); res.sendStatus(200); }
catch { res.sendStatus(503); }
});
process.on('SIGTERM', () => {
shuttingDown = true; // readiness fails, so the LB stops routing here
setTimeout(() => server.close(() => process.exit(0)), 10_000); // finish in-flight requests, then exit
});Tap any box to see what it is and what it talks to.
- Client
- Load balancer
- Service
Text description
- 13:00, a sale starts at 900 requests a second. C's database pool is exhausted: its calls go from 20 ms to 2 s.
- C still gets 300 requests a second. At 2 s each, about 600 are in flight on C against about 6 normally.
- Clients time out at 1 s and retry, so each failed call adds a new one. Load climbs well past 900.
- The retries land on A and B too, now running hotter than they can sustain.
Show data table
| Utilisation | Response time (x service time) |
|---|---|
| 30% | 1.43 |
| 50% | 2 |
| 60% | 2.5 |
| 70% | 3.33 |
| 80% | 5 |
| 90% | 10 |
| 95% | 20 |
Quick checkRound robin keeps feeding one slow server. What do clients do?
REST API design and safe retries
Be predictable: plural nouns for resources, HTTP for verbs, one level of nesting.
Read more · 804 words
Clients guess your API correctly when it follows the rules they already know.
Resources and verbs. Model nouns as resources and use the HTTP method as the verb. GET /orders lists, GET /orders/123 reads one, POST creates, PUT replaces, PATCH changes some fields, and DELETE deletes. Other actions are sub-resources: POST /orders/123/cancel. Do not put verbs in paths like /getOrders. Keep nesting to one level (/users/7/orders); deeper trees force clients to know every parent id.
Status codes tell clients and proxies what happened, so pick them by asking the questions in order. Walk the tree, then use the table as a reference.
| Code | Use it for |
|---|---|
| 200 OK | Success with a body: reads, updates returning the resource |
| 201 Created | Created by a POST; add a Location header |
| 202 Accepted | Queued, not done: return a job id or status URL |
| 204 No Content | Success, no body: DELETE, or PUT with nothing to return |
| 301 / 302 | Moved permanently / temporarily; browsers cache 301 |
| 304 Not Modified | Client's copy still valid (conditional GET, If-None-Match) |
| 400 Bad Request | Malformed: invalid JSON, a missing field |
| 401 Unauthorized | Not authenticated: missing or expired token |
| 403 Forbidden | Authenticated, not allowed: Alice reading Bob's orders |
| 404 Not Found | No such resource (or hiding that one exists) |
| 409 Conflict | Clashes with current state: duplicate, version mismatch, key in progress |
| 422 Unprocessable Content | Well-formed but invalid: end date before start date |
| 429 Too Many Requests | Rate limited; always add Retry-After |
| 500 Internal Server Error | Bug or unexpected failure, never a client mistake |
| 503 Service Unavailable | Overloaded, shedding load or failing readiness; add Retry-After |
The worst mistake is returning 200 with an error in the body: clients, proxies and monitoring all count it as a success.
Long jobs: accept, then poll. A job over about 1 to 2 seconds (exports, video processing, bulk imports) should not hold a connection open. Return 202 Accepted with a status URL, run the work on a queue, and let the client poll. The cost is a second endpoint and a job record.
Versioning and errors. Put the major version in the path (/v1/orders). Adding an optional field needs no new version; removing or renaming one does. Scene: you rename total to totalCents on Friday. Mobile app 4.2, which users update slowly, reads undefined and shows a blank price until you roll back. Keep changes additive, and run /v1 and /v2 side by side until old clients fade.
Errors should be machine-readable. RFC 9457 "problem details" is the standard shape:
Pagination and safe retries. Page 50 of a feed, and the same POST sent twice: every busy API meets both. Offsets suit admin tables; feeds need cursors; any POST that moves money needs an .
Offset: ?page=3&limit=20 | Cursor: ?cursor=…&limit=20 | |
|---|---|---|
| Says | "Skip N rows, then give me 20" | "Give me 20 after the last one I saw"; the cursor encodes that row |
| Query | LIMIT 20 OFFSET 40 | WHERE id < $1 ORDER BY id DESC LIMIT 21 |
| Deep pages | Slow: reads and discards every skipped row | Fast: seeks straight into the index, so every page costs the same |
| New items arrive | Rows shift: duplicates and skips | Stable |
| Jump to page 50 | Yes | No, only next (and previous) |
| Good for | Admin tables, numbered search results with a few hundred pages at most | Feeds, timelines, infinite scroll, sync APIs; the default for anything that grows |
OFFSET 999980 reads about a million rows to return 20. The chart shows rows read as the page number grows; a cursor stays flat at 21.
The cursor's sort key must be unique and indexed, so sort on (created_at, id) when timestamps can tie.
Cursor on (created_at, id)? Send the timestamp as text (created_at::text), or rows get skipped: Postgres keeps microseconds, JavaScript only milliseconds.
Networks fail after the server did the work, so clients retry. A retried POST /payments must not charge twice. The client sends one UUID as an Idempotency-Key; the server claims it with SET NX, stores the response and replays it for repeats. Scene: Black Friday, 20:02. A shopper's phone loses signal just after sending a 49.00 payment. The charge succeeded, the response was lost, the app retries. Without a key the card is charged twice; with one, the retry replays the first response. Play Retry after a timeout.
A duplicate that arrives while the first is still running gets 409. Keep keys for about 24 hours (86,400 s in the snippet), and scope them to the user so one customer cannot replay another's.
GET, PUT and DELETE are safe to retry; PATCH isn't necessarily. Never auto-retry a POST without a key.
Did the request succeed?
{ "type": "https://example.com/problems/out-of-stock", "title": "Out of stock",
"status": 409, "detail": "SKU a1 has 0 units left", "instance": "/orders/123" }Show data table
| rows | |
|---|---|
| Offset, page 1 | 20 rows |
| Offset, page 50 | 1,000 rows |
| Offset, page 5,000 | 100,000 |
| Offset, page 50,000 | 1,000,000 |
| Cursor, any page | 21 rows |
// Stable pages even while new rows arrive
const encode = v => Buffer.from(JSON.stringify(v)).toString('base64url');
const decode = s => JSON.parse(Buffer.from(s, 'base64url').toString());
app.get('/api/posts', async (req, res) => {
const limit = Math.min(Number(req.query.limit) || 20, 100);
const cursor = req.query.cursor ? decode(req.query.cursor) : null; // { id } of the last item seen
const { rows } = await pool.query(
`SELECT id, title, created_at FROM posts
WHERE ($1::bigint IS NULL OR id < $1)
ORDER BY id DESC LIMIT $2`, // ids are time-ordered (bigserial / Snowflake)
[cursor?.id ?? null, limit + 1]); // one extra row tells us if there's a next page
const page = rows.slice(0, limit);
res.json({ data: page, nextCursor: rows.length > limit ? encode({ id: page.at(-1).id }) : null });
});- Client
- Service
- Cache
- Database
Text description
- Client → API: POST /payments + Idempotency-Key k1
- API → Redis: SET idem:k1 pending NX
- Redis replies to API: OK, claimed
- API → Payments DB: charge 49.00
- API → Redis: store 201 + body
- API replies to Client: 201 Created
// Express middleware. Repeats of a POST with the same Idempotency-Key replay the stored response.
export function idempotency(redis, ttlSec = 86_400) {
return async (req, res, next) => {
const key = req.get('Idempotency-Key');
if (!key) return next();
const redisKey = `idem:${req.user?.id ?? 'anon'}:${key}`;
const claimed = await redis.set(redisKey, JSON.stringify({ state: 'pending' }), 'EX', ttlSec, 'NX');
if (!claimed) {
const saved = JSON.parse((await redis.get(redisKey)) ?? '{}');
if (saved.state === 'done') return res.status(saved.status).json(saved.body);
return res.status(409).json({ error: 'A request with this key is still in progress' });
}
const json = res.json.bind(res);
res.json = body => {
const write = res.statusCode < 500
? redis.set(redisKey, JSON.stringify({ state: 'done', status: res.statusCode, body }), 'EX', ttlSec)
: redis.del(redisKey); // let the client retry server errors
write.catch(() => {});
return json(body);
};
next();
};
}
// If a handler crashes without responding, the key stays "pending" until the TTL expires: keep TTLs modest.API styles, gateway and BFF
Pick the style by who is calling.
Read more · 580 words
REST is the default (every proxy, , browser and tool understands it, and responses by URL); each other style earns its place by solving one specific problem.
| Style | Best fit | Strengths | Costs |
|---|---|---|---|
| REST | Public APIs, CRUD on resources | Simple, HTTP caching, tooling everywhere | Over- or under-fetching; several round trips per screen |
| GraphQL | Many clients (web, iOS, Android, partners) wanting different shapes of the same data | Client picks fields in one request against a typed schema: about 80 ms of network instead of 240 | N+1 resolvers (batch with a loader); HTTP caching by URL is gone because everything is a POST; needs query depth and cost limits, or one query can ask for the whole database |
| gRPC | Internal service-to-service calls with tight latency or streaming | Compact binary Protobuf from a shared .proto over long-lived HTTP/2; generated clients, so a wrong field is a compile error | Browsers need a gRPC-Web proxy; binary is harder to debug; long connections need L7 or client-side balancing |
| WebSocket | Chat, multiplayer, collaborative editing; only when both directions are frequent | Full duplex, low latency | Stateful servers: Alice and Bob on different servers need pub/sub between them. At about 100,000 sockets per machine (an assumption; measure yours), 1 million users need 10 machines plus headroom. A deploy drops every socket, so clients reconnect with jitter |
| SSE | Notifications, live scores, progress, streamed LLM tokens | Plain HTTP, so far cheaper than WebSockets; the browser reconnects and resumes from the last event id | Server to client only; text only |
Answer these questions in order to choose. The polling, SSE and WebSockets deep dive compares the real-time options.
An is the single front door: it checks the token, enforces and routes each path to a service, so services stick to business logic and failing requests never reach them. Use it as soon as you have several services, or any public API. Play Rejected early: a caller sending 300 req/s against a limit of 100 is stopped at the door with a 429. The cost is an extra hop and a possible : run several gateways behind a balancer, and keep business logic out of it.
A backend for frontend (BFF) is a thin layer per client type that calls the services and shapes the response for its own screens. Use it when one generic API forces a client to make many calls or download fields it never shows, for example a phone on a slow network. Each client team owns its BFF. The cost is more services to run, and logic can drift between BFFs. is the alternative when you would rather let clients pick their own fields.
Does the server push updates?
Tap any box to see what it is and what it talks to.
- Client
- API gateway
- Service
- Search index
Text description
- GET /orders/123 with a bearer token.
- The gateway verifies the token, checks the caller's rate limit, and routes /orders to the orders service.
- The service returns data and never wrote any auth code.
API and load balancing in the interview
Bring this up the moment you draw more than one server, then say five decisions in order.
Read more · 543 words
The order shows you have done it before.
Make the services stateless
Sessions in or a signed token, files in , work on queues. Say it first: it is what makes the next steps possible.
Put a balancer in front
One managed L7 balancer by default, with an L4 tier in front only for raw or very large scale. Round robin to start.
Pick the algorithm by the workload
Round robin for uniform calls, least connections or P2C for mixed or slow ones, only when a key needs the same server.
Say how it fails
Active plus passive health checks, narrow readiness, draining on deploy, a retry budget, and enough headroom to lose a zone.
Define the API contract
REST with correct status codes, for feeds, and an on every POST that moves money.
| If the interviewer says | You raise |
|---|---|
| "Traffic is 10x on sale day" | Stateless instances, headroom, autoscaling with slow start |
| "One server gets slow" | Least connections or P2C, outlier ejection, a retry budget |
| "Users get logged out at random" | Sticky or in-memory sessions; move to Redis or a signed token |
| "A client retries the payment" | Idempotency key, 409 for in-flight duplicates |
| "Infinite scroll feed" | Cursor pagination, never offset |
| "Web and mobile need different data" | A BFF, or GraphQL if clients vary a lot |
Quick checkWhat do you say first when scaling a service to many servers?
Interview questions
Why must services be stateless to scale horizontally?
So the balancer can send any request to any instance. A crash loses only in-flight requests.
L4 or L7?
L4 for fast forwarding; L7 for path routing and retries, usually behind an L4 balancer.
How does the LB know an instance is unhealthy?
Active /readyz checks (3 failures down, 2 up) plus passive ejection on error spikes.
One server gets slow under round robin. What happens?
Its queue grows, clients time out and retry, and the retries load the rest. Use least connections or P2C, eject outliers, and cap retries with a budget.
REST, GraphQL or gRPC?
REST for public resources; when clients need different shapes; for internal calls.
How do you make POST retries safe?
An Idempotency-Key the server claims with SET NX, replaying the stored response for repeats.
Check yourself
Question 1 of 5Score 2 out of 3 or better to mark this page complete.