Async Processing and Messaging
Slow or unreliable work (email, video, payments) should not hold the user's request. Queues absorb spikes and let work retry safely.
- 720Kbacklog after an hour at 1,000 in, 800 out
- 1, 2, 4, 8 sexponential backoff for transient failures
- 200 slotsworkers for 1,000 jobs/s at 200 ms each
Show what to say
“I move slow, unreliable or spiky work off the request path onto a queue: the API records the intent and answers fast, and a worker retries until it succeeds. Delivery is at-least-once, so consumers are idempotent with a unique constraint on the message id, transient failures retry with backoff, and poison messages go to a dead letter queue.”
Trade-offs
- Sync vs async: async keeps the request fast and absorbs spikes, but work happens later, maybe twice, maybe out of order.
- At-most-once vs at-least-once: acking first can lose a message in a crash; acking after processing can redeliver it, so dedupe.
- Outbox vs write then publish: the outbox adds a relay and may publish a row twice, but a crash never loses the event.
Bring it up before they ask
A DLQ nobody alerts on is a silent loss, so alert on depth above zero and on the age of the oldest message.
New to the words? Read the one-minute primer
Move slow, flaky or spiky work onto a queue. Delivery is , so make , retry with , and use a instead of "write the database, then publish".
Why it matters
Send the welcome email inline and signup is as slow as the email provider (say 800 ms).
Read more · 284 words
Queue it instead: the API answers in tens of milliseconds and a worker sends the email, retrying if it fails. If the provider is down for an hour, inline signups fail; queued ones just wait.
The price: work happens later, maybe twice, maybe out of order. Skip the queue for fast work the user needs in the response: it adds a to run. Play Burst absorbed.
If the user needs the answer to carry on ("was my card accepted?"), keep it sync. Go async when work is slow, unreliable, spiky or fans out to many reactions. Walk the tree with your own example.
| Request | Sync part (user waits) | Async part (worker) |
|---|---|---|
| Sign up | Create the user row | Welcome email, CRM sync |
| Upload a video | Store the file, return an id | Transcode renditions, thumbnails |
| Place an order | Reserve stock, authorise the card | Confirmation email, invoice PDF, analytics |
| Publish a post | Save the post | Fan out to followers' feeds |
Most flows split in two: the API saves what it must, publishes a job and returns 202 Accepted with a job id. Use it when the work takes more than about a second, or depends on something you do not control (an email provider, a card network, a transcoder). The client either polls GET /jobs/8841 every couple of seconds or gets a push when the job ends. The cost is a status endpoint to build, and a failure that arrives after the user has left, so you also need a way to tell them (an email, a notification, a status badge).
Tap any box to see what it is and what it talks to.
- Service
- Queue
- Worker
- Database
Text description
- A user signs up. The API publishes a send-welcome-email job and answers 202 Accepted straight away.
- Worker 1 pulls the message. The broker hides it from other workers while it is in flight.
- Worker 1 does the slow part (calls the email provider) and records the result.
- Worker 1 acks the message only after the work succeeded, so the broker deletes it.
- More signups arrive while that was happening.
- The two workers take messages in parallel. Adding workers adds throughput.
- Both jobs finish and are acked. The user never waited for any of this.
Does the user need the result to carry on?
Queues, pub/sub and Kafka
With a queue in between, the can be slow, mid-deploy or briefly down and nothing is lost.
Read more · 580 words
That is decoupling: each side scales and fails on its own. "Queue" covers three tools, and the difference is who gets each message. First, put the missing box back.
Which box lets signup succeed while the email provider is down?
| Tool | How it works | Use it for | Watch out for |
|---|---|---|---|
| Work queue (default) | Each message goes to exactly one consumer, so more workers means more throughput. The broker hides a taken message for a visibility timeout (30 s by default in SQS) and deletes it only on ack; no ack means it reappears | Jobs run once: emails, thumbnails, webhooks, invoice PDFs. SQS on AWS with no servers; RabbitMQ for routing rules or priorities | Once acked a message is gone, so no replay. A standard SQS queue only promises best-effort order |
| Pub/sub | Every message goes to every subscriber. The publisher does not know who listens, so adding a search indexer touches no existing code | One event, several services care: order placed, then email, analytics and search | Plain pub/sub is fire-and-forget: a subscriber that is down misses the event. Fix: a queue behind each subscriber (SNS topic to one SQS queue per consumer) |
| Log (Kafka) | Appends messages and keeps them. Reading does not delete; each consumer group tracks its own position, so any group can rewind | Event streams, analytics, change feeds, a second consumer next month that needs history | You run a cluster (or pay for a managed one); order holds only inside a partition; consumers manage their own offsets |
Decide on one question first: does anyone besides the consumer need these messages, or need to replay them? If yes, a log. If no, a work queue. The default interview answer: "a queue for jobs, for event streams".
Kafka is a log, not a mailbox. A topic splits into partitions. A key always lands in the same partition, and order holds only within it. In a consumer group each partition has one consumer; other groups read the same topic independently.
- Partitions are the unit of order and parallelism. The producer hashes the key to pick one. With 6 partitions and 8 consumers, 2 sit idle. Add consumers past the partition count in the simulator.
- Choose the partition count for the consumers you expect at peak, with headroom. You can add partitions later but never remove them, and adding them changes which partition a key maps to, which breaks per-key order for in-flight data.
- Consumer groups: use one group per application (billing is a group, analytics another, and each sees every message). Scale a group up to the partition count, not past it. When a consumer joins, leaves or crashes, Kafka reassigns partitions (a rebalance) and reads pause briefly.
- Replay and retention: the log keeps messages for a set time (7 days by default), read or not. Consumers commit their position (an offset), so a group can rewind after a bug fix and rebuild its state.
- Retention is a disk bill. A topic taking 5 MB/s writes 432 GB a day (5 MB x 86,400 s). Seven days is about 3 TB, and with 3 about 9 TB. Shorten retention, compact by key, or compress before you buy disks.
Do several services need the same events, or need to replay them?
Tap any box to see what it is and what it talks to.
- Service
- Queue
- Worker
Text description
- The producer appends an event. Reading never removes it.
- The billing group reads it and commits its position (offset).
- The analytics group reads the same event at its own pace. A work queue would have given it to one of them only.
Delivery and ordering
Your will see the same message twice.
Read more · 801 words
redeliver after crashes: that is , the default. "Exactly once" really means at-least-once plus work: doing it twice has the effect of doing it once. When you decides what a crash costs.
| Guarantee | How | Use it for | Watch out for |
|---|---|---|---|
| At-most-once | Ack first, then process. A crash loses the message | Data where one lost item costs nothing: a metrics ping, a debug log line, a position update replaced in a second | Never for emails, orders or money: a loss is silent |
| At-least-once (default in SQS, RabbitMQ, Kafka) | Process, then ack. A crash before the ack means redelivery: the work can run twice, never zero times | Almost everything | Duplicates. Scene: at 9:03 a worker charges a card, its pod is evicted before the ack, and a second worker charges again after the visibility timeout. You cannot prevent redelivery, only make it harmless |
| Effectively once | At-least-once plus an idempotent consumer | Anything with side effects | Needs a dedupe table or idempotent writes |
Two ways to make work idempotent. A table: record the message id under a unique constraint, in the same as the effect. A redelivery inserts zero rows and is skipped.
Idempotent writes: SET status = 'shipped' is safe twice; balance = balance + 10 isn't. For a call to another system, send an (payment providers accept one) so the same key never charges twice. The dedupe table needs a clean-up job: keep ids for longer than the broker can redeliver, for example 7 days.
"Exactly once" is real only in narrow cases. 's works for read, process, write that stays inside Kafka. SQS FIFO drops a duplicate send that has the same dedupe id within 5 minutes. Neither stops your worker sending an email twice, so you still need an idempotent consumer.
Ordering. "Order shipped" before "order paid" is a bug, and Kafka orders only within a partition. So ask which messages must be ordered against each other: almost never "all of them".
- Key by the entity that needs order (
userId,orderId): one key, one partition, one consumer, so events apply in send order, while other users run in parallel. Use it for anything with a lifecycle (created, paid, shipped). The cost is a if one key is huge: a celebrity user with 40% of the traffic pins one consumer while the others idle. Split the key (user:7#0to#3) only if order inside that user does not matter. - Retries reorder. A failed message parked on a retry topic for 30 s lets the next one run first: the order ships before it is paid. Where order matters, retry in place and let the messages behind it wait. The price: one stuck message blocks its partition, so cap the attempts (say 5), then move it to the and carry on. Enable the idempotent producer (
idempotent: truein the snippet) so a producer retry cannot duplicate or reorder a send. - Sequence numbers. A per-key
seqlets the consumer drop duplicates and spot gaps, without relying on the broker alone. Keep the last appliedseqper key.
Global order is rarely needed. It needs one partition and one consumer, which caps at what a single consumer can do. Ask what the business really needs: one user's events in order, not two strangers'.
Tap any box to see what it is and what it talks to.
- Queue
- Worker
- External API
Text description
- The worker receives a send-receipt job.
- It acks immediately, so the broker deletes the message.
- The worker crashes before it calls the provider.
- Nothing is left to redeliver. The receipt is never sent and nobody knows.
BEGIN;
INSERT INTO processed_messages (message_id) VALUES ($1)
ON CONFLICT (message_id) DO NOTHING;
-- 0 rows inserted means a duplicate: ROLLBACK and ack the message
UPDATE accounts SET balance = balance + $2 WHERE id = $3;
COMMIT;if (event.seq <= last) return; // duplicate or stale: drop it
if (event.seq > last + 1) return parkUntil(last + 1); // gap: wait for the missing one
apply(event); last = event.seq;Retries, dead letters and delayed jobs
Some failures heal; some never will.
Read more · 594 words
Retry the first, waiting longer each time (). Park the second in a (DLQ). The triage is one question: could this exact message ever succeed?
| Failure | What happens | Fix |
|---|---|---|
| Transient (timeouts, 503s, deadlocks) | Usually heals if you wait. Without jitter, 1,000 consumers that failed together retry together at 1 s, 2 s and 4 s, and the recovering service sees synchronized waves | Exponential backoff plus jitter: 1 s, 2 s, 4 s, 8 s is 15 s of waiting over four retries. Cap attempts at about 5 and the delay at tens of seconds |
| Poison message (bad payload or a bug) | Always fails; without a DLQ it retries forever and burns worker time | Permanent errors go straight to the DLQ; the rest after the last attempt. In SQS set maxReceiveCount on the redrive policy (for example 5). Alert when the DLQ is not empty. Fix the bug, then redrive. Kafka has no built-in DLQ: your consumer publishes the bad record to a dead letter topic |
| Visibility timeout too short | A job takes 60 s, the timeout is 30 s. At second 30 the broker gives the job to another worker, so it runs twice at once | Set the timeout above the slowest job (p99 plus margin), or extend it while the worker is still busy |
| One bad message blocks a partition | Kafka commits offsets in order, so a message that fails forever at offset 100 stops offsets 101 and up, and lag grows. It looks like a healthy consumer that processes nothing | After the attempt limit, publish it to a dead letter topic and commit past it. Alert on consumer lag too |
Use backoff by default for any call to another service. Try the simulator and compare no retries, fixed, exponential and jittered.
Delayed and scheduled jobs. To cancel unpaid orders after 15 minutes, use a job that is invisible until its time:
- Short delays: the . SQS
DelaySeconds(up to 15 minutes) or BullMQ'sdelay: one line, no extra table. Use it under 15 minutes if you can tolerate the job living in the broker. A Redis-backed delay survives a restart only with persistence on. - Long delays: a
run_attable and a poller. For hours or days, storerun_atand poll due rows withFOR UPDATE SKIP LOCKED, or use a scored by run time. The job re-reads the state when it fires, so a paid order is left alone. It is durable, queryable and cancelable; precision is one poll interval (about a second); the cost is a poller and a partial index on pending rows. - Cron-style schedules sometimes run twice, because two scheduler instances can fire at once after a deploy. Key the job on the period (
report:2026-10-24) and record that key under a unique constraint.
Could this exact message ever succeed on a retry?
const delay = Math.random() * Math.min(capMs, baseMs * 2 ** attempt); // full jitterSELECT id, payload FROM jobs
WHERE run_at <= now() AND done_at IS NULL
ORDER BY run_at
LIMIT 50
FOR UPDATE SKIP LOCKED;Backpressure and monitoring
1,000 messages arrive each second and workers handle 800.
Read more · 590 words
The backlog grows to 720,000 after an hour, and a new arrival waits 15 minutes. A queue never fixes a rate mismatch; it only hides it for a while.
: items in a system = arrival rate x time in it. It sizes fleets: 1,000 jobs/s x 200 ms = 200 jobs in progress, so at least 200 worker slots. Add margin: at 70% utilisation you want about 290.
The arithmetic: 200 extra per second x 3,600 s = 720,000. A new arrival waits 720,000 / 800 per second = 900 s, which is 15 minutes.
is your answer when can't keep up. Four tools, and you often use two.
| Tool | How it works | Use it for | Watch out for |
|---|---|---|---|
| Bound the queue | Slow producers when it is full: answer 429 or 503 with Retry-After. The backlog stops growing | Public APIs and producers you do not control | Some requests are refused. That is the point |
| Autoscale on queue depth | Scale on depth or the age of the oldest message, not CPU: workers waiting on a slow downstream look idle on CPU while the backlog climbs | Bursts you can outrun. A new worker takes a minute to start, but the queue holds the spike meanwhile | Cap the fleet so you do not stampede the database behind it |
| Shed low-value work | Drop or delay cheap work first: a marketing push can wait an hour, a password reset cannot | Overload | Decide the order before the incident, not during it |
| One queue per priority | A million-message campaign in a shared queue sits in front of every login code | Urgent plus bulk work | Needs separate queues and separate workers |
A queue turns a spike into delay instead of errors. Say workers handle 2,000/s, normal traffic is 1,000/s, and a flash sale sends 5,000/s for 60 s. The backlog grows by 3,000/s x 60 = 180,000. After the sale it drains at 2,000 - 1,000 = 1,000/s net, so it takes 180 s to clear. At the peak, a new job waits 180,000 / 2,000 = 90 s.
Add workers or make jobs faster in the simulator, then hit the spike.
What to monitor. Four numbers tell you whether async work is healthy. Depth alone misleads, because 10,000 waiting messages is fine at 5,000/s and an outage at 5/s.
| Signal | Alert when | It means |
|---|---|---|
| Age of oldest message | Above your deadline (say 60 s for email) | Consumers cannot keep up |
| Queue depth (Kafka: lag) | Rising for 5 minutes | Backlog growing |
| DLQ depth | Above zero | A poison message needs a human |
| Consumer error rate | Above baseline | A dependency is failing |
Age is the one users feel. In the flash sale above, the peak depth is 180,000 but the number to alert on is the 90 s wait. Peak backlog 180K, drain time 180 s at +1,000/s net, depth target 0.
Tap any box to see what it is and what it talks to.
- Client
- Service
- Queue
- Worker
Text description
- Jobs arrive faster than the workers finish them, and the queue fills to its limit.
- A new request arrives. The API checks the depth before publishing.
- It answers 429 with Retry-After: 5. The backlog stops growing and callers wait their turn instead of the queue swallowing everything.
Show data table
| Seconds | Backlog (msgs) |
|---|---|
| 0 | 0 |
| 30 | 90000 |
| 60 | 180000 |
| 120 | 120000 |
| 180 | 60000 |
| 240 | 0 |
| 300 | 0 |
Quick checkWhich queue metric do users feel most directly?
Outbox and sagas
"Save the order, then publish order.created" can't be atomic: the database and the share no .
Read more · 833 words
Crash in between: the order exists, the event is lost, and nothing is charged or shipped. Publish first: if the transaction rolls back, act on an order that does not exist. At 100 orders per second, if just 0.01% of requests die in that window, you lose about 1 order every 100 seconds. It never throws an error, which is why it survives to production.
The (the default) writes the order and an outbox row in one local transaction. A relay then publishes unpublished rows. A crash can send a row twice, so consumers by event id. The event is late at worst, never lost.
Use it whenever a state change must also produce an event: orders, payments, account changes. The relay polls every 100 ms to 1 s, so events trail the commit by about that much.
- CDC (change data capture), for example Debezium, reads the database and publishes each committed outbox row: no query, no
published_atto update, events in commit order. Use it when you need milliseconds, or polling load on a busy table hurts. The cost is more infrastructure (a connector, usually on Connect) and a trap: a dead connector's slot makes Postgres keep all WAL, and the disk fills. - Duplicates: the relay can publish a row twice after a crash, so consumers dedupe by event id.
- Growth: delete published rows after a day or so, or the table and its index bloat.
- Lag: alert on the age of the oldest unpublished row; past a minute, the relay is stuck. With several relays, claim rows with
FOR UPDATE SKIP LOCKEDand key the events by entity so order still holds per key.
A purchase spans orders, inventory, payments and shipping, each with its own database. A chains local transactions, each with a compensating action that undoes it:
Orders: create the order as PENDING
Compensation: mark it cancelled.
Inventory: reserve the stock
Compensation: release the reservation.
Payments: charge the card
Compensation: refund. If the charge fails, undo in reverse.
Shipping: create the shipment
If this fails, refund, release and cancel. If it works, mark the order CONFIRMED.
Who drives the steps is the choice.
| Style | How it works | Use it for | Watch out for |
|---|---|---|---|
| Choreography | No coordinator: each service reacts to the previous service's event and publishes its own. Loose coupling, nothing central to run | 2 or 3 steps with simple failures | The flow lives in many services: to answer "where is order 17 stuck?" you read several codebases and trace events |
| Orchestration | A coordinator holds the whole flow, calls each service in turn and runs compensations in reverse on failure. One place to time out, retry and monitor | 4 or more steps, or whenever you must answer for the state of every order. Tools: Temporal, AWS Step Functions | One more service to run; it becomes a god object if business rules leak into it |
A saga has no isolation. Between reserve stock and charge card, another shopper sees the item as out of stock. Show a PENDING state, and expire reservations. Compensations can fail too, so make them and retry them until they succeed: a business undo (a refund), not a delete.
The numbers: if each of 4 steps succeeds 99.9% of the time, the saga completes 0.999 to the fourth, about 99.6%. One order in 250 needs at least one compensation, so compensation is a normal path, not an edge case. across services is the alternative, but it blocks while the coordinator is down, so teams rarely use it.
- Service
- Database
- Worker
- Queue
Text description
- Orders API → Postgres: BEGIN; INSERT order
- Orders API → Postgres: INSERT outbox row; COMMIT
- Postgres replies to Orders API: committed
- Outbox relay → Postgres: SELECT … SKIP LOCKED
- Postgres replies to Outbox relay: event rows
- Outbox relay sends (async) to Kafka: publish order.created (id)
- Outbox relay → Postgres: SET published_at = now()
- Kafka sends (async) to Consumers: deliver (at least once)
CREATE TABLE outbox (
id bigserial PRIMARY KEY,
topic text NOT NULL,
payload jsonb NOT NULL,
published_at timestamptz
);
CREATE INDEX outbox_pending ON outbox (id) WHERE published_at IS NULL;
BEGIN;
INSERT INTO orders (id, user_id, total_cents) VALUES ($1, $2, $3);
INSERT INTO outbox (topic, payload) VALUES ('order.created', $4);
COMMIT;Tap any box to see what it is and what it talks to.
- Service
Text description
- The orchestrator tells Inventory to reserve the stock.
- Reserved. The orchestrator records the step.
- Next it tells Payments to charge the card.
- Charged.
- Finally it creates the shipment and marks the order CONFIRMED.
Code
idempotent-consumer.js fixes a classic bug: marking a message "done" before sending, so a failed email was skipped on every retry.
Read more · 68 words
outbox-relay.js uses FOR UPDATE SKIP LOCKED, so parallel relays never publish the same row.
// At-least-once delivery means duplicates. Claim → do the work → mark done.
// On failure, release the claim so the retry actually runs (the classic version marked "done" first,
// so a failed send was skipped as a "duplicate" on every retry and never delivered).
export async function handle(msg) {
const doneKey = `done:${msg.id}`, lockKey = `processing:${msg.id}`;
if (await redis.exists(doneKey)) return; // already delivered
if (!(await redis.set(lockKey, 1, 'EX', 60, 'NX'))) return; // another worker is on it
try {
await sendEmail(msg); // pass msg.id to the provider as an idempotency key too
await redis.set(doneKey, 1, 'EX', 7 * 86_400); // only now is it "done"
} catch (err) {
const attempts = msg.attempts ?? 0;
if (attempts >= 4) await queue.publish('email.dlq', { ...msg, error: String(err) }); // 5th failure: park it
else await queue.publish('email.retry', { ...msg, attempts: attempts + 1 }, { delayMs: 2 ** attempts * 1_000 }); // 1 s, 2 s, 4 s, 8 s
} finally {
await redis.del(lockKey);
}
}Async messaging in the interview
Bring up a queue when slow, flaky or spiky work sits in the request path. Then walk five decisions in order.
Read more · 645 words
Decide what leaves the request path
Only what the user does not need in the response: email, transcode, . Say what stays sync.
Pick the tool
A work queue (SQS) for jobs; when several services read the events or need replay.
State the delivery guarantee
with after processing, and : a table keyed by message id.
Plan for failure
with , about 5 attempts, a with an alert, and (bounded queue, on depth).
Close the dual write
An for any event that must go with a database change; a with compensations if it spans services.
| If the interviewer says | You raise |
|---|---|
| "Send a notification after signup" | A queue and a worker; API returns fast, worker retries |
| "Millions of events, many consumers" | Kafka, partitions keyed by entity, one group per consumer |
| "What if the consumer crashes mid-job?" | At-least-once redelivery, so an idempotent consumer |
| "Events for one order must be in order" | Key by orderId; retry in place |
| "Traffic spikes 5x on a sale" | The queue absorbs it; autoscale on depth; 429 at the edge |
| "Save the order and tell other services" | Transactional outbox, not write then publish |
Quick checkWhich delivery guarantee do you state first for a job queue?
Interview questions
How do you avoid processing a message twice?
Make the : record the message id under a unique constraint, in the same .
How do you keep events for one user in order?
Key by userId, so one partition and one consumer hold them in order.
Kafka or RabbitMQ/SQS?
for streams that many groups read and replay; RabbitMQ or SQS for task queues.
What is backpressure?
Slowing producers when consumers can't keep up: bounded queues, 429/503.
Why an outbox instead of “write the DB, then publish”?
A crash in between loses the event. The commits it with the data.
A message fails every time. What happens to it?
Retry with up to about 5 attempts, then move it to a and alert. Fix the bug, then redrive it.
Your consumer crashes after processing but before the ack. What now?
The redelivers, so the consumer sees it twice. A table or an idempotent write makes the second pass harmless.
Check yourself
Question 1 of 5Score 2 out of 3 or better to mark this page complete.