Build · Do it later

Queues and jobs

How do I do slow work without making the user wait?

A queue is a to-do list for your code. Your app drops a job on the list and answers the user right away. A separate worker program picks jobs up one by one and does the slow part, like sending email or resizing images.

It is a restaurant order slip: the waiter pins the slip up and walks away, and the kitchen cooks when it is ready.

Add it when Something takes more than a second or two, can fail and be retried, or talks to another company's API.

It can wait when Everything finishes instantly inside the request and nothing needs a retry.

1 Which one should I pick?

Answer what you can. The ranking updates as you go, and you can ignore it if you disagree.

Where does your app run?

What do you need most?

What matters most right now?

Most people start with

2 Side by side

Three pips is strong, one is weak. Pick a row to read about it below.

Option	Quick to set up	Cheap while small	Easy to leave	Room to grow	Price
					Free tier, then pay per request
					Free library; you pay for Redis and a server
					Free tier, then pay per run
					Free to self-host, or pay a hosting company
					Free to self-host; managed versions are pay as you go

3 Look closer, then wire it in

Cloud primitive

AWS SQS

Amazon's plain, reliable queue. You send messages and workers read them.

Pick it when You already run on AWS and want a queue you never have to look after.

Skip it if You want a nice dashboard, scheduling and retries without wiring them up yourself.

Nothing to run or patch
Very reliable and scales on its own
Built-in dead-letter queue for failed jobs

AWS accounts and permissions are confusing at first
You write the worker loop yourself

Quick to set up
Cheap while small
Easy to leave
Room to grow

PriceFree tier, then pay per request

How to connect it

Create a queue in the AWS console and note its URL.
Create an access key that is only allowed to use that queue.
Send a message from your app with the AWS SDK.
Run a worker that reads messages, does the work, then deletes each message.

Put these in your .env file

AWS_REGION
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
SQS_QUEUE_URL

enqueue.ts

import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";

const sqs = new SQSClient({ region: process.env.AWS_REGION });

export const enqueue = (job: object) =>
  sqs.send(new SendMessageCommand({
    QueueUrl: process.env.SQS_QUEUE_URL,
    MessageBody: JSON.stringify(job),
  }));

4 Watch it work

Press play, or step through one message at a time.

Client
Service
Queue
Worker

Text description

Browser → Your app: Request: export my report
Your app → Queue: Add job: export report 42
Your app replies to Browser: Got it, we are working on it
Worker → Queue: Any jobs for me?
Queue replies to Worker: Job: export report 42
Worker: Do the slow work (fails once)
Worker → Queue: Failed: try again later
Queue replies to Worker: Same job again, second try
Worker → Queue: Done: remove the job

5 Mistakes to skip

Doing slow work inside the request.
Put a job on a queue and reply right away. Tell the user it is in progress.
Writing jobs that are not safe to run twice.
Queues can deliver a job more than once. Make jobs idempotent: check if the work is already done before doing it.
Ignoring jobs that keep failing.
Send them to a dead-letter queue (a holding area for failed jobs) and set an alert so you notice.

See where it sits in a whole app

The stack builder draws your product with queues and jobs in it and animates a request through it.

Open the stack builder

What it is