Connections: Pooling, Keep-Alive and Limits

Why it matters

Opening a connection is like phoning a shop and waiting on hold before you can ask anything.

Connection cost and keep-alive

Connection	Setup	Server holds	Reuse with
TCP	1 round trip	A socket and a file descriptor	Keep-alive
TLS 1.3 / 1.2	+1 / +2 round trips, plus crypto	Session keys	Keep-alive, session resumption
HTTP/2	One TCP + TLS for many requests	Streams on one connection	Multiplexing
HTTP/3 (QUIC)	1 round trip for transport and TLS together	QUIC connection state	Multiplexing, 0-RTT resumption
PostgreSQL	TCP + TLS + auth: up to tens of ms	A whole process (a few MB)	A pool, then PgBouncer
MySQL	TCP + TLS + auth	A thread	A pool, ProxySQL
Redis	TCP + `AUTH`: under 1 ms nearby	A few KB	One shared, pipelined client

Pool sizing and PgBouncer

Each instance opens its own pool, so the rule is about the total:

Question	Answer
Top down: what is the default?	An app-side pool keeps a few open connections inside each instance and lends them to requests, so only the first request on a connection pays the handshakes. 15 per instance gives 12 × 15 = 180 server connections, under the 200 limit with 20 reserved. The cost: the total moves with your instance count, so autoscaling to 30 instances quietly makes it 450
Bottom up: is it big enough?	Busy connections equal queries per second times the time each holds one (Little's law). The pool needs room for that number plus bursts
Too big?	A pool bigger than the database can use only adds waiting. A useful rule is about twice the cores plus one, so roughly 17 on 8 cores. A pool of 200 on that database puts 25 backends on every core, so each query runs slower and the timeouts you wanted to cure get worse
Too many instances?	The fix is a proxy, not bigger pools. 50 instances × 20 = 1,000 connections against a limit near 200 needs PgBouncer. With a few instances, skip it

Mode	A server connection lasts	Use it for
Session	As long as the client stays connected. Saves handshakes but almost no connections	Session features (`LISTEN`, advisory locks, temporary tables) and a modest client count: a migration runner, a `LISTEN` worker
Transaction (the usual pick)	From `BEGIN` to `COMMIT`. Between transactions the client holds nothing, which is what lets 240 clients share 40	Most web apps, and the default answer in an interview. Size the server-side pool by what the database can run at once, not by the client count
Statement	One statement, so multi-statement transactions are refused	Autocommit-only workloads, such as simple key lookups. Rare in practice

Average time vs pool size at 340 req/s (axis stops at 150 ms; timeouts go off the top, see the table)
Pool size (connections)	Wait for a connection (ms)	Wait + query (ms)
0	2000	2000
1	1999.49	2002.43
2	1998.77	2004.66
3	1997.68	2006.5
4	1995.8	2007.56
5	1991.83	2006.54
6	1977.94	1995.59
7	91.55	111.55
8	9.54	29.54
9	8.43	29.92
10	7.49	30.29
11	6.69	30.65
12	6.01	31
13	5.44	31.34
14	4.94	31.68
15	4.52	32.01
16	4.15	32.33
17	3.84	32.64
18	3.56	32.95
19	3.33	33.25
20	3.12	33.55
21	2.95	33.84
22	2.8	34.14
23	2.67	34.43
24	2.57	34.73
25	2.48	35.02
26	2.41	35.32
27	2.35	35.63
28	2.31	35.94
29	2.29	36.26
30	2.28	36.6
31	2.28	36.94
32	2.31	37.3
33	2.34	37.68
34	2.4	38.09
35	2.48	38.52
36	2.58	38.99
37	2.71	39.51
38	2.87	40.07
39	3.07	40.7
40	3.33	41.4
41	3.64	42.21
42	4.04	43.15
43	4.55	44.26
44	5.21	45.59
45	6.07	47.22
46	7.24	49.29
47	8.86	51.98
48	11.2	55.64
49	14.75	60.85
50	20.48	68.8
51	30.6	82.02
52	50.85	106.97
53	98.14	162.18
54	224.92	303.93
55	549.48	656.23
56	1074.4	1215.44
57	1481.69	1643.3
58	1673.5	1842.95
59	1762.2	1935.54
60	1811.58	1988.02

Busy connections on average
	connections
100 queries/s × 5 ms	0.5 connections
1,000 queries/s × 5 ms	5 connections
1,000 queries/s × 50 ms	50 connections

Timeouts, deadlines and limits

A timeout bounds one call and a deadline bounds the whole request.

Timeout	What it bounds	Typical setting
Connect	Opening the connection	1 to 3 s
Pool acquire	Waiting for a free connection	1 to 2 s, then fail fast
Request or query	One call	A few times its p99
Deadline	The whole request, every hop	From the SLO, e.g. 2 s

Descriptors, shutdown and HTTP versions

Every socket and open file uses one file descriptor, and a process has a limit on them.

Version	How it works	Use it when
HTTP/1.1	One request at a time per connection, so a browser opens about 6 and pays six handshakes. Thirty files take 5 rounds	Legacy clients only. Do not shard domains to get more connections: under HTTP/2 that splits one good connection into several
HTTP/2	One TCP connection shared by many streams, so one handshake serves all 30 files. The catch: one lost packet stalls every stream	The default, from browser to edge and between services. In a datacenter, packet loss is rare, so the stall seldom matters
HTTP/3	Runs over QUIC: each stream has its own recovery path, a new connection costs one round trip, and the connection survives a change of network	The edge or CDN for mobile and lossy users, with HTTP/2 as the fallback when UDP is blocked. 0-RTT resumption can be replayed, so accept only safe, idempotent requests like `GET` in it

What goes wrong with connections

Connection trouble shows up as one symptom, timeouts, from four different causes.

Failure	Pool looks like	Database looks like
Exhaustion	Full, many waiting	Busy with slow queries
Storm	Opening many at once	At `max_connections`
Idle in transaction	Full, many waiting	Idle sessions, held locks
Leak	Full, none ever return	Idle

Code

Database connection pooling

import pg from 'pg';
import { poolMaxPerInstance } from './pool-size.js';

// Postgres max_connections = 200, keep 20 for admin and migrations, 12 app instances
const poolMax = poolMaxPerInstance({ maxConnections: 200, reserved: 20, instances: 12 });   // 15. More instances? Add PgBouncer first.

export const pool = new pg.Pool({
  max: poolMax,
  idleTimeoutMillis: 30_000,          // close connections idle for 30 s
  connectionTimeoutMillis: 2_000,     // wait at most 2 s for a free connection, then fail fast
});

// A single query borrows a connection and returns it automatically
export const findUser = async id => (await pool.query('SELECT * FROM users WHERE id = $1', [id])).rows[0];

// A transaction must run every statement on ONE connection, and must always give it back
export async function transfer(from, to, cents) {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    await client.query('UPDATE accounts SET balance = balance - $1 WHERE id = $2', [cents, from]);
    await client.query('UPDATE accounts SET balance = balance + $1 WHERE id = $2', [cents, to]);
    await client.query('COMMIT');
  } catch (e) {
    await client.query('ROLLBACK');
    throw e;
  } finally {
    client.release();                 // a forgotten release leaks a connection: 15 leaks and this instance is stuck
  }
}

HTTP keep-alive

keep-alive.js

import { Agent, setGlobalDispatcher } from 'undici';

// Node's fetch (undici) already reuses connections; tune its pool for heavy service-to-service traffic
setGlobalDispatcher(new Agent({ connections: 128, keepAliveTimeout: 10_000, keepAliveMaxTimeout: 60_000 }));

// The server side, behind a load balancer: keep idle sockets open LONGER than the LB does (AWS ALB: 60 s).
// Otherwise Node closes a socket at the moment the LB reuses it, and the user gets a 502.
server.keepAliveTimeout = 65_000;
server.headersTimeout = 66_000;       // keep this above keepAliveTimeout

Timeouts and deadlines

deadline.js

// Deadlines: give the whole request a budget and pass what's left downstream
app.use((req, _res, next) => {
  const fromCaller = Number(req.get('x-deadline-ms')) || Infinity;   // an upstream service may have less time left
  req.deadline = Date.now() + Math.min(2_000, fromCaller);
  next();
});
const remaining = req => Math.max(0, req.deadline - Date.now());

app.get('/checkout', async (req, res) => {
  const user = await fetch(`${USERS_URL}/${req.userId}`, { signal: AbortSignal.timeout(Math.min(500, remaining(req))) }).then(r => r.json());
  const quote = await fetch(`${PRICING_URL}/quote`, {
    method: 'POST', body: JSON.stringify(user.cart),
    headers: { 'Content-Type': 'application/json', 'x-deadline-ms': String(remaining(req)) },   // downstream can give up early too
    signal: AbortSignal.timeout(remaining(req)),
  });
  res.json(await quote.json());
});

// A timed-out dependency becomes a fast 504 instead of a hung request
app.use((err, _req, res, _next) => res.status(err.name === 'TimeoutError' ? 504 : 500).end());

Limiting concurrency

map-limit.js

// Run fn over items with at most `limit` calls in flight. Results keep the input order.
export async function mapLimit(items, limit, fn) {
  const out = [];
  let i = 0;
  await Promise.all(Array.from({ length: limit }, async () => {   // `limit` workers share one cursor
    while (i < items.length) {
      const n = i++;                                               // claim an index synchronously: no two workers get the same one
      out[n] = await fn(items[n]);
    }
  }));
  return out;
}

Graceful shutdown

shutdown.js

// Graceful shutdown for an HTTP + WebSocket server (extends the source card)
let shuttingDown = false;
app.get('/ready', (_req, res) => res.sendStatus(shuttingDown ? 503 : 200));   // the load balancer's readiness probe

process.on('SIGTERM', () => {
  shuttingDown = true;                                         // readiness → 503: the LB stops sending new requests
  wss?.clients.forEach(c => c.close(1001, 'server restarting'));   // clients reconnect elsewhere (with jitter)
  server.close(() => pool.end().then(() => process.exit(0)));  // wait for in-flight requests, then close the DB pool
  server.closeIdleConnections();                               // drop idle keep-alive sockets now
  setTimeout(() => process.exit(1), 25_000).unref();           // hard stop before the orchestrator's kill timeout
});

Pool guardrails

pool-guard.js

import pg from 'pg';

export const pool = new pg.Pool({
  max: 15,
  connectionTimeoutMillis: 2_000,                  // fail fast when the pool is dry
  statement_timeout: 5_000,                        // the server cancels any single query after 5 s
  idle_in_transaction_session_timeout: 10_000,     // the server kills a session that sits in a transaction for 10 s
});

// An idle client that errors must not crash the process
pool.on('error', err => console.error('idle client error', err.message));

// Log the three numbers that reveal exhaustion and leaks
setInterval(() => {
  const { totalCount, idleCount, waitingCount } = pool;
  if (waitingCount > 0 && idleCount === 0) {
    console.warn(`pool dry: ${totalCount} open, 0 idle, ${waitingCount} waiting`);   // alert on this
  }
}, 5_000).unref();

// And ask the server who is stuck in a transaction:
//   SELECT pid, now() - xact_start AS age, query FROM pg_stat_activity WHERE state = 'idle in transaction' ORDER BY age DESC;

Connection pooling in the interview

Bring connections up the moment the design has many app servers talking to one database, then walk through five decisions in order.

If the interviewer says	You raise
"We autoscale to 100 instances"	Top-down pool math, then PgBouncer in transaction mode
"The database CPU is at 100%"	Pool too big or a slow query; fewer connections, not more
"Requests hang when a dependency is slow"	Timeouts inside a deadline, bounded concurrency, a breaker
"We deploy during peak traffic"	Rolling deploys, readiness, drain, jittered reconnect
"Mobile users on bad networks"	HTTP/3 at the edge, HTTP/2 as fallback

Interview questions

We scaled from 10 to 30 app instances and Postgres started refusing connections. Why?

Each instance opens its own pool: 30 × 20 = 600, far past max_connections. Divide the limit by instances, and add PgBouncer.

Why can a bigger pool make things slower?

Past about the database's core count, extra connections only add waiting. A small pool with a short queue serves more.

How do you deploy WebSocket servers without dropping users?

Fail readiness, send close 1001, and let clients reconnect elsewhere with .

How would you size the connection pool?

Top down: the total at maximum fits under the limit. Bottom up: × hold time, plus burst room.

A downstream service got slow, and now our service is down too. What happened?

Calls without timeouts held every slot. Add timeouts inside a deadline, a and a fallback.

Check yourself

Question 1 of 5

Score 2 out of 3 or better to mark this page complete.

1. 12 app instances each open a pool of 20, and PostgreSQL has max_connections = 200. What happens?FineConnection errors under loadPostgreSQL queues the extra connections

Connections: Pooling, Keep-Alive and Limits

Trade-offs

Bring it up before they ask

Why it matters

Connection cost and keep-alive

Handshake timeline

Pool sizing and PgBouncer

Connection pool simulator

Timeouts, deadlines and limits

Descriptors, shutdown and HTTP versions

Fail readiness

Stop accepting and close idle sockets

Finish in-flight work

Exit before the kill

What goes wrong with connections

Code

Database connection pooling

HTTP keep-alive

Timeouts and deadlines

Limiting concurrency

Graceful shutdown

Pool guardrails

Connection pooling in the interview

Name the scarce resource

Size the pools from the top

Add PgBouncer only when instances multiply

Bound the waiting

Own the failures

Interview questions

Check yourself

Deep dives