BackendMedium35 min

Design a Unique ID Generator

Make unique, roughly time-ordered 64-bit ids on every server with no network call, and survive clock skew and shared worker ids.

Say it in 30 seconds
  • 69.7 yearsof time in 41 bits
  • 4,096 / msids per worker from 12 bits
  • 1,024workers from 10 bits
Show what to say

“I'd use Snowflake-style 64-bit ids: 41 bits of milliseconds since a custom epoch, 10 bits of worker id and 12 bits of sequence. Each server makes ids locally, 4,096 per millisecond, so ids are unique and roughly time-ordered with no network call. Worker ids are leased with a TTL, and the generator refuses to issue ids if the clock steps back.”

Trade-offs

  • Snowflake vs UUID v4: 64 bits, local and time-ordered, but needs worker ids and sane clocks; UUID needs neither and is 128 random bits.
  • Ticket server vs Snowflake: strictly ordered and simple, but a shared database and a round trip, or a block with gaps.
  • Wait vs refuse on clock skew: small backward steps are waited out, but a large step must fail loudly to avoid duplicates.

Bring it up before they ask

Clock going backwards: refuse to issue ids on a big step and fail over, because a duplicate primary key is worse than a short error.

0 of 0 sections ticked
30-second answer

Pack time, a worker id and a counter into 64 bits (a ) so every server makes unique, roughly time-ordered ids locally, with no network call. The risks are a clock that goes backwards and two servers sharing a worker id.

Functional

  • Generate globally unique ids across servers and data centers
  • Ids are 64-bit numbers
  • Ids sort roughly by creation time
  • No central lookup on the hot path

Non-functional

  • 100K ids a second at peak across the fleet
  • Under 1 ms to make an id
  • No duplicates, ever, even during failures
  • Keeps working if the id registry is briefly down
  • Lasts decades

Out of scope

  • Gapless sequences (invoice numbers with legal rules)
  • Hiding how many ids were issued
  • Ids that are hard to guess (use random tokens for that)

Clarifying questions, scale and interface

Read more · 250 words
  • How big and how sorted? Must the id fit 64 bits? Must it sort by time, or just be unique?
  • How many ids, from how many servers? This sets the bit split.
  • How strict is "unique"? One duplicate in a billion is too many for .

Capacity estimate

QuantityCalculationResult
Demand100K ids/s ÷ 20 servers5K/s per server
One worker's limit4,096 per ms × 1,000 ms4.096M/s, about 800x headroom
Fleet limit1,024 workers × 4.096M4.19 billion/s
Lifetime2^41 ms ÷ (1000 × 3600 × 24 × 365.25)69.7 years from the epoch
Rows10B × 8 B (64-bit)80 GB of keys
Rows as UUIDs10B × 16 B160 GB, and bigger indexes

Speed is not the problem. Clock correctness and who owns which worker id are.

API

Usually a library call inside each service: nextId(). A shared service is optional.

MethodPathNotes
POST/v1/ids?count=100Returns ids as strings, in increasing order
GET/v1/ids/:idDecodes time, worker and sequence (debugging)

JavaScript numbers are exact only to 2^53 - 1, but these ids reach 2^63: send them as strings.

Data model

Nothing about ids is stored. Only leases are.

WhereKeyValue
etcd/workers/17owner and a 30 second TTL
Orders tableid bigint PRIMARY KEYthe generated id
-- the ticket-server alternative: one row per counter
CREATE TABLE tickets (id smallint PRIMARY KEY, next bigint NOT NULL);
UPDATE tickets SET next = next + 1000 WHERE id = 1 RETURNING next - 1000 AS first;

Architecture

Each server makes its own ids.

Read more · 33 words

The only shared piece is the lease registry, used at boot and every ten seconds after. Play Boot and first id, 4,097 ids in one millisecond, Clock moves backwards and Lease lost.

Press Play to watch it run · 6 steps

Each app server runs an id generator in its own process. At boot it leases a worker id from a small registry (etcd). After that, every id is made locally from the clock, the worker id and a per-millisecond sequence, with no network call. A leaked or lost lease, or a clock that jumps back, makes the generator stop rather than risk a duplicate.

Snowflake-style id generationEach app server runs an id generator in its own process. At boot it leases a worker id from a small registry (etcd). After that, every id is made locally from the clock, the worker id and a per-millisecond sequence, with no network call. A leaked or lost lease, or a clock that jumps back, makes the generator stop rather than risk a duplicate.nextId()now()synclease, renewINSERTApp servercreates ordersA library inside each server. nextId() takes about a microsecond and never calls the network.ID generatorin processSystem clockmillisecondsNTP serverskeep clocks closeHands out worker ids 0 to 1,023 as expiring leases, so two live servers never share one.Worker leasesetcd, 30 s TTLOrders DBbigint primary key

Tap any box to see what it is and what it talks to.

  • Service
  • ID generator
  • External API
  • Database
Open in playground
Text description
  1. A server starts. Its generator asks the registry for a worker id lease.
  2. It gets worker 17 for 30 seconds, and will renew every 10. No other live server can hold 17.
  3. The app needs an id for a new order.
  4. The generator reads the clock: 4,512,000 ms since our epoch.
  5. It returns time (41 bits), worker 17 (10 bits) and sequence 0 (12 bits), packed into one 64-bit number.
  6. The app inserts the row. No call left the machine to make that id.

The Snowflake layout

Split 64 bits into fields. The top field is time, so ids sort by time.

Read more · 33 words

The worker field stops two servers colliding. The sequence field separates ids made in the same millisecond.

Use a custom epoch (2026 here), not 1970, so the 41 bits last until about 2095.

Sign
1 bit
always 0
Time
41 bits
69.7 years of ms
Worker
10 bits
1,024 servers
Sequence
12 bits
4,096 per ms

Clocks, workers and failure

Uniqueness depends on two promises. Time never repeats for one worker: if the clock steps back, wait out a few milliseconds, but refuse and alert on a big step.

Read more · 72 words

No two live servers share a worker id: lease ids with a , and stop issuing a few seconds before the lease ends.

Ids are only roughly ordered across servers, because clocks differ by milliseconds. Do not use them to decide which event happened first.

Choosing a scheme

Pick by constraints, not by habit. Click through the tree.

Which id scheme?yesnoyesnoyesnoyesyesyesnoyesnoyesnononoTicket serverSnowflake-style idUnder about 10,000 ids asecond in total?Snowflake-style idCan every writer cheaplyreach one small databasefor ids?UUID v7Must the id fit in 64bits (smaller indexes,bigint columns)?UUID v4Ticket serverSnowflake-style idUnder about 10,000 ids asecond in total?Snowflake-style idCan every writer cheaplyreach one small databasefor ids?UUID v7Must the id fit in 64bits (smaller indexes,bigint columns)?Is a 128-bit random idacceptable (largeindexes, no locality)?Do ids need to sort bycreation time?

Do ids need to sort by creation time?

How to approach it

  1. Clarify

    64 bits? Sorted? How many ids, how strict?

  2. Estimate

    5K ids a second per server against 4.1M of capacity.

  3. Compare schemes

    Auto-increment, UUID, , .

  4. Design the bits

    41 time, 10 worker, 12 sequence.

  5. Failure

    Clock skew, worker leases, sequence overflow.

Read more · 211 words

Step-by-step answer

  1. Rule out the easy answers

    A database auto-increment is one machine and a round trip per id. A random UUID is 128 bits, unordered, and scatters inserts across the index.

  2. Spend the bits

    1 sign, 41 time, 10 worker, 12 sequence. Say what each buys: order, no collisions, .

  3. Generate locally

    Read the clock, bump the sequence in the same millisecond, wait for the next one if it overflows. No network call.

  4. Assign worker ids safely

    Lease ids from etcd or ZooKeeper with a . Renew often, and stop early when a renewal fails.

  5. Defend against the clock

    Wait out tiny backward steps. Refuse large ones, alert, and fail over. Keep NTP healthy with slewing rather than stepping.

  6. Mention the alternatives

    Ticket servers with blocks and odd and even machines; UUID v7 when 128 bits is fine and you want no coordination.

  7. Show the gotchas

    Ids as strings in JSON; gaps after crashes are normal; ids are roughly, not strictly, ordered across servers.

Implementation

Tabs: the generator, a with block allocation, UUID v7, and the worker lease registry.

Read more · 56 words
  • The Snowflake tests check the layout numbers (69.7 years, 4,096 per ms, 1,024 workers), the sequence overflow wait, and the clock-back cases.
  • The ticket test shows 1 million ids cost 1,000 round trips, and a crash leaves gaps, never duplicates.
  • The registry test shows an id reused only after its lease expires.
// Snowflake-style 64-bit ids, generated locally with no coordination:
//
//   | 1 bit | 41 bits: ms since our epoch | 10 bits: worker | 12 bits: sequence |
//     sign          time (about 69.7 years)     1,024 workers    4,096 ids per ms
//
// Ids sort by creation time (to the millisecond), fit in a signed 64-bit integer, and two workers can never
// collide because the worker bits differ. The clock is injected so tests can move it.
export const EPOCH = 1_767_225_600_000n;          // 2026-01-01T00:00:00Z, our "time zero"
const WORKER_BITS = 10n;
const SEQ_BITS = 12n;
const MAX_SEQ = (1n << SEQ_BITS) - 1n;            // 4095
const MAX_WORKER = (1n << WORKER_BITS) - 1n;      // 1023

export class Snowflake {
  #last = -1n;
  #seq = 0n;

  // maxBackwardMs: a clock that steps back this far or less is waited out; more than that is an error.
  constructor(workerId, { now = () => BigInt(Date.now()), maxBackwardMs = 5n } = {}) {
    const w = BigInt(workerId);
    if (w < 0n || w > MAX_WORKER) throw new RangeError(`workerId must be 0..${MAX_WORKER}`);
    this.worker = w;
    this.now = now;
    this.maxBackwardMs = maxBackwardMs;
  }

  nextId() {
    let ts = this.now() - EPOCH;
    if (ts < this.#last) {
      const behind = this.#last - ts;
      if (behind > this.maxBackwardMs) throw new Error(`clock moved back ${behind} ms; refusing to issue ids`);
      while (ts < this.#last) ts = this.now() - EPOCH;        // a tiny step back (NTP slew): wait it out
    }
    if (ts === this.#last) {
      this.#seq = (this.#seq + 1n) & MAX_SEQ;
      if (this.#seq === 0n) {                                  // 4,096 ids used up in this millisecond
        while (ts <= this.#last) ts = this.now() - EPOCH;      // wait for the next one
      }
    } else {
      this.#seq = 0n;
    }
    this.#last = ts;
    return (ts << (WORKER_BITS + SEQ_BITS)) | (this.worker << SEQ_BITS) | this.#seq;
  }
}

export function decode(id) {
  const n = BigInt(id);
  return {
    timeMs: Number((n >> (WORKER_BITS + SEQ_BITS)) + EPOCH),
    worker: Number((n >> SEQ_BITS) & MAX_WORKER),
    seq: Number(n & MAX_SEQ),
  };
}

// Handy numbers for the page.
export const limits = {
  yearsOfTime: Number(1n << 41n) / (1000 * 3600 * 24 * 365.25),
  idsPerMsPerWorker: Number(MAX_SEQ + 1n),
  workers: Number(MAX_WORKER + 1n),
};

Failure modes and trade-offs

Read more · 155 words
What breaksEffectMitigation
Clock steps backDuplicate idsWait out small steps; refuse big ones; alert
Two servers share a worker idDuplicate idsLeases with TTL; stop before expiry
Registry (etcd) is downNew servers cannot startRunning servers keep their lease margin; cache the id on disk
Sequence overflowWould repeat an idWait for the next millisecond
Server crashesUnused ids lostGaps are fine
Epoch runs outIds overflow in decadesChoose the epoch; plan a format change

Trade-offs

ChoiceGainCost
Snowflake64 bits, local, orderedNeeds worker ids and good clocks
UUID v4No coordination128 bits, random inserts
UUID v7Sortable, no coordination128 bits
Ticket serverSimple, strictly orderedSingle point of contention, round trips

Follow-up questions

What if the sequence overflows?

Wait for the next millisecond. 4,096 per ms per worker is far above real demand; if not, add worker bits.

Are the ids strictly ordered?

No. Within one worker yes; across workers only to clock accuracy. Use a log or database sequence if you need strict order.

How do you handle data centers?

Split the 10 worker bits, for example 5 for the data center and 5 for the machine, or just lease from one global registry.

Check yourself

Question 1 of 3

Score 2 out of 3 or better to mark this page complete.

1. In a Snowflake-style id (41 bits time, 10 bits worker, 12 bits sequence), how many ids can one worker make per millisecond, and per second?

Patterns behind this

The coding patterns this design is built on. Practise them and the design questions get easier.