Scaling 100k WebSocket connections: the reconnect storm

A real-time platform I work on holds over a hundred thousand concurrent client connections. The socket count was never the hard part. What actually broke us was the reconnect storm.

The socket count is tuning

Node handles that many connections fine. Idle connections are cheap, and the hard limits people worry about are mostly tuning: file descriptors, ephemeral ports, keepalive intervals. You hit one, you read a man page, you move on.

The reconnect storm

A network blip drops thirty thousand clients, and every one of them reconnects. If they all retry immediately they arrive together, and the thundering herd takes down a service that was fine two seconds earlier. It restarts, they all reconnect again, and now you have a loop that keeps itself down.

The fixes have little to do with sockets

The first fix lives on the client: jittered exponential backoff, so reconnects spread over a window instead of landing at once.

function reconnectDelay(attempt: number) {
  const base = Math.min(30_000, 1_000 * 2 ** attempt);
  return Math.random() * base; // full jitter
}

The server gets a connection budget. Once it's over budget it sheds load and keeps running. Rejecting some connections quickly beats accepting all of them and falling over.

Then make the reconnect itself cheap with resumable sessions. A returning client skips the full re-auth and state rebuild, so each one costs the server far less than a fresh connection.

Once the socket count is tuned, the scaling problem that's left is what the system does in the ten seconds after something goes wrong.