The WebSocket Connections That Refused to Die

A client's real-time notification service kept crashing every few days. Restarts fixed it temporarily. The root cause was 14,000 zombie WebSocket connections from clients that had disconnected days ago.

The support ticket said "notification service keeps going down." The engineering team's response had been the same for three weeks: restart the pod, watch it recover, move on. It crashed every three to five days, and because the service wasn't in the critical payment path, nobody had investigated beyond the restart script.

I got pulled into this engagement at a logistics company last month for a broader reliability review, but this one kept nagging at me. Services that crash on a predictable but irregular cadence usually aren't random.

The first clue nobody followed

When I asked the team what they'd checked, the answer was "the logs." Fair enough. The crash logs showed a consistent pattern: the Node.js process hit its file descriptor limit and the OS refused to let it open new connections. The team had already bumped ulimit once, from 1024 to 4096. That bought them an extra day between crashes.

That should have been the clue. Raising the limit didn't fix the problem — it delayed it. Something was accumulating.

I pulled the connection metrics from their Prometheus instance. The graph looked like a sawtooth: a steady upward climb over days, then a vertical drop when the pod crashed and restarted. At the point of each crash, the service was holding between 12,000 and 16,000 open WebSocket connections.

The company had about 800 active users of the real-time notification feature.

800 users, 14,000 connections

The math didn't add up, and that's what made it interesting. Even if every user had multiple tabs open, 800 users shouldn't produce 14,000 connections. So I started counting what was actually connected.

I wrote a quick diagnostic endpoint that dumped connection metadata — client ID, connection timestamp, last message received. The results were damning. Most connections hadn't sent or received a message in hours. Some in days. The oldest active connection was nine days old, opened by a user whose browser had presumably crashed or whose laptop had gone to sleep and never come back.

These were zombie connections. The clients were long gone, but the server didn't know.

Why TCP didn't save us

Most developers assume that if a client disappears — closes the browser, loses Wi-Fi, gets hit by a bus — the TCP connection will eventually notice and clean up. In theory, that's true. TCP has a keepalive mechanism that sends probe packets after a period of inactivity. If the probes go unanswered, the connection gets torn down.

In practice, the Linux default for TCP keepalive is two hours of idle time before the first probe, then nine more probes at 75-second intervals. That's roughly two hours and eleven minutes before a dead connection gets cleaned up — if keepalive is even enabled on the socket, which it often isn't by default.

But that wasn't the whole story here. The architecture had a wrinkle.

The WebSocket server sat behind an AWS Application Load Balancer. ALB has its own idle timeout — 60 seconds by default. If no data flows over a connection for 60 seconds, ALB drops it. But here's the subtle part: ALB drops the connection between itself and the client. The backend connection — the one between ALB and the server — can stay open. ALB doesn't always send a clean TCP FIN to the backend when it drops the frontend side due to idleness. The server is left holding a socket that goes nowhere.

So the real lifecycle of a zombie connection looked like this: client disappears, ALB drops the client-facing side after 60 seconds of silence, backend socket lingers, TCP keepalive doesn't kick in for two hours, and even when it does, the probes might succeed because ALB's backend port was still technically reachable. The connection just sat there, doing nothing, consuming a file descriptor, forever.

The fix was boring and that's the point

The solution had three parts, and none of them were clever.

Application-level heartbeats. I added a ping/pong mechanism at the WebSocket protocol level. The server sends a ping frame every 30 seconds. If a client doesn't respond with a pong within 10 seconds, the server closes the connection. This is the single most important thing, because it doesn't rely on TCP or the load balancer to detect dead clients.

const HEARTBEAT_INTERVAL = 30_000;
const HEARTBEAT_TIMEOUT = 10_000;
 
wss.on('connection', (ws) => {
  ws.isAlive = true;
  ws.on('pong', () => { ws.isAlive = true; });
});
 
setInterval(() => {
  for (const ws of wss.clients) {
    if (!ws.isAlive) {
      ws.terminate();
      continue;
    }
    ws.isAlive = false;
    ws.ping();
  }
}, HEARTBEAT_INTERVAL);

TCP keepalive tuning. I dropped the OS-level keepalive idle time from two hours to five minutes, and the probe interval from 75 seconds to 10 seconds. This acts as a second line of defense for connections that somehow survive the application heartbeat.

Connection age limit. Any connection older than 24 hours gets gracefully closed with a reconnection signal. Clients already had reconnection logic (most WebSocket client libraries do), so this was painless. It puts a hard cap on how long any single connection can accumulate as a zombie if the other mechanisms fail.

What didn't fix it

Before I got involved, someone on the team had tried setting ALB's idle timeout to 3600 seconds, thinking that would keep connections alive longer and fix the drops. It did the opposite — it made zombie connections survive even longer on the ALB side, which made the leak worse.

Another suggestion was to just "increase the file descriptor limit to 100,000." That's not fixing the leak; that's buying a bigger bucket for the dripping faucet. The server was holding connections that were consuming memory, goroutine equivalents (in the Node event loop), and file descriptors — all for clients that would never send another byte.

The part that still bothers me

After deploying the fix, the steady-state connection count dropped from a saw-toothing 14,000 to a stable 900-1,100. The service hasn't crashed in four weeks.

But what gets me is that this ran for three weeks without investigation. The team had monitoring. They had alerts. They even had a Grafana dashboard that showed the connection count climbing. Nobody looked at it because the service wasn't "important enough" to investigate — the restart script worked, and there were always higher-priority fires.

I've seen this pattern a dozen times now. The problems that eventually take down production are rarely the ones teams are actively worried about. They're the quiet ones in the corner, growing linearly, patiently waiting for a limit that was never designed for them.

If you have a service that needs periodic restarts to stay healthy, that's not a healthy service — it's a time bomb with a known fuse length. Stop resetting the timer and find what's leaking.