The Graceful Shutdown That Wasn't

Every deploy dropped a handful of requests. The service had a SIGTERM handler. It just didn't do what anyone thought it did.

The client's Slack channel had a bot that posted deploy notifications. Fourteen times a day, on average. Right after each notification, if you scrolled up in their #api-alerts channel, you'd find a small cluster of 502 errors. Five here, twelve there. Sometimes thirty.

Their API gateway logged about 3,200 requests per minute during business hours. Losing 15 requests per deploy didn't set off any alarms. But 15 requests times 14 deploys is 210 dropped requests a day. Some of those were payment confirmations from a webhook provider that retried three times and then gave up. They'd been silently losing about two orders a week for months before a customer complained.

When I joined the engagement, the team's theory was a load balancer misconfiguration. They'd already spent two weeks tuning ALB settings. The actual problem was much more mundane.

"We handle SIGTERM"

The service was a Node.js API running on Kubernetes — three replicas behind an internal ALB. When I asked whether they handled graceful shutdown, the lead engineer pulled up the code immediately. He'd written it himself.

process.on('SIGTERM', () => {
  console.log('Received SIGTERM, shutting down');
  server.close();
  process.exit(0);
});

Three lines. Looks fine. It wasn't.

server.close() in Node.js stops the server from accepting new connections, but it doesn't wait for in-flight requests to finish. It calls its callback — or in this case, nothing at all — once all connections are drained. But process.exit(0) fires immediately, on the next tick. Any request that was mid-response when SIGTERM arrived got its TCP connection yanked out from under it.

That was problem one. But even fixing it wouldn't have been enough.

The race condition nobody draws on whiteboards

Kubernetes pod termination is a sequence of parallel events that most people think is sequential. When a pod enters the Terminating state, two things happen at the same time:

1. The kubelet sends SIGTERM to the container
2. The endpoints controller removes the pod from the Service's endpoint list

These are not ordered. The endpoint removal propagates through kube-proxy, then through the ingress controller or cloud load balancer, and eventually traffic stops routing to that pod. That propagation takes time — typically 1 to 5 seconds, sometimes longer with cloud load balancers.

So there's a window where the pod has received SIGTERM and started shutting down, but traffic is still arriving because the load balancer hasn't caught up. If your SIGTERM handler immediately stops accepting connections or exits, those in-flight routing updates turn into 502s.

The standard fix is a preStop hook that sleeps for a few seconds, giving the load balancer time to deregister the pod before the application starts shutting down. This client had no preStop hook. Nobody on the team had heard of one.

Putting it together

The fix was three changes, all in the same PR.

First, the preStop hook in the pod spec to buy time for endpoint propagation:

lifecycle:
  preStop:
    exec:
      command: ["sleep", "5"]

Second, a shutdown handler that actually drains:

process.on('SIGTERM', () => {
  console.log('SIGTERM received, draining connections');
  server.close(() => {
    console.log('All connections drained, exiting');
    process.exit(0);
  });
 
  setTimeout(() => {
    console.error('Forced shutdown after timeout');
    process.exit(1);
  }, 25000);
});

server.close() now gets a callback that only exits after all connections are done. The 25-second timeout is a safety net — if something hangs, the process still exits before Kubernetes sends SIGKILL at the terminationGracePeriodSeconds default of 30.

Third, we set terminationGracePeriodSeconds to 40 in the pod spec. Five seconds of preStop sleep, plus up to 25 seconds of connection draining, plus a small buffer before the SIGKILL. The math needs to work or you're back to hard kills.

Zero dropped requests

After the fix, we ran a load test during deploys. Zero 502s across 40 consecutive deployments. The Slack channel got quieter.

But what stuck with me about this engagement wasn't the technical fix. It was a three-line change plus some YAML. The thing that stuck was how long the problem had existed.

The SIGTERM handler had been in the codebase for two years. It was written during the initial Kubernetes migration by an engineer who googled "node.js graceful shutdown" and copied the first Stack Overflow answer. The answer had 847 upvotes. It was wrong — or rather, incomplete in a way that only matters under real concurrency and real network propagation delays.

Nobody ever tested it. Not manually, not in CI. The team had thorough integration tests for the API itself — request in, response out, validate the shape. They had load tests that ran monthly. But every test started the server and exercised it. No test ever stopped the server while it was handling requests.

This is the pattern I keep seeing. Teams test startup exhaustively. They have health checks and readiness probes and dependency validation on boot. They test the request path obsessively. But the shutdown path — the ten seconds where your application transitions from alive to dead — is a blind spot. Nobody writes tests for dying.

And it's not just a Kubernetes thing. The same class of bug shows up in Lambda cold starts, ECS task replacements, and VM scale-in events. Anywhere a process needs to finish what it's doing before the infrastructure reclaims it, there's a graceful shutdown handler that someone wrote once, never tested, and assumed was working.

Grep your codebase for SIGTERM. If you find a handler, ask yourself: when was the last time someone verified that it actually drains in-flight work? If the answer is "never," you might want to deploy during low traffic and watch closely. Or better yet, find out before a customer does.