The Service That Needed a Restart Every 72 Hours

The runbook had a step that caught my eye: "If the notification service becomes unresponsive, restart the pod. This typically happens every 3-4 days."

I was two weeks into an engagement helping a fintech client prepare their platform for higher traffic. They expected a 3x increase in users over the next quarter following a partnership deal, and the CTO wanted an outside review of their infrastructure before the ramp. Most services looked fine. The notification service did not.

The band-aid nobody questioned

The service handled all outbound notifications — emails, SMS, push notifications, webhooks to partner systems. About 12,000 messages per hour at current load. It ran on Kubernetes with three replicas, each allocated 512MB of memory.

The pattern was predictable. After a fresh deploy or restart, each pod sat at around 180MB RSS. Over three days, that climbed steadily to 480MB, then 510MB, then the OOM killer did its thing. Kubernetes restarted the pod, memory dropped back to 180MB, and the cycle repeated.

The team's solution was a CronJob that bounced the pods every 72 hours, staggered so at least two replicas were always up. It had been running for five months. Nobody had filed a ticket to investigate the underlying leak because, in their words, "it works."

It did work. Until I pointed out that at 3x the current message volume, that 72-hour window would shrink to about 24 hours. And under spike conditions — a partner triggering a bulk webhook delivery — it could hit the limit within hours.

Hunting the leak

My first step was attaching to a staging replica and taking heap snapshots at intervals. Snapshot at startup. Another after simulating 10,000 notifications. Another after 50,000.

The comparison told the story immediately. Between the 10K and 50K snapshots, the number of EventEmitter objects had grown by roughly 40,000. They weren't being garbage collected.

I traced it to the service's Redis connection handler. Each incoming notification request called a function that subscribed to a Redis channel to wait for delivery confirmation:

async function waitForConfirmation(messageId) {
  const redis = getRedisClient();
 
  return new Promise((resolve, reject) => {
    const timeout = setTimeout(() => {
      reject(new Error('Confirmation timeout'));
    }, 30000);
 
    redis.on('message', (channel, data) => {
      const parsed = JSON.parse(data);
      if (parsed.messageId === messageId) {
        clearTimeout(timeout);
        resolve(parsed);
      }
    });
 
    redis.subscribe(`confirmations:${messageId}`);
  });
}

See the problem? Every call to waitForConfirmation added a new listener to the shared Redis client's message event. When the confirmation arrived or the timeout fired, the promise resolved or rejected — but the listener stayed attached. Forever.

At 12,000 messages per hour, that's 12,000 anonymous functions piling onto the same EventEmitter every hour. Each one held a closure over messageId, timeout, and parsed. After 72 hours, roughly 2.5 million orphaned listeners were sitting in memory doing nothing.

Node.js even warned about this. Buried in the logs were lines like MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 message listeners added. The warning fires at 11 listeners and then goes quiet. The team had seen it during development, googled it, and added redis.setMaxListeners(0) to suppress the warning.

The fix

Name the handler. Remove it when you're done. Unsubscribe from the channel.

async function waitForConfirmation(messageId) {
  const redis = getRedisClient();
 
  return new Promise((resolve, reject) => {
    const timeout = setTimeout(() => {
      redis.removeListener('message', handler);
      redis.unsubscribe(`confirmations:${messageId}`);
      reject(new Error('Confirmation timeout'));
    }, 30000);
 
    const handler = (channel, data) => {
      const parsed = JSON.parse(data);
      if (parsed.messageId === messageId) {
        clearTimeout(timeout);
        redis.removeListener('message', handler);
        redis.unsubscribe(`confirmations:${messageId}`);
        resolve(parsed);
      }
    };
 
    redis.on('message', handler);
    redis.subscribe(`confirmations:${messageId}`);
  });
}

After deploying the fix, memory held steady at 190MB under sustained load. The 72-hour restart CronJob got deleted. The setMaxListeners(0) override got deleted too.

What stayed with me

The leak itself was a well-documented antipattern. What I keep thinking about is the five months. Five months where a team of solid engineers lived with a workaround that masked a six-line bug. Not because they were lazy — the CronJob was actually well-engineered, with health checks and staggered restarts. They put more effort into the band-aid than the fix would have taken.

I see this pattern constantly. A workaround that's "good enough" absorbs the urgency. The original problem stops being a problem and becomes a background characteristic of the system. "Oh, that service just needs restarts" — the same way you'd say "oh, that door just sticks a bit."

What finally justified digging into it wasn't the leak itself. It was a change in load projections that made the workaround untenable. Without that business trigger, I suspect the CronJob would still be running today.

How many of your services have a quirk like that? A restart schedule, a manual cache clear, a "just retry it" step in the runbook? The question isn't whether the workaround works. It's whether it'll still work when the conditions change.