The Memory Leak That Only Showed Up on Tuesdays

A Node.js service kept getting OOMKilled, but only on Tuesdays. The batch job was a red herring. The real problem was a lazy cache eviction strategy that nobody thought to question.

The Grafana dashboard told a weird story. Monday: flat memory usage, hovering around 280MB. Tuesday: a slow, steady climb starting at 9 AM, crossing 512MB by lunch, OOMKilled by 2 PM. Wednesday through Friday: flat again. The following Tuesday: same thing.

I was two months into an engagement with a logistics company, helping them stabilize a Node.js API that powered their shipment tracking portal. The service had been running fine for almost a year. Then, about three weeks before I got involved, the Tuesday crashes started. The team had already burned a week trying to find the cause.

The obvious suspect

Tuesdays were when the finance team ran their weekly reconciliation. A separate batch process would pull shipment data for the previous week, cross-reference it with billing records, and generate a report. The batch job hit the same API as external clients.

The team assumed the batch job was the problem. They profiled it, reduced its concurrency from 20 parallel requests to 5, added pagination to limit response sizes. The next Tuesday, the API still crashed. Just a bit later — 3 PM instead of 2 PM.

So they tried running the batch job on a Wednesday as a test. Memory stayed flat. The API was fine.

That ruled out the batch job itself, but it also made everything more confusing. The crash was somehow tied to Tuesdays, yet the only thing that was different about Tuesdays was the batch job. Classic heisenbug territory.

Heap snapshots at the wrong time

The team had taken heap snapshots on Mondays and found nothing concerning. A few hundred cached objects, some buffers, nothing growing. I asked them to take one on a Tuesday morning, thirty minutes after the batch job started.

The difference was immediately obvious. The snapshot showed a Map object holding over 40,000 entries, consuming about 180MB. On Monday, the same Map had around 300 entries.

It was the response cache.

The cache that never forgot

The API used a homegrown caching layer — a plain Map wrapped in a helper class. Each cache entry had a TTL of 15 minutes. Straightforward enough. But the eviction logic was lazy: expired entries only got removed when someone tried to read them.

class ResponseCache {
  private store = new Map<string, CacheEntry>();
 
  get(key: string): CachedResponse | null {
    const entry = this.store.get(key);
    if (!entry) return null;
    if (Date.now() > entry.expiresAt) {
      this.store.delete(key);
      return null;
    }
    return entry.data;
  }
 
  set(key: string, data: CachedResponse, ttlMs = 900_000) {
    this.store.set(key, { data, expiresAt: Date.now() + ttlMs });
  }
}

On a normal day, the portal served a few hundred active users tracking shipments. The same shipment IDs got looked up over and over. Cache entries were written, read, and evicted in a natural cycle. The Map never grew beyond a few hundred entries because popular keys kept getting accessed, which triggered the expiration check.

The Tuesday batch job changed the access pattern completely. The reconciliation process queried every shipment from the past seven days — thousands of unique shipment-date combinations. Each query generated a cache entry with a unique key. But the batch job never read the same key twice. It fetched each record once and moved on.

So those entries sat in the Map, waiting for a read that would never come. The TTL was 15 minutes, but 15 minutes meant nothing when nobody was checking. The entries were technically expired, but the Map still held references to them. From V8's perspective, they were very much alive.

Why Wednesdays were fine

When the pod got OOMKilled on Tuesday afternoon and restarted, the cache was empty. Wednesday's batch job test ran against a fresh cache, and Wednesday's organic traffic patterns were the normal read-heavy profile. The entries from the batch job expired and got cleaned up by subsequent reads to nearby keys — or the cache just never grew large enough to matter because the batch only ran once and the natural traffic covered most of the same keys.

The bug needed two conditions: the cache had to already be warm from normal traffic, and then the batch job had to inject thousands of write-only entries on top of it. A cold cache could absorb the batch. A warm cache without the batch stayed small. Only the combination of both overflowed.

The fix

We considered a few options. A background sweep on a timer. A WeakRef-based approach. Switching to an LRU library.

We went with the simplest thing that solved the problem: a hard size cap on the Map.

set(key: string, data: CachedResponse, ttlMs = 900_000) {
  if (this.store.size >= this.maxSize) {
    const oldest = this.store.keys().next().value;
    this.store.delete(oldest);
  }
  this.store.set(key, { data, expiresAt: Date.now() + ttlMs });
}

We set maxSize to 1,000. Insertion order in a Map gives us a rough FIFO eviction, which was good enough. Memory flatlined at around 310MB on the next Tuesday. The batch job still ran. The cache still worked. The entries just fell off the end when the map got too big.

Total time from first crash to fix: three weeks. Lines of code changed: four.

The thing that sticks with me

The cache had been in production for eleven months without issues. It wasn't buggy — it worked exactly as designed. The design just had an implicit assumption baked into it: that read patterns would cover the keyspace. Nobody wrote that assumption down. Nobody thought to, because it was true right up until it wasn't.

The most durable bugs aren't the ones that fail obviously. They're the ones that work perfectly under the access patterns you tested and silently break under the patterns you didn't think to simulate.

I wonder how many other quiet assumptions are sitting in production right now, waiting for the right workload to disprove them.