The Token Refresh That Turned Into a Thundering Herd

Every 30 minutes, the auth server spiked to 100% CPU and shed requests. The pattern was so regular you could set a watch by it. Turned out every user's JWT expired at the same time.

The Grafana dashboard told the whole story in a single chart. Every 30 minutes, almost to the second, the auth service spiked to 100% CPU, request latency shot past 8 seconds, and about 15% of token refresh requests failed outright. Then everything settled back to normal. Half an hour later, it happened again.

The client — a B2B SaaS company with about 12,000 concurrent users during business hours — had been living with this for three weeks. They'd already tried the obvious fix: scaling up the auth service from 3 pods to 8. That moved the failure rate from 40% down to 15%, but the spikes were still there, and throwing more hardware at the problem was costing them an extra $2,800 a month in compute.

Following the spike

I started where I usually start: the logs. The auth service was a Node.js app backed by Redis for session storage and PostgreSQL for user records. During the spikes, I could see thousands of /token/refresh requests arriving within a 2-3 second window. Not a gradual ramp — a wall of traffic.

The JWT configuration was straightforward. Access tokens had a 30-minute TTL. Refresh tokens lasted 7 days. The client-side code checked the access token's exp claim before every API call and, if it was expired or within 30 seconds of expiry, called the refresh endpoint.

Here's what the frontend was doing:

async function getAccessToken(): Promise<string> {
  const token = localStorage.getItem('access_token');
  if (token) {
    const payload = JSON.parse(atob(token.split('.')[1]));
    const expiresIn = payload.exp * 1000 - Date.now();
    if (expiresIn > 30_000) return token;
  }
  return refreshAccessToken();
}

Clean code. Correct logic. And the direct cause of a thundering herd.

Why everything expired at once

The deployment pipeline told the rest of the story. Every deploy invalidated all existing sessions — a security-conscious choice, but one with a side effect. After a Tuesday morning deploy at 9:15 AM, every active user got a new access token within a few minutes as they interacted with the app. All those tokens had the same 30-minute TTL. So at 9:45, all of them expired. At 10:15, again. At 10:45, again.

The effect persisted even hours after the deploy because each successful refresh issued a new token with — you guessed it — exactly 30 minutes of life. The herd never spread out. It just marched in lockstep, every half hour, indefinitely.

The fix was jitter, not scale

The solution had two parts, and neither involved adding more pods.

First, we added jitter to the token TTL on the server side. Instead of a flat 30 minutes, each token got a TTL between 25 and 35 minutes. That alone was enough to spread the refresh wave across a 10-minute window instead of a 3-second spike.

function generateAccessToken(userId: string): string {
  const jitterSeconds = Math.floor(Math.random() * 600) - 300;
  const ttl = 1800 + jitterSeconds; // 25-35 minutes
  return jwt.sign({ sub: userId }, SECRET, { expiresIn: ttl });
}

Second, we changed the client-side refresh logic to add its own randomized buffer. Instead of refreshing when the token was within 30 seconds of expiry, each client picked a random threshold between 30 and 180 seconds. If two users had tokens expiring at the exact same second, one might refresh 45 seconds early and the other 2 minutes early.

const REFRESH_BUFFER = 30_000 + Math.random() * 150_000;
 
async function getAccessToken(): Promise<string> {
  const token = localStorage.getItem('access_token');
  if (token) {
    const payload = JSON.parse(atob(token.split('.')[1]));
    const expiresIn = payload.exp * 1000 - Date.now();
    if (expiresIn > REFRESH_BUFFER) return token;
  }
  return refreshAccessToken();
}

After deploying both changes, the next morning's spike looked like a gentle hill instead of a cliff. Peak CPU went from 100% to 35%. The failure rate dropped to zero. We scaled back down to 4 pods and the auth service barely noticed.

The pattern shows up everywhere

This wasn't a novel discovery. Thundering herds from synchronized TTLs are a well-documented distributed systems problem. Cache expiration, DNS TTL renewal, certificate rotation, cron jobs — any system where a bunch of things expire at the same time will produce the same traffic spike. The solution is always some variant of jitter.

But knowing the pattern and recognizing it in your own system are different things. The client's team had three senior engineers who could have explained thundering herds on a whiteboard. None of them connected it to the auth spikes because the symptoms looked like a capacity problem, not a synchronization problem. They were scaling horizontally when they needed to scatter temporally.

The most expensive debugging sessions I've been in share this trait: the team reaches for the infrastructure dial when the real issue is in the timing.