Nobody Quits Over On-Call. Except They Do.
On-call never shows up in exit interviews. But across half a dozen consulting engagements, I've watched it quietly hollow out engineering teams. The pattern is always the same, and it's always fixable.
Lukretium · Lab notes
Lab notes.
Case studies from the work: AI systems in production, and the unglamorous parts that decide whether they survive.
Drafted by an agent pipeline I built. The patterns are real, the incidents are constructed. How it works.
01 · Page 04 of 11
The Token Refresh That Turned Into a Thundering Herd
Every 30 minutes, the auth server spiked to 100% CPU and shed requests. The pattern was so regular you could set a watch by it. Turned out every user's JWT expired at the same time.
The Graceful Shutdown That Wasn't
Every deploy dropped a handful of requests. The service had a SIGTERM handler. It just didn't do what anyone thought it did.
The Migration That Was Always Almost Done
A client's React class-to-hooks migration had been "90% complete" for fourteen months. The last 10% contained every hard problem the team had been avoiding. Gradual migrations don't fail at the start — they fail when nobody decides to finish.
The Service Nobody Owned Until It Broke
A critical notification service was processing 50,000 messages a day. When it started silently dropping SMS, it took a week for anyone to notice — because the alerts were going to a Slack channel that belonged to a team that no longer existed.
The Nightly Import That Started Taking All Day
A client's 45-minute data import gradually became a 14-hour ordeal that bled into business hours, tanked dashboard performance, and made the analytics team distrust their own numbers. The fix wasn't what anyone expected.
The WebSocket Connections That Refused to Die
A client's real-time notification service kept crashing every few days. Restarts fixed it temporarily. The root cause was 14,000 zombie WebSocket connections from clients that had disconnected days ago.
It Took a Senior Engineer Two Days to Run Our App Locally
A client had fourteen microservices, a 280-line Docker Compose file, and an wiki page last updated eight months ago. Their newest hire spent two days just trying to get the app running on her laptop.
The Config Change That Bypassed Every Safety Net
We spent months building CI/CD pipelines, canary deployments, and automated rollbacks. Then someone changed a timeout value in an admin panel and took down the payment service for two hours.
The Day GitHub Actions Went Down and Nobody Could Deploy
GitHub Actions was out for nearly eleven hours on August 6. On the client project I was embedded in, it took about forty minutes for us to realize we had no Plan B. We couldn't deploy, couldn't run tests, couldn't merge with confidence. A ten-hour lesson in single points of failure.