A client's real-time notification service kept crashing every few days. Restarts fixed it temporarily. The root cause was 14,000 zombie WebSocket connections from clients that had disconnected days ago.
- Published On
Hello, I'm Luke
Personal blog about my experiences and thoughts on software development.
Latest Posts
- A client had fourteen microservices, a 280-line Docker Compose file, and an wiki page last updated eight months ago. Their newest hire spent two days just trying to get the app running on her laptop.
- Published On
- We spent months building CI/CD pipelines, canary deployments, and automated rollbacks. Then someone changed a timeout value in an admin panel and took down the payment service for two hours.
- Published On
- GitHub Actions was out for nearly eleven hours on August 6. On the client project I was embedded in, it took about forty minutes for us to realize we had no Plan B. We couldn't deploy, couldn't run tests, couldn't merge with confidence. A ten-hour lesson in single points of failure.
- Published On
- A client's team built the core feature in five days. Then came the edge cases, the migration script, the security review, the load test that failed, and the staging bug that took longer to fix than the feature itself.
- Published On