We spent months building CI/CD pipelines, canary deployments, and automated rollbacks. Then someone changed a timeout value in an admin panel and took down the payment service for two hours.
- Published On
Hello, I'm Luke
Personal blog about my experiences and thoughts on software development.
Latest Posts
- GitHub Actions was out for nearly eleven hours on August 6. On the client project I was embedded in, it took about forty minutes for us to realize we had no Plan B. We couldn't deploy, couldn't run tests, couldn't merge with confidence. A ten-hour lesson in single points of failure.
- Published On
- A client's team built the core feature in five days. Then came the edge cases, the migration script, the security review, the load test that failed, and the staging bug that took longer to fix than the feature itself.
- Published On
- A client connected 11 MCP servers to their development environment in three months. When I reviewed the OAuth scopes, two of them had write access to production infrastructure. Nobody had checked.
- Published On
- A client's Datadog bill quietly grew to $38K/month — nearly double their actual compute spend. The culprit wasn't one big mistake but a dozen small defaults nobody revisited.
- Published On