We ran a data backfill on 140,000 subscription records while the app was live. Three weeks later, 1,800 customers had the wrong renewal date. The script looked fine. The bug was in the timing.
- Published On
Blog
Engineering case studies from the work I do: AI systems in production, and the unglamorous parts that decide whether they survive.
Drafted by an agent pipeline I built. The patterns are real, the incidents are constructed. How it works.
- On-call never shows up in exit interviews. But across half a dozen consulting engagements, I've watched it quietly hollow out engineering teams. The pattern is always the same, and it's always fixable.
- Published On
- Every 30 minutes, the auth server spiked to 100% CPU and shed requests. The pattern was so regular you could set a watch by it. Turned out every user's JWT expired at the same time.
- Published On
- Every deploy dropped a handful of requests. The service had a SIGTERM handler. It just didn't do what anyone thought it did.
- Published On
- A client's React class-to-hooks migration had been "90% complete" for fourteen months. The last 10% contained every hard problem the team had been avoiding. Gradual migrations don't fail at the start — they fail when nobody decides to finish.
- Published On
Tags
consulting (84)architecture (35)debugging (35)developer-experience (26)reliability (20)devops (14)ai (12)performance (11)code-quality (10)observability (9)security (7)engineering-culture (6)microservices (6)database (6)postgres (6)technical-debt (6)databases (6)kubernetes (5)testing (4)ci-cd (4)productivity (4)api-design (4)infrastructure (4)tooling (3)javascript (3)postgresql (3)nodejs (3)dependencies (3)code-review (2)cost-optimization (2)deployment (2)on-call (2)goals (2)monitoring (2)configuration (2)distributed-systems (2)resilience (2)refactoring (2)opentelemetry (2)workflow (1)api (1)contracts (1)cloud (1)typescript (1)caching (1)mdx (1)mongodb (1)migration (1)error-handling (1)feature-flags (1)alerting (1)queues (1)learning (1)journey (1)sonarcube (1)retention (1)orm (1)finops (1)containers (1)platform-engineering (1)incident-management (1)nestjs (1)angular (1)sandbox (1)environments (1)logging (1)secrets-management (1)data-integrity (1)github-actions (1)dns (1)node.js (1)mcp (1)memory (1)data (1)multi-tenant (1)authorization (1)terraform (1)authentication (1)webhooks (1)websockets (1)project-management (1)opinion (1)