Every 30 minutes, the auth server spiked to 100% CPU and shed requests. The pattern was so regular you could set a watch by it. Turned out every user's JWT expired at the same time.
- Published On
distributed systems
- A consulting story about a nightly billing job that quietly started double-charging customers after a Kubernetes migration — and the boring lock that finally fixed it.
- Published On
Tags
consulting (72)debugging (30)architecture (29)developer-experience (22)reliability (20)devops (14)ai (11)performance (10)observability (8)code-quality (7)technical-debt (6)database (6)kubernetes (5)engineering-culture (5)microservices (5)security (5)infrastructure (4)testing (4)productivity (4)postgres (4)ci-cd (3)databases (3)postgresql (3)tooling (3)distributed-systems (2)refactoring (2)configuration (2)resilience (2)cost-optimization (2)dependencies (2)nodejs (2)code-review (2)deployment (2)opentelemetry (2)monitoring (2)javascript (2)goals (2)authentication (1)node.js (1)data (1)websockets (1)project-management (1)mcp (1)finops (1)multi-tenant (1)authorization (1)terraform (1)dns (1)opinion (1)api-design (1)memory (1)error-handling (1)secrets-management (1)webhooks (1)containers (1)environments (1)queues (1)cloud (1)orm (1)on-call (1)alerting (1)incident-management (1)typescript (1)caching (1)platform-engineering (1)feature-flags (1)api (1)contracts (1)logging (1)mongodb (1)migration (1)workflow (1)nestjs (1)angular (1)sandbox (1)sonarcube (1)learning (1)journey (1)mdx (1)