A client's Terraform code said one thing. Production said another. It took a routine capacity change to reveal that 40% of their infrastructure had been modified by hand and nobody knew.
- Published On
infrastructure
- A "30-minute" DNS provider switch turned into a 14-hour partial outage. Stale TTLs, an undocumented CNAME chain, and monitoring that only checked from the US — all the ingredients for a problem nobody could see.
- Published On
- Most healthcheck endpoints return 200 OK as long as the process is running. That's not a healthcheck — it's a pulse check. Here's what happened when we confused the two, and what a real healthcheck should verify.
- Published On
Tags
consulting (56)debugging (24)developer-experience (19)architecture (18)reliability (16)devops (12)ai (10)performance (9)observability (7)code-quality (6)database (5)productivity (4)technical-debt (4)kubernetes (4)postgres (4)infrastructure (3)tooling (3)security (3)microservices (3)testing (3)engineering-culture (3)nodejs (2)code-review (2)deployment (2)databases (2)opentelemetry (2)monitoring (2)postgresql (2)ci-cd (2)javascript (2)goals (2)terraform (1)dns (1)opinion (1)api-design (1)memory (1)configuration (1)error-handling (1)secrets-management (1)webhooks (1)resilience (1)containers (1)environments (1)queues (1)cloud (1)cost-optimization (1)orm (1)refactoring (1)on-call (1)alerting (1)incident-management (1)typescript (1)distributed-systems (1)caching (1)platform-engineering (1)dependencies (1)feature-flags (1)api (1)contracts (1)logging (1)mongodb (1)migration (1)workflow (1)nestjs (1)angular (1)sandbox (1)sonarcube (1)learning (1)journey (1)mdx (1)