6 notes
During a traffic spike, our autoscaler saw dropping CPU and scaled down. The database was the actual bottleneck. Fewer instances meant more connections per pod, which hit the connection limit and turned a slow page into a full outage.
A "30-minute" DNS provider switch turned into a 14-hour partial outage. Stale TTLs, an undocumented CNAME chain, and monitoring that only checked from the US — all the ingredients for a problem nobody could see.
Every deploy dropped a handful of requests. The service had a SIGTERM handler. It just didn't do what anyone thought it did.
An internal root CA certificate expired on a Saturday afternoon. Mutual TLS failed across nine services, and the team spent four hours figuring out why everything could reach the internet but nothing could talk to itself.
A client's Terraform code said one thing. Production said another. It took a routine capacity change to reveal that 40% of their infrastructure had been modified by hand and nobody knew.
Most healthcheck endpoints return 200 OK as long as the process is running. That's not a healthcheck — it's a pulse check. Here's what happened when we confused the two, and what a real healthcheck should verify.