reliability

A client's staging environment had drifted so far from production that developers stopped using it. Tests passed in staging and failed in prod. Tests failed in staging and passed in prod. Eventually the team just stopped looking.

A client's API started throwing 500s every weekday afternoon like clockwork. The database was fine. The queries were fast. The problem was a reporting job that quietly hogged every available connection during peak traffic.

A client's notification queue was draining normally and all dashboards showed green. But three weeks of transactional emails had vanished into a catch block nobody thought to monitor.

A Node.js service was writing UTC timestamps to a PostgreSQL database configured for Europe/Berlin. Nobody noticed the mismatch until a DST transition made an entire hour of orders vanish from daily reports.

A single slow database query triggered aggressive retries across four microservices. Within minutes, the entire order pipeline was down. Here's how we traced it and what we changed.

We ran blameless postmortems after every incident. We wrote detailed action items with owners and deadlines. Six months later, 78% of those items were still open — and the same incidents kept happening.

A consulting story about a nightly billing job that quietly started double-charging customers after a Kubernetes migration — and the boring lock that finally fixed it.