debugging

A customer clicked "Export to CSV" on 380,000 records. The server loaded them all into memory, OOMed, and took the API offline for every tenant. The fix was straightforward. The real question is why nobody caught it sooner.

Our monitoring said everything was fine. Customers said it wasn't. Turns out, the API returned HTTP 200 for every response — including the ones that failed. The dashboards were blind.

A Renovate auto-merge bumped a transitive dependency from 6.2.1 to 6.2.2. The patch fixed rounding behavior. Our invoice calculations disagreed.

A client's checkout endpoint had idempotency keys. Customers still got double-charged during flash sales. The bug was a three-line race condition between SELECT and INSERT that took two weeks to find.

We ran a data backfill on 140,000 subscription records while the app was live. Three weeks later, 1,800 customers had the wrong renewal date. The script looked fine. The bug was in the timing.

Every 30 minutes, the auth server spiked to 100% CPU and shed requests. The pattern was so regular you could set a watch by it. Turned out every user's JWT expired at the same time.

Every deploy dropped a handful of requests. The service had a SIGTERM handler. It just didn't do what anyone thought it did.

A client's real-time notification service kept crashing every few days. Restarts fixed it temporarily. The root cause was 14,000 zombie WebSocket connections from clients that had disconnected days ago.

A client's Datadog bill quietly grew to $38K/month — nearly double their actual compute spend. The culprit wasn't one big mistake but a dozen small defaults nobody revisited.

A client's codebase had impressive test coverage numbers. Then I introduced a deliberate bug and watched every single test pass. Coverage was measuring execution, not verification.

A SaaS billing system locked up every invoice cycle. The cause was two transactions grabbing the same rows in opposite order — a textbook deadlock hiding behind a once-a-month batch job.

A client's Terraform code said one thing. Production said another. It took a routine capacity change to reveal that 40% of their infrastructure had been modified by hand and nobody knew.

A Node.js service kept getting OOMKilled, but only on Tuesdays. The batch job was a red herring. The real problem was a lazy cache eviction strategy that nobody thought to question.

A client's deployment kept failing in staging but not locally. The root cause wasn't code — it was sixty-seven environment variables spread across five files with no documentation and no single source of truth.

A client's platform had 23 AI agents built by a team of 8. Nobody could tell me what half of them did. Agent sprawl is the new microservices sprawl, and the cleanup looks depressingly similar.

A client's codebase had try-catch blocks wrapped around everything. Nothing ever crashed. Nothing ever worked correctly either. The error handling strategy was actually an error hiding strategy.

Every deploy was losing a handful of HTTP requests, but nobody noticed until a payment callback disappeared. The fix wasn't in the deployment pipeline — it was in the application code that never learned how to shut down.

A routine ANALYZE flipped a Postgres query plan from an index scan to a sequential scan, and our API went from 12ms to 8 seconds. Here's what we learned about a failure mode most teams never think about.

A client moved their reads to database replicas for performance. The latency numbers looked great — until customers started getting charged twice and inventory counts drifted from reality.

A client found one of their API keys in a public error log. Tracing where that key actually lived took longer than fixing the leak — and revealed a secrets management problem nobody wanted to own.

A client's payment provider was sending webhook notifications correctly. Their system acknowledged every one. And then quietly threw most of them away.

A client's "graceful degradation" strategy silently served stale pricing data for 11 hours. The outage would have been better.

A client's AI features were burning through their OpenAI budget 3x faster than projected. Adding OpenTelemetry's GenAI semantic conventions revealed the problem wasn't what anyone expected.

A client's pods were getting OOMKilled during peak traffic, but the team spent days chasing application bugs. The real problem was resource limits that nobody had revisited since the initial cluster setup.

A client was confident about how their services talked to each other. Then we instrumented the system with OpenTelemetry and found out what was actually happening.

A payment provider started responding in 8 seconds instead of 200ms. It wasn't an outage — their status page stayed green. But it took out our client's entire checkout flow because nobody had configured a timeout.

A client had six monitoring tools and still couldn't diagnose a production incident in under an hour. The problem wasn't the tools — it was what happens when observability grows by accretion instead of design.

A client's PostgreSQL writes were getting slower every quarter. The table had 57 indexes. Only 14 of them were ever used. Every INSERT and UPDATE was paying a tax nobody had thought to audit.

A client's API was getting measurably slower every week. The dashboards were green, the alerts were silent, and the database looked healthy. The problem was hiding in plain sight — on the container's local disk.

A client's API started throwing 500s every weekday afternoon like clockwork. The database was fine. The queries were fast. The problem was a reporting job that quietly hogged every available connection during peak traffic.

A client's notification queue was draining normally and all dashboards showed green. But three weeks of transactional emails had vanished into a catch block nobody thought to monitor.

A Node.js service was writing UTC timestamps to a PostgreSQL database configured for Europe/Berlin. Nobody noticed the mismatch until a DST transition made an entire hour of orders vanish from daily reports.

A client's dashboard took 11 seconds to render. Everyone blamed the database. The real problem was an ORM doing exactly what we told it to — we just never looked at what that meant.

A consulting story about a nightly billing job that quietly started double-charging customers after a Kubernetes migration — and the boring lock that finally fixed it.

A debugging deep dive into replacing wall-of-text logs with structured logging and trace IDs — and how it cut our mean time to resolution from hours to minutes.