The Query Plan That Changed Its Mind at 3AM
A routine ANALYZE flipped a Postgres query plan from an index scan to a sequential scan, and our API went from 12ms to 8 seconds. Here's what we learned about a failure mode most teams never think about.
Lukretium · Lab notes
Lab notes.
Case studies from the work: AI systems in production, and the unglamorous parts that decide whether they survive.
Drafted by an agent pipeline I built. The patterns are real, the incidents are constructed. How it works.
01 · Page 07 of 11
The Read Replica That Lied to Us
A client moved their reads to database replicas for performance. The latency numbers looked great — until customers started getting charged twice and inventory counts drifted from reality.
The API Key That Outlived Three Engineers
A client found one of their API keys in a public error log. Tracing where that key actually lived took longer than fixing the leak — and revealed a secrets management problem nobody wanted to own.
The Webhook That Silently Dropped Forty Thousand Events
A client's payment provider was sending webhook notifications correctly. Their system acknowledged every one. And then quietly threw most of them away.
The Fallback That Was Worse Than the Failure
A client's "graceful degradation" strategy silently served stale pricing data for 11 hours. The outage would have been better.
We Instrumented Our LLM Calls and Found the Budget Leak
A client's AI features were burning through their OpenAI budget 3x faster than projected. Adding OpenTelemetry's GenAI semantic conventions revealed the problem wasn't what anyone expected.
Your Healthcheck Endpoint Is Probably Lying
Most healthcheck endpoints return 200 OK as long as the process is running. That's not a healthcheck — it's a pulse check. Here's what happened when we confused the two, and what a real healthcheck should verify.
The Memory Limit We Copy-Pasted From Stack Overflow
A client's pods were getting OOMKilled during peak traffic, but the team spent days chasing application bugs. The real problem was resource limits that nobody had revisited since the initial cluster setup.
We Added Tracing and the Architecture Diagram Was Wrong
A client was confident about how their services talked to each other. Then we instrumented the system with OpenTelemetry and found out what was actually happening.
The Third-Party API That Went Slow, Not Down
A payment provider started responding in 8 seconds instead of 200ms. It wasn't an outage — their status page stayed green. But it took out our client's entire checkout flow because nobody had configured a timeout.