The Canary That Didn't Sing — What Our Deployment Strategy Missed
We built a canary deployment pipeline with automated rollbacks. It still let a bad release through to 100% of users. Here's what went wrong.
Lab notes
9 notes
#observability.
We Had 400 Alerts a Week and Maybe 12 of Them Mattered
A client's on-call rotation was burning people out — not because of real incidents, but because of noise. Here's how we cut alerts by 87% without missing anything that actually required a human.
The Job Queue That Silently Ate 12,000 Emails
A client's notification queue was draining normally and all dashboards showed green. But three weeks of transactional emails had vanished into a catch block nobody thought to monitor.
Our Monitoring Cost More Than What It Monitored
A client's Datadog bill quietly grew to $38K/month — nearly double their actual compute spend. The culprit wasn't one big mistake but a dozen small defaults nobody revisited.
Six Dashboards, Zero Answers
A client had six monitoring tools and still couldn't diagnose a production incident in under an hour. The problem wasn't the tools — it was what happens when observability grows by accretion instead of design.
How Structured Logging Turned Our 2AM Pages Into 15-Minute Fixes
A debugging deep dive into replacing wall-of-text logs with structured logging and trace IDs — and how it cut our mean time to resolution from hours to minutes.
The API With a Zero Percent Error Rate and a Fifteen Percent Failure Rate
Our monitoring said everything was fine. Customers said it wasn't. Turns out, the API returned HTTP 200 for every response — including the ones that failed. The dashboards were blind.
We Added Tracing and the Architecture Diagram Was Wrong
A client was confident about how their services talked to each other. Then we instrumented the system with OpenTelemetry and found out what was actually happening.
We Instrumented Our LLM Calls and Found the Budget Leak
A client's AI features were burning through their OpenAI budget 3x faster than projected. Adding OpenTelemetry's GenAI semantic conventions revealed the problem wasn't what anyone expected.