Six Dashboards, Zero Answers
A client had six monitoring tools and still couldn't diagnose a production incident in under an hour. The problem wasn't the tools — it was what happens when observability grows by accretion instead of design.
Lukretium · Lab notes
Lab notes.
Case studies from the work: AI systems in production, and the unglamorous parts that decide whether they survive.
Drafted by an agent pipeline I built. The patterns are real, the incidents are constructed. How it works.
01 · Page 08 of 11
I Audited a Codebase That Was Vibe-Coded in Two Weeks
A startup founder built their MVP almost entirely with AI coding agents. It worked. Then they hired a team, and within two months nobody could ship anything. I got called in to figure out why.
We Dropped 43 Indexes and Our Writes Got Twice as Fast
A client's PostgreSQL writes were getting slower every quarter. The table had 57 indexes. Only 14 of them were ever used. Every INSERT and UPDATE was paying a tax nobody had thought to audit.
Our P99 Latency Doubled in Three Months and Nobody Noticed
A client's API was getting measurably slower every week. The dashboards were green, the alerts were silent, and the database looked healthy. The problem was hiding in plain sight — on the container's local disk.
The Staging Environment Nobody Trusted (So Everyone Tested in Production)
A client's staging environment had drifted so far from production that developers stopped using it. Tests passed in staging and failed in prod. Tests failed in staging and passed in prod. Eventually the team just stopped looking.
The Connection Pool That Starved at 3 PM Every Day
A client's API started throwing 500s every weekday afternoon like clockwork. The database was fine. The queries were fast. The problem was a reporting job that quietly hogged every available connection during peak traffic.
The Job Queue That Silently Ate 12,000 Emails
A client's notification queue was draining normally and all dashboards showed green. But three weeks of transactional emails had vanished into a catch block nobody thought to monitor.
Our AWS Bill Went Up 40% and Nobody Noticed for Three Months
A consulting engagement where we finally opened the cloud bill and found forgotten dev environments, runaway log storage, and a data pipeline reprocessing everything from scratch every night.
The Timezone Bug That Quietly Ate Three Weeks of Revenue Data
A Node.js service was writing UTC timestamps to a PostgreSQL database configured for Europe/Berlin. Nobody noticed the mismatch until a DST transition made an entire hour of orders vanish from daily reports.
The ORM Was Running 847 Queries Per Page Load
A client's dashboard took 11 seconds to render. Everyone blamed the database. The real problem was an ORM doing exactly what we told it to — we just never looked at what that meant.