Lukretium · Lab notes
Case studies from the work: AI systems in production, and the unglamorous parts that decide whether they survive.
Drafted by an agent pipeline I built. The patterns are real, the incidents are constructed. How it works.
01 · Page 02 of 11
An internal root CA certificate expired on a Saturday afternoon. Mutual TLS failed across nine services, and the team spent four hours figuring out why everything could reach the internet but nothing could talk to itself.
A client implemented soft deletes across their entire data model for compliance reasons. Six months later, a GDPR request revealed they hadn't actually deleted anything — and the workarounds had introduced three new bugs.
Two years of green checkmarks on the nightly backup job. Then we actually needed to restore data, and discovered the backups were missing half the schema.
A client's discount calculations kept applying rules that weren't in the application code. After two days of searching, I found them — in a PostgreSQL trigger written by someone who left the company three years ago.
The native Go compiler promised 10x faster builds. We got that, plus a week of chasing down every tool in our pipeline that assumed tsc was slow.
A client's AI-assisted codebase passed every lint check and had solid test coverage. But under the surface, modules had drifted into contradictory patterns — three retry strategies, four error-handling philosophies, and a data validation layer that depended on which AI session wrote the code.
A team built a quick Slack bot for deploy notifications. Eighteen months later, it ran deploys, managed on-call rotation, provisioned environments, and tracked incidents. It had no tests, no docs, and ran on a single EC2 instance nobody monitored.
The most common timeout value I see in production codebases is 30 seconds. It's almost never the right choice. Here's why most timeout configs silently make outages worse, and what to set instead.
A client's product search showed deleted items, missed new ones, and displayed yesterday's prices. The Elasticsearch index had been quietly diverging from PostgreSQL for months.
The app started with three roles. When I audited it, there were 94 permissions, 17 roles, and a function called checkAccess that was 400 lines of nested conditionals.