The ALTER TABLE That Locked Production for Nine Seconds
A one-line schema migration brought the checkout page to a halt during peak traffic. The column addition was instant. The backfill running in the same transaction was not.
Lab notes
12 notes
#databases.
The Backfill That Raced Against Live Traffic
We ran a data backfill on 140,000 subscription records while the app was live. Three weeks later, 1,800 customers had the wrong renewal date. The script looked fine. The bug was in the timing.
The Backup That Worked Until We Tried to Restore It
Two years of green checkmarks on the nightly backup job. Then we actually needed to restore data, and discovered the backups were missing half the schema.
The Database Two Services Shared Until They Couldn't
Two teams, one PostgreSQL instance, zero ownership boundaries. A consulting story about what happens when microservices share a database — and how we eventually untangled it.
The Deadlock That Only Happened During the Monthly Invoice Run
A SaaS billing system locked up every invoice cycle. The cause was two transactions grabbing the same rows in opposite order — a textbook deadlock hiding behind a once-a-month batch job.
The Paginated API That Lost Records Between Pages
A data sync was silently dropping customer records. The offset-based pagination looked correct in every test. It just couldn't survive a production database that kept changing underneath it.
The Payment API That Charged Customers Twice
A client's checkout endpoint had idempotency keys. Customers still got double-charged during flash sales. The bug was a three-line race condition between SELECT and INSERT that took two weeks to find.
The Query Plan That Changed Its Mind at 3AM
A routine ANALYZE flipped a Postgres query plan from an index scan to a sequential scan, and our API went from 12ms to 8 seconds. Here's what we learned about a failure mode most teams never think about.
The Read Replica That Lied to Us
A client moved their reads to database replicas for performance. The latency numbers looked great — until customers started getting charged twice and inventory counts drifted from reality.
The Search Index That Drifted From the Database
A client's product search showed deleted items, missed new ones, and displayed yesterday's prices. The Elasticsearch index had been quietly diverging from PostgreSQL for months.
The Soft Delete That Deleted Nothing
A client implemented soft deletes across their entire data model for compliance reasons. Six months later, a GDPR request revealed they hadn't actually deleted anything — and the workarounds had introduced three new bugs.
The Stored Procedure Nobody Knew Existed
A client's discount calculations kept applying rules that weren't in the application code. After two days of searching, I found them — in a PostgreSQL trigger written by someone who left the company three years ago.