The Database Migration That Ate Our Sprint
A consulting war story about a PostgreSQL-to-MongoDB migration that went sideways, what we missed in planning, and the uncomfortable lessons about knowing when not to migrate.
Lab notes
7 notes
#postgres.
Implementing Sandbox Environments with PostgreSQL Schemas
A journey through implementing efficient sandbox environments using PostgreSQL schemas, exploring different approaches and their trade-offs.
The ALTER TABLE That Locked Production for Nine Seconds
A one-line schema migration brought the checkout page to a halt during peak traffic. The column addition was instant. The backfill running in the same transaction was not.
The Backfill That Raced Against Live Traffic
We ran a data backfill on 140,000 subscription records while the app was live. Three weeks later, 1,800 customers had the wrong renewal date. The script looked fine. The bug was in the timing.
The Payment API That Charged Customers Twice
A client's checkout endpoint had idempotency keys. Customers still got double-charged during flash sales. The bug was a three-line race condition between SELECT and INSERT that took two weeks to find.
The Query Plan That Changed Its Mind at 3AM
A routine ANALYZE flipped a Postgres query plan from an index scan to a sequential scan, and our API went from 12ms to 8 seconds. Here's what we learned about a failure mode most teams never think about.
The Read Replica That Lied to Us
A client moved their reads to database replicas for performance. The latency numbers looked great — until customers started getting charged twice and inventory counts drifted from reality.