A SaaS billing system locked up every invoice cycle. The cause was two transactions grabbing the same rows in opposite order — a textbook deadlock hiding behind a once-a-month batch job.
A routine ANALYZE flipped a Postgres query plan from an index scan to a sequential scan, and our API went from 12ms to 8 seconds. Here's what we learned about a failure mode most teams never think about.
A client moved their reads to database replicas for performance. The latency numbers looked great — until customers started getting charged twice and inventory counts drifted from reality.