The "Harmless" API Change That Broke Four Services
A consulting story about a minor field rename in an internal API that cascaded into a production incident, and what we put in place to stop it from happening again.
Lab notes
44 notes
#architecture.
Our AWS Bill Went Up 40% and Nobody Noticed for Three Months
A consulting engagement where we finally opened the cloud bill and found forgotten dev environments, runaway log storage, and a data pipeline reprocessing everything from scratch every night.
The Caching Layer That Made Everything Worse
We added Redis to fix slow API responses. Instead we got stale data, thundering herds, and a system that was harder to debug than the original problem.
The Database Migration That Ate Our Sprint
A consulting war story about a PostgreSQL-to-MongoDB migration that went sideways, what we missed in planning, and the uncomfortable lessons about knowing when not to migrate.
I Audited a Codebase That Was Vibe-Coded in Two Weeks
A startup founder built their MVP almost entirely with AI coding agents. It worked. Then they hired a team, and within two months nobody could ship anything. I got called in to figure out why.
The Load Test That Told Us Exactly What We Wanted to Hear
We ran load tests before a big product launch, got green across the board, and watched the system buckle under real traffic two days later. The tests weren't wrong — they just weren't testing reality.
We Built an Internal Developer Platform and Nobody Used It
A consulting story about a platform engineering initiative that checked every box on paper but collected dust in practice — and the uncomfortable reasons why.
The Retry Storm That Took Down Three Services
A single slow database query triggered aggressive retries across four microservices. Within minutes, the entire order pipeline was down. Here's how we traced it and what we changed.
Implementing Sandbox Environments with PostgreSQL Schemas
A journey through implementing efficient sandbox environments using PostgreSQL schemas, exploring different approaches and their trade-offs.
Stop Calling Everything Technical Debt
A client showed me their "tech debt backlog" with 214 items. Missing tests, outdated dependencies, bad naming, a database nobody understood, and an actual architectural shortcut from 2019. Lumping all of that into one bucket is why none of it gets fixed.
The Auth Logic That Lived in Twelve Different Places
A client asked me to review their microservices security. I found JWT validation in some services, session cookies in others, one service with no auth at all, and an API gateway that covered barely half the traffic.
The Branch That Lived for Four Months
A client's "big rewrite" feature branch had 847 commits, was 14,000 lines behind main, and three engineers were trapped on it. The merge took a week. It didn't have to be this way.
The Codebase That Slowly Stopped Agreeing With Itself
A client's AI-assisted codebase passed every lint check and had solid test coverage. But under the surface, modules had drifted into contradictory patterns — three retry strategies, four error-handling philosophies, and a data validation layer that depended on which AI session wrote the code.
The Config Change That Bypassed Every Safety Net
We spent months building CI/CD pipelines, canary deployments, and automated rollbacks. Then someone changed a timeout value in an admin panel and took down the payment service for two hours.
The CSV Export That Took Down the API
A customer clicked "Export to CSV" on 380,000 records. The server loaded them all into memory, OOMed, and took the API offline for every tenant. The fix was straightforward. The real question is why nobody caught it sooner.
The Database Two Services Shared Until They Couldn't
Two teams, one PostgreSQL instance, zero ownership boundaries. A consulting story about what happens when microservices share a database — and how we eventually untangled it.
The Fallback That Was Worse Than the Failure
A client's "graceful degradation" strategy silently served stale pricing data for 11 hours. The outage would have been better.
The GraphQL Migration We Quietly Abandoned
A client was six months into migrating from REST to GraphQL. Half their endpoints were converted, developer experience was arguably worse, and the frontend team was maintaining two data layers. Sometimes the right engineering decision is to stop.
The Internal API That Had External Users
A client asked me to refactor their order management API. Nobody mentioned the twelve companies that had been calling it from outside the network for two years.
The MCP Servers Nobody Audited
A client connected 11 MCP servers to their development environment in three months. When I reviewed the OAuth scopes, two of them had write access to production infrastructure. Nobody had checked.
The Migration That Was Always Almost Done
A client's React class-to-hooks migration had been "90% complete" for fourteen months. The last 10% contained every hard problem the team had been avoiding. Gradual migrations don't fail at the start — they fail when nobody decides to finish.
The Nightly Import That Started Taking All Day
A client's 45-minute data import gradually became a 14-hour ordeal that bled into business hours, tanked dashboard performance, and made the analytics team distrust their own numbers. The fix wasn't what anyone expected.
The Permission Model That Grew Until Nobody Understood It
The app started with three roles. When I audited it, there were 94 permissions, 17 roles, and a function called checkAccess that was 400 lines of nested conditionals.
The Rate Limiter That Let Everything Through
A client's API had rate limiting configured and enforced. It still couldn't prevent a single customer from tanking performance for everyone else. The problem wasn't the limiter — it was what we were counting.
The Read Replica That Lied to Us
A client moved their reads to database replicas for performance. The latency numbers looked great — until customers started getting charged twice and inventory counts drifted from reality.
The Search Index That Drifted From the Database
A client's product search showed deleted items, missed new ones, and displayed yesterday's prices. The Elasticsearch index had been quietly diverging from PostgreSQL for months.
The Service Nobody Owned Until It Broke
A critical notification service was processing 50,000 messages a day. When it started silently dropping SMS, it took a week for anyone to notice — because the alerts were going to a Slack channel that belonged to a team that no longer existed.
The Shared Package Five Teams Were Afraid to Touch
A client's internal utility package started as a convenience. Two years later, it had 47 dependents, undocumented side effects, and a change to one date formatter broke invoice generation for three services.
The Slack Bot That Became Load-Bearing Infrastructure
A team built a quick Slack bot for deploy notifications. Eighteen months later, it ran deploys, managed on-call rotation, provisioned environments, and tracked incidents. It had no tests, no docs, and ran on a single EC2 instance nobody monitored.
The Soft Delete That Deleted Nothing
A client implemented soft deletes across their entire data model for compliance reasons. Six months later, a GDPR request revealed they hadn't actually deleted anything — and the workarounds had introduced three new bugs.
The Stored Procedure Nobody Knew Existed
A client's discount calculations kept applying rules that weren't in the application code. After two days of searching, I found them — in a PostgreSQL trigger written by someone who left the company three years ago.
The Team That Had More AI Agents Than Engineers
A client's platform had 23 AI agents built by a team of 8. Nobody could tell me what half of them did. Agent sprawl is the new microservices sprawl, and the cleanup looks depressingly similar.
The Tenant Boundary That Only Existed in the Frontend
A client's multi-tenant SaaS had menus, routes, and role checks — all in the browser. The API behind it would happily serve any tenant's data to anyone with a valid session token. It took eight months for someone to notice.
The Token Refresh That Turned Into a Thundering Herd
Every 30 minutes, the auth server spiked to 100% CPU and shed requests. The pattern was so regular you could set a watch by it. Turned out every user's JWT expired at the same time.
The Webhook That Silently Dropped Forty Thousand Events
A client's payment provider was sending webhook notifications correctly. Their system acknowledged every one. And then quietly threw most of them away.
The WebSocket Connections That Refused to Die
A client's real-time notification service kept crashing every few days. Restarts fixed it temporarily. The root cause was 14,000 zombie WebSocket connections from clients that had disconnected days ago.
The Third-Party API That Went Slow, Not Down
A payment provider started responding in 8 seconds instead of 200ms. It wasn't an outage — their status page stayed green. But it took out our client's entire checkout flow because nobody had configured a timeout.
Thirty Seconds Is Not a Reasonable Timeout
The most common timeout value I see in production codebases is 30 seconds. It's almost never the right choice. Here's why most timeout configs silently make outages worse, and what to set instead.
It Took a Senior Engineer Two Days to Run Our App Locally
A client had fourteen microservices, a 280-line Docker Compose file, and an wiki page last updated eight months ago. Their newest hire spent two days just trying to get the app running on her laptop.
We Added Tracing and the Architecture Diagram Was Wrong
A client was confident about how their services talked to each other. Then we instrumented the system with OpenTelemetry and found out what was actually happening.
We Finished the Feature in a Week. It Shipped Three Months Later.
A client's team built the core feature in five days. Then came the edge cases, the migration script, the security review, the load test that failed, and the staging bug that took longer to fix than the feature itself.
We Merged Six Microservices Back Into a Monolith
A 20-person team was running 14 microservices with three full-time engineers just keeping the infrastructure alive. We consolidated six of them into a modular monolith and cut their deploy time by 70%.
We Tripled Code Output. Delivery Speed Didn't Budge.
A client's engineering team adopted AI coding tools across the board. Individual velocity soared. But their DORA metrics flatlined — and the reasons had nothing to do with the tools themselves.
What I Look for in the First Hour of a New Codebase
After parachuting into dozens of client projects, I've developed a mental checklist for the first sixty minutes. Most of what matters isn't in the code itself.