We Let AI Review Every Pull Request for Three Months. Here's What It Actually Caught.
AI code review tools are everywhere now. After rolling one out on a real project, I learned more about our team than about the tool itself.
Lab notes
103 notes
#consulting.
What I Actually Learned Using AI as a Pair Programming Partner
Practical insights from months of using AI-assisted coding in real client projects — what works, what doesn't, and how it changed my workflow.
AI Tooling Won't Fix Your Broken Engineering Culture
The latest DORA report confirms what I've seen on consulting engagements — AI tools amplify existing conditions. If your processes are solid, AI accelerates you. If they're not, you just create technical debt faster.
The "Harmless" API Change That Broke Four Services
A consulting story about a minor field rename in an internal API that cascaded into a production incident, and what we put in place to stop it from happening again.
Our AWS Bill Went Up 40% and Nobody Noticed for Three Months
A consulting engagement where we finally opened the cloud bill and found forgotten dev environments, runaway log storage, and a data pipeline reprocessing everything from scratch every night.
I Replaced ESLint and Prettier with Biome on a Client Project. Here's the Honest Report.
A tool comparison from a consulting engagement — what Biome actually delivered, where it fell short, and the decision I'd make again (and the one I wouldn't).
The Caching Layer That Made Everything Worse
We added Redis to fix slow API responses. Instead we got stale data, thundering herds, and a system that was harder to debug than the original problem.
The Canary That Didn't Sing — What Our Deployment Strategy Missed
We built a canary deployment pipeline with automated rollbacks. It still let a bad release through to 100% of users. Here's what went wrong.
The CI Pipeline Nobody Was Allowed to Touch
A 47-minute build pipeline had become sacred infrastructure. When we finally opened it up, we found cargo-culted steps, redundant checks, and a team afraid of their own tooling.
The Connection Pool That Starved at 3 PM Every Day
A client's API started throwing 500s every weekday afternoon like clockwork. The database was fine. The queries were fast. The problem was a reporting job that quietly hogged every available connection during peak traffic.
The Database Migration That Ate Our Sprint
A consulting war story about a PostgreSQL-to-MongoDB migration that went sideways, what we missed in planning, and the uncomfortable lessons about knowing when not to migrate.
We Tracked Developer Time for Two Weeks. The Results Were Uncomfortable.
A consulting engagement where we measured where engineering hours actually went — and discovered the biggest productivity killer wasn't what anyone expected.
The E2E Test Suite That Cried Wolf
How a 45-minute end-to-end test suite trained an entire team to ignore test failures — and what we did about it.
Every Exception Was Caught. No Error Was Handled.
A client's codebase had try-catch blocks wrapped around everything. Nothing ever crashed. Nothing ever worked correctly either. The error handling strategy was actually an error hiding strategy.
The Feature Flag Graveyard We Built Over Two Years
How 340 feature flags turned a codebase into a minefield — and the boring process that dug us out.
We Dropped 43 Indexes and Our Writes Got Twice as Fast
A client's PostgreSQL writes were getting slower every quarter. The table had 57 indexes. Only 14 of them were ever used. Every INSERT and UPDATE was paying a tax nobody had thought to audit.
We Had 400 Alerts a Week and Maybe 12 of Them Mattered
A client's on-call rotation was burning people out — not because of real incidents, but because of noise. Here's how we cut alerts by 87% without missing anything that actually required a human.
I Audited a Codebase That Was Vibe-Coded in Two Weeks
A startup founder built their MVP almost entirely with AI coding agents. It worked. Then they hired a team, and within two months nobody could ship anything. I got called in to figure out why.
The Job Queue That Silently Ate 12,000 Emails
A client's notification queue was draining normally and all dashboards showed green. But three weeks of transactional emails had vanished into a catch block nobody thought to monitor.
The Load Test That Told Us Exactly What We Wanted to Hear
We ran load tests before a big product launch, got green across the board, and watched the system buckle under real traffic two days later. The tests weren't wrong — they just weren't testing reality.
The Memory Limit We Copy-Pasted From Stack Overflow
A client's pods were getting OOMKilled during peak traffic, but the team spent days chasing application bugs. The real problem was resource limits that nobody had revisited since the initial cluster setup.
More PRs, Slower Delivery: The AI Agent Throughput Trap
A client's team tripled their PR output with coding agents. Six weeks later, their cycle time had doubled and nobody could explain why.
Nobody Quits Over On-Call. Except They Do.
On-call never shows up in exit interviews. But across half a dozen consulting engagements, I've watched it quietly hollow out engineering teams. The pattern is always the same, and it's always fixable.
The ORM Was Running 847 Queries Per Page Load
A client's dashboard took 11 seconds to render. Everyone blamed the database. The real problem was an ORM doing exactly what we told it to — we just never looked at what that meant.
Our Monitoring Cost More Than What It Monitored
A client's Datadog bill quietly grew to $38K/month — nearly double their actual compute spend. The culprit wasn't one big mistake but a dozen small defaults nobody revisited.
Our P99 Latency Doubled in Three Months and Nobody Noticed
A client's API was getting measurably slower every week. The dashboards were green, the alerts were silent, and the database looked healthy. The problem was hiding in plain sight — on the container's local disk.
We Built an Internal Developer Platform and Nobody Used It
A consulting story about a platform engineering initiative that checked every box on paper but collected dust in practice — and the uncomfortable reasons why.
The Postmortems We Wrote But Never Acted On
We ran blameless postmortems after every incident. We wrote detailed action items with owners and deadlines. Six months later, 78% of those items were still open — and the same incidents kept happening.
The Retry Storm That Took Down Three Services
A single slow database query triggered aggressive retries across four microservices. Within minutes, the entire order pipeline was down. Here's how we traced it and what we changed.
Reviewing AI Code Is the New Bottleneck
Last quarter I tracked my time across three consulting engagements. I spent 11 hours a week reviewing AI-generated code and under 8 writing my own. The ratio surprised me, and it changed how I think about developer productivity.
Six Dashboards, Zero Answers
A client had six monitoring tools and still couldn't diagnose a production incident in under an hour. The problem wasn't the tools — it was what happens when observability grows by accretion instead of design.
The Sixty-Seven Environment Variables Nobody Documented
A client's deployment kept failing in staging but not locally. The root cause wasn't code — it was sixty-seven environment variables spread across five files with no documentation and no single source of truth.
The Staging Environment Nobody Trusted (So Everyone Tested in Production)
A client's staging environment had drifted so far from production that developers stopped using it. Tests passed in staging and failed in prod. Tests failed in staging and passed in prod. Eventually the team just stopped looking.
Stop Calling Everything Technical Debt
A client showed me their "tech debt backlog" with 214 items. Missing tests, outdated dependencies, bad naming, a database nobody understood, and an actual architectural shortcut from 2019. Lumping all of that into one bucket is why none of it gets fixed.
How Structured Logging Turned Our 2AM Pages Into 15-Minute Fixes
A debugging deep dive into replacing wall-of-text logs with structured logging and trace IDs — and how it cut our mean time to resolution from hours to minutes.
The AI Coding Bill Nobody Budgeted For
GitHub Copilot moved to usage-based billing and a client's engineering budget lost all predictability overnight. Developer tools now bill like cloud infrastructure, and most teams aren't ready for it.
The ALTER TABLE That Locked Production for Nine Seconds
A one-line schema migration brought the checkout page to a halt during peak traffic. The column addition was instant. The backfill running in the same transaction was not.
The API Key That Outlived Three Engineers
A client found one of their API keys in a public error log. Tracing where that key actually lived took longer than fixing the leak — and revealed a secrets management problem nobody wanted to own.
The API With a Zero Percent Error Rate and a Fifteen Percent Failure Rate
Our monitoring said everything was fine. Customers said it wasn't. Turns out, the API returned HTTP 200 for every response — including the ones that failed. The dashboards were blind.
The Auth Logic That Lived in Twelve Different Places
A client asked me to review their microservices security. I found JWT validation in some services, session cookies in others, one service with no auth at all, and an API gateway that covered barely half the traffic.
The Autoscaler That Made the Outage Worse
During a traffic spike, our autoscaler saw dropping CPU and scaled down. The database was the actual bottleneck. Fewer instances meant more connections per pod, which hit the connection limit and turned a slow page into a full outage.
The Backfill That Raced Against Live Traffic
We ran a data backfill on 140,000 subscription records while the app was live. Three weeks later, 1,800 customers had the wrong renewal date. The script looked fine. The bug was in the timing.
The Backup That Worked Until We Tried to Restore It
Two years of green checkmarks on the nightly backup job. Then we actually needed to restore data, and discovered the backups were missing half the schema.
The Branch That Lived for Four Months
A client's "big rewrite" feature branch had 847 commits, was 14,000 lines behind main, and three engineers were trapped on it. The merge took a week. It didn't have to be this way.
The Codebase That Slowly Stopped Agreeing With Itself
A client's AI-assisted codebase passed every lint check and had solid test coverage. But under the surface, modules had drifted into contradictory patterns — three retry strategies, four error-handling philosophies, and a data validation layer that depended on which AI session wrote the code.
The Coding Agent That Could Push to Main
A client's AI coding agent had contents:write on every repo in the org. When I asked who approved that, the answer was "the setup wizard." The defaults are the danger zone.
The Config Change That Bypassed Every Safety Net
We spent months building CI/CD pipelines, canary deployments, and automated rollbacks. Then someone changed a timeout value in an admin panel and took down the payment service for two hours.
The Cron Job That Ran Twice (And Charged Everyone Twice Too)
A consulting story about a nightly billing job that quietly started double-charging customers after a Kubernetes migration — and the boring lock that finally fixed it.
The CSV Export That Took Down the API
A customer clicked "Export to CSV" on 380,000 records. The server loaded them all into memory, OOMed, and took the API offline for every tenant. The fix was straightforward. The real question is why nobody caught it sooner.
The Database Password We Rotated Into an Outage
A security audit said rotate your credentials. We did. Three services silently lost database access for 45 minutes because they baked the connection string into the container image at build time and the health checks never noticed.
The Database Two Services Shared Until They Couldn't
Two teams, one PostgreSQL instance, zero ownership boundaries. A consulting story about what happens when microservices share a database — and how we eventually untangled it.
The Day GitHub Actions Went Down and Nobody Could Deploy
GitHub Actions was out for nearly eleven hours on August 6. On the client project I was embedded in, it took about forty minutes for us to realize we had no Plan B. We couldn't deploy, couldn't run tests, couldn't merge with confidence. A ten-hour lesson in single points of failure.
The Deadlock That Only Happened During the Monthly Invoice Run
A SaaS billing system locked up every invoice cycle. The cause was two transactions grabbing the same rows in opposite order — a textbook deadlock hiding behind a once-a-month batch job.
The Deploy That Dropped Requests in Silence
Every deploy was losing a handful of HTTP requests, but nobody noticed until a payment callback disappeared. The fix wasn't in the deployment pipeline — it was in the application code that never learned how to shut down.
The DNS Migration We Should Have Tested
A "30-minute" DNS provider switch turned into a 14-hour partial outage. Stale TTLs, an undocumented CNAME chain, and monitoring that only checked from the US — all the ingredients for a problem nobody could see.
The Dockerfile Nobody Had Touched in Two Years
A client's Docker build took 18 minutes. Developers had stopped building locally and were pushing to CI for every change. The fix took an afternoon, but the real cost was the two years of compounded slowness nobody measured.
The Fallback That Was Worse Than the Failure
A client's "graceful degradation" strategy silently served stale pricing data for 11 hours. The outage would have been better.
The Graceful Shutdown That Wasn't
Every deploy dropped a handful of requests. The service had a SIGTERM handler. It just didn't do what anyone thought it did.
The GraphQL Migration We Quietly Abandoned
A client was six months into migrating from REST to GraphQL. Half their endpoints were converted, developer experience was arguably worse, and the frontend team was maintaining two data layers. Sometimes the right engineering decision is to stop.
The Internal API That Had External Users
A client asked me to refactor their order management API. Nobody mentioned the twelve companies that had been calling it from outside the network for two years.
The Internal Certificate Nobody Was Watching
An internal root CA certificate expired on a Saturday afternoon. Mutual TLS failed across nine services, and the team spent four hours figuring out why everything could reach the internet but nothing could talk to itself.
The MCP Servers Nobody Audited
A client connected 11 MCP servers to their development environment in three months. When I reviewed the OAuth scopes, two of them had write access to production infrastructure. Nobody had checked.
The Memory Leak That Only Showed Up on Tuesdays
A Node.js service kept getting OOMKilled, but only on Tuesdays. The batch job was a red herring. The real problem was a lazy cache eviction strategy that nobody thought to question.
The Migration That Was Always Almost Done
A client's React class-to-hooks migration had been "90% complete" for fourteen months. The last 10% contained every hard problem the team had been avoiding. Gradual migrations don't fail at the start — they fail when nobody decides to finish.
The Nightly Import That Started Taking All Day
A client's 45-minute data import gradually became a 14-hour ordeal that bled into business hours, tanked dashboard performance, and made the analytics team distrust their own numbers. The fix wasn't what anyone expected.
The Node.js Upgrade We Put Off Until a CVE Forced Our Hand
A client's API had been running Node 16 for two years past end-of-life. When a critical OpenSSL vulnerability dropped, the "we'll upgrade next quarter" plan collapsed into a three-week fire drill. The upgrade itself wasn't hard. Undoing two years of drift was.
The Paginated API That Lost Records Between Pages
A data sync was silently dropping customer records. The offset-based pagination looked correct in every test. It just couldn't survive a production database that kept changing underneath it.
The Patch Bump That Fixed a Bug and Broke Our Invoices
A Renovate auto-merge bumped a transitive dependency from 6.2.1 to 6.2.2. The patch fixed rounding behavior. Our invoice calculations disagreed.
The Payment API That Charged Customers Twice
A client's checkout endpoint had idempotency keys. Customers still got double-charged during flash sales. The bug was a three-line race condition between SELECT and INSERT that took two weeks to find.
The Permission Model That Grew Until Nobody Understood It
The app started with three roles. When I audited it, there were 94 permissions, 17 roles, and a function called checkAccess that was 400 lines of nested conditionals.
The Query Plan That Changed Its Mind at 3AM
A routine ANALYZE flipped a Postgres query plan from an index scan to a sequential scan, and our API went from 12ms to 8 seconds. Here's what we learned about a failure mode most teams never think about.
The Rate Limiter That Let Everything Through
A client's API had rate limiting configured and enforced. It still couldn't prevent a single customer from tanking performance for everyone else. The problem wasn't the limiter — it was what we were counting.
The Read Replica That Lied to Us
A client moved their reads to database replicas for performance. The latency numbers looked great — until customers started getting charged twice and inventory counts drifted from reality.
The Rollback That Didn't Roll Back
We practiced deployments religiously but never tested a rollback. When a release broke checkout and we hit the big red button, we found out half the system couldn't actually go backward.
The Search Index That Drifted From the Database
A client's product search showed deleted items, missed new ones, and displayed yesterday's prices. The Elasticsearch index had been quietly diverging from PostgreSQL for months.
The Security Scanner We Trained Ourselves to Ignore
A client added a SAST scanner to their CI pipeline after a compliance audit. Six months later, nobody looked at the findings. Then a real SQL injection shipped to production.
The Service Nobody Owned Until It Broke
A critical notification service was processing 50,000 messages a day. When it started silently dropping SMS, it took a week for anyone to notice — because the alerts were going to a Slack channel that belonged to a team that no longer existed.
The Shared Package Five Teams Were Afraid to Touch
A client's internal utility package started as a convenience. Two years later, it had 47 dependents, undocumented side effects, and a change to one date formatter broke invoice generation for three services.
The Slack Bot That Became Load-Bearing Infrastructure
A team built a quick Slack bot for deploy notifications. Eighteen months later, it ran deploys, managed on-call rotation, provisioned environments, and tracked incidents. It had no tests, no docs, and ran on a single EC2 instance nobody monitored.
The Soft Delete That Deleted Nothing
A client implemented soft deletes across their entire data model for compliance reasons. Six months later, a GDPR request revealed they hadn't actually deleted anything — and the workarounds had introduced three new bugs.
The Stored Procedure Nobody Knew Existed
A client's discount calculations kept applying rules that weren't in the application code. After two days of searching, I found them — in a PostgreSQL trigger written by someone who left the company three years ago.
The Team That Had More AI Agents Than Engineers
A client's platform had 23 AI agents built by a team of 8. Nobody could tell me what half of them did. Agent sprawl is the new microservices sprawl, and the cleanup looks depressingly similar.
The Tenant Boundary That Only Existed in the Frontend
A client's multi-tenant SaaS had menus, routes, and role checks — all in the browser. The API behind it would happily serve any tenant's data to anyone with a valid session token. It took eight months for someone to notice.
The Terraform State That Drifted for Six Months
A client's Terraform code said one thing. Production said another. It took a routine capacity change to reveal that 40% of their infrastructure had been modified by hand and nobody knew.
The Test Suite With 90% Coverage That Caught Zero Bugs
A client's codebase had impressive test coverage numbers. Then I introduced a deliberate bug and watched every single test pass. Coverage was measuring execution, not verification.
The Token Refresh That Turned Into a Thundering Herd
Every 30 minutes, the auth server spiked to 100% CPU and shed requests. The pattern was so regular you could set a watch by it. Turned out every user's JWT expired at the same time.
The Webhook That Silently Dropped Forty Thousand Events
A client's payment provider was sending webhook notifications correctly. Their system acknowledged every one. And then quietly threw most of them away.
The WebSocket Connections That Refused to Die
A client's real-time notification service kept crashing every few days. Restarts fixed it temporarily. The root cause was 14,000 zombie WebSocket connections from clients that had disconnected days ago.
The Third-Party API That Went Slow, Not Down
A payment provider started responding in 8 seconds instead of 200ms. It wasn't an outage — their status page stayed green. But it took out our client's entire checkout flow because nobody had configured a timeout.
Thirty Seconds Is Not a Reasonable Timeout
The most common timeout value I see in production codebases is 30 seconds. It's almost never the right choice. Here's why most timeout configs silently make outages worse, and what to set instead.
The Timezone Bug That Quietly Ate Three Weeks of Revenue Data
A Node.js service was writing UTC timestamps to a PostgreSQL database configured for Europe/Berlin. Nobody noticed the mismatch until a DST transition made an entire hour of orders vanish from daily reports.
It Took a Senior Engineer Two Days to Run Our App Locally
A client had fourteen microservices, a 280-line Docker Compose file, and an wiki page last updated eight months ago. Their newest hire spent two days just trying to get the app running on her laptop.
We Added Tracing and the Architecture Diagram Was Wrong
A client was confident about how their services talked to each other. Then we instrumented the system with OpenTelemetry and found out what was actually happening.
We Deleted 30% of the Codebase and Nothing Broke
A client's team was drowning in a 200K-line TypeScript monolith. When we finally measured what was actually running in production, we found that almost a third of it was dead code nobody had touched in over a year.
We Finished the Feature in a Week. It Shipped Three Months Later.
A client's team built the core feature in five days. Then came the edge cases, the migration script, the security review, the load test that failed, and the staging bug that took longer to fix than the feature itself.
We Instrumented Our LLM Calls and Found the Budget Leak
A client's AI features were burning through their OpenAI budget 3x faster than projected. Adding OpenTelemetry's GenAI semantic conventions revealed the problem wasn't what anyone expected.
We Merged Six Microservices Back Into a Monolith
A 20-person team was running 14 microservices with three full-time engineers just keeping the infrastructure alive. We consolidated six of them into a modular monolith and cut their deploy time by 70%.
We Tripled Code Output. Delivery Speed Didn't Budge.
A client's engineering team adopted AI coding tools across the board. Individual velocity soared. But their DORA metrics flatlined — and the reasons had nothing to do with the tools themselves.
We Upgraded to TypeScript 7 and the Build Got Weird
The native Go compiler promised 10x faster builds. We got that, plus a week of chasing down every tool in our pipeline that assumed tsc was slow.
What I Look for in the First Hour of a New Codebase
After parachuting into dozens of client projects, I've developed a mental checklist for the first sixty minutes. Most of what matters isn't in the code itself.
Your AI Coding Tool Just Got Acquired. Now What?
The SpaceX-Cursor deal forced a lot of teams to rethink their AI tooling strategy overnight. But the real problem isn't the acquisition — it's that most teams never had a strategy to begin with.
Your Dependencies Are Someone Else's Keys to Your Server
The Axios supply chain attack reminded me of a dependency audit I ran at a client last year. What I found was worse than any vulnerability scanner could flag.
Your Healthcheck Endpoint Is Probably Lying
Most healthcheck endpoints return 200 OK as long as the process is running. That's not a healthcheck — it's a pulse check. Here's what happened when we confused the two, and what a real healthcheck should verify.