consulting

A client had fourteen microservices, a 280-line Docker Compose file, and an wiki page last updated eight months ago. Their newest hire spent two days just trying to get the app running on her laptop.

We spent months building CI/CD pipelines, canary deployments, and automated rollbacks. Then someone changed a timeout value in an admin panel and took down the payment service for two hours.

GitHub Actions was out for nearly eleven hours on August 6. On the client project I was embedded in, it took about forty minutes for us to realize we had no Plan B. We couldn't deploy, couldn't run tests, couldn't merge with confidence. A ten-hour lesson in single points of failure.

A client's team built the core feature in five days. Then came the edge cases, the migration script, the security review, the load test that failed, and the staging bug that took longer to fix than the feature itself.

A client connected 11 MCP servers to their development environment in three months. When I reviewed the OAuth scopes, two of them had write access to production infrastructure. Nobody had checked.

A client's Datadog bill quietly grew to $38K/month — nearly double their actual compute spend. The culprit wasn't one big mistake but a dozen small defaults nobody revisited.

A client's codebase had impressive test coverage numbers. Then I introduced a deliberate bug and watched every single test pass. Coverage was measuring execution, not verification.

A client's multi-tenant SaaS had menus, routes, and role checks — all in the browser. The API behind it would happily serve any tenant's data to anyone with a valid session token. It took eight months for someone to notice.

A client's internal utility package started as a convenience. Two years later, it had 47 dependents, undocumented side effects, and a change to one date formatter broke invoice generation for three services.

A SaaS billing system locked up every invoice cycle. The cause was two transactions grabbing the same rows in opposite order — a textbook deadlock hiding behind a once-a-month batch job.

A client's Terraform code said one thing. Production said another. It took a routine capacity change to reveal that 40% of their infrastructure had been modified by hand and nobody knew.

A "30-minute" DNS provider switch turned into a 14-hour partial outage. Stale TTLs, an undocumented CNAME chain, and monitoring that only checked from the US — all the ingredients for a problem nobody could see.

A client's engineering team adopted AI coding tools across the board. Individual velocity soared. But their DORA metrics flatlined — and the reasons had nothing to do with the tools themselves.

The SpaceX-Cursor deal forced a lot of teams to rethink their AI tooling strategy overnight. But the real problem isn't the acquisition — it's that most teams never had a strategy to begin with.

A client's API had been running Node 16 for two years past end-of-life. When a critical OpenSSL vulnerability dropped, the "we'll upgrade next quarter" plan collapsed into a three-week fire drill. The upgrade itself wasn't hard. Undoing two years of drift was.

A client's API had rate limiting configured and enforced. It still couldn't prevent a single customer from tanking performance for everyone else. The problem wasn't the limiter — it was what we were counting.

Last quarter I tracked my time across three consulting engagements. I spent 11 hours a week reviewing AI-generated code and under 8 writing my own. The ratio surprised me, and it changed how I think about developer productivity.

A Node.js service kept getting OOMKilled, but only on Tuesdays. The batch job was a red herring. The real problem was a lazy cache eviction strategy that nobody thought to question.

A client's deployment kept failing in staging but not locally. The root cause wasn't code — it was sixty-seven environment variables spread across five files with no documentation and no single source of truth.

We practiced deployments religiously but never tested a rollback. When a release broke checkout and we hit the big red button, we found out half the system couldn't actually go backward.

A client's platform had 23 AI agents built by a team of 8. Nobody could tell me what half of them did. Agent sprawl is the new microservices sprawl, and the cleanup looks depressingly similar.

A client's codebase had try-catch blocks wrapped around everything. Nothing ever crashed. Nothing ever worked correctly either. The error handling strategy was actually an error hiding strategy.

Every deploy was losing a handful of HTTP requests, but nobody noticed until a payment callback disappeared. The fix wasn't in the deployment pipeline — it was in the application code that never learned how to shut down.

A routine ANALYZE flipped a Postgres query plan from an index scan to a sequential scan, and our API went from 12ms to 8 seconds. Here's what we learned about a failure mode most teams never think about.

A client moved their reads to database replicas for performance. The latency numbers looked great — until customers started getting charged twice and inventory counts drifted from reality.

A client found one of their API keys in a public error log. Tracing where that key actually lived took longer than fixing the leak — and revealed a secrets management problem nobody wanted to own.

A client's payment provider was sending webhook notifications correctly. Their system acknowledged every one. And then quietly threw most of them away.

A client's "graceful degradation" strategy silently served stale pricing data for 11 hours. The outage would have been better.

A client's AI features were burning through their OpenAI budget 3x faster than projected. Adding OpenTelemetry's GenAI semantic conventions revealed the problem wasn't what anyone expected.

Most healthcheck endpoints return 200 OK as long as the process is running. That's not a healthcheck — it's a pulse check. Here's what happened when we confused the two, and what a real healthcheck should verify.

A client's pods were getting OOMKilled during peak traffic, but the team spent days chasing application bugs. The real problem was resource limits that nobody had revisited since the initial cluster setup.

A client was confident about how their services talked to each other. Then we instrumented the system with OpenTelemetry and found out what was actually happening.

A payment provider started responding in 8 seconds instead of 200ms. It wasn't an outage — their status page stayed green. But it took out our client's entire checkout flow because nobody had configured a timeout.

A client had six monitoring tools and still couldn't diagnose a production incident in under an hour. The problem wasn't the tools — it was what happens when observability grows by accretion instead of design.

A startup founder built their MVP almost entirely with AI coding agents. It worked. Then they hired a team, and within two months nobody could ship anything. I got called in to figure out why.

A client's PostgreSQL writes were getting slower every quarter. The table had 57 indexes. Only 14 of them were ever used. Every INSERT and UPDATE was paying a tax nobody had thought to audit.

A client's API was getting measurably slower every week. The dashboards were green, the alerts were silent, and the database looked healthy. The problem was hiding in plain sight — on the container's local disk.

A client's staging environment had drifted so far from production that developers stopped using it. Tests passed in staging and failed in prod. Tests failed in staging and passed in prod. Eventually the team just stopped looking.

A client's API started throwing 500s every weekday afternoon like clockwork. The database was fine. The queries were fast. The problem was a reporting job that quietly hogged every available connection during peak traffic.

A client's notification queue was draining normally and all dashboards showed green. But three weeks of transactional emails had vanished into a catch block nobody thought to monitor.

A consulting engagement where we finally opened the cloud bill and found forgotten dev environments, runaway log storage, and a data pipeline reprocessing everything from scratch every night.

A Node.js service was writing UTC timestamps to a PostgreSQL database configured for Europe/Berlin. Nobody noticed the mismatch until a DST transition made an entire hour of orders vanish from daily reports.

A client's dashboard took 11 seconds to render. Everyone blamed the database. The real problem was an ORM doing exactly what we told it to — we just never looked at what that meant.

A 20-person team was running 14 microservices with three full-time engineers just keeping the infrastructure alive. We consolidated six of them into a modular monolith and cut their deploy time by 70%.

A client's team was drowning in a 200K-line TypeScript monolith. When we finally measured what was actually running in production, we found that almost a third of it was dead code nobody had touched in over a year.

We ran load tests before a big product launch, got green across the board, and watched the system buckle under real traffic two days later. The tests weren't wrong — they just weren't testing reality.

A client's on-call rotation was burning people out — not because of real incidents, but because of noise. Here's how we cut alerts by 87% without missing anything that actually required a human.

A single slow database query triggered aggressive retries across four microservices. Within minutes, the entire order pipeline was down. Here's how we traced it and what we changed.

A client's team tripled their PR output with coding agents. Six weeks later, their cycle time had doubled and nobody could explain why.

A 47-minute build pipeline had become sacred infrastructure. When we finally opened it up, we found cargo-culted steps, redundant checks, and a team afraid of their own tooling.

We ran blameless postmortems after every incident. We wrote detailed action items with owners and deadlines. Six months later, 78% of those items were still open — and the same incidents kept happening.

A tool comparison from a consulting engagement — what Biome actually delivered, where it fell short, and the decision I'd make again (and the one I wouldn't).

A consulting story about a nightly billing job that quietly started double-charging customers after a Kubernetes migration — and the boring lock that finally fixed it.

We added Redis to fix slow API responses. Instead we got stale data, thundering herds, and a system that was harder to debug than the original problem.

A consulting story about a platform engineering initiative that checked every box on paper but collected dust in practice — and the uncomfortable reasons why.

The Axios supply chain attack reminded me of a dependency audit I ran at a client last year. What I found was worse than any vulnerability scanner could flag.

We built a canary deployment pipeline with automated rollbacks. It still let a bad release through to 100% of users. Here's what went wrong.

AI code review tools are everywhere now. After rolling one out on a real project, I learned more about our team than about the tool itself.

How a 45-minute end-to-end test suite trained an entire team to ignore test failures — and what we did about it.

How 340 feature flags turned a codebase into a minefield — and the boring process that dug us out.

A consulting story about a minor field rename in an internal API that cascaded into a production incident, and what we put in place to stop it from happening again.

The latest DORA report confirms what I've seen on consulting engagements — AI tools amplify existing conditions. If your processes are solid, AI accelerates you. If they're not, you just create technical debt faster.

A consulting engagement where we measured where engineering hours actually went — and discovered the biggest productivity killer wasn't what anyone expected.

A debugging deep dive into replacing wall-of-text logs with structured logging and trace IDs — and how it cut our mean time to resolution from hours to minutes.

A consulting war story about a PostgreSQL-to-MongoDB migration that went sideways, what we missed in planning, and the uncomfortable lessons about knowing when not to migrate.

Practical insights from months of using AI-assisted coding in real client projects — what works, what doesn't, and how it changed my workflow.