consulting

A client's codebase had try-catch blocks wrapped around everything. Nothing ever crashed. Nothing ever worked correctly either. The error handling strategy was actually an error hiding strategy.

Every deploy was losing a handful of HTTP requests, but nobody noticed until a payment callback disappeared. The fix wasn't in the deployment pipeline — it was in the application code that never learned how to shut down.

A routine ANALYZE flipped a Postgres query plan from an index scan to a sequential scan, and our API went from 12ms to 8 seconds. Here's what we learned about a failure mode most teams never think about.

A client moved their reads to database replicas for performance. The latency numbers looked great — until customers started getting charged twice and inventory counts drifted from reality.

A client found one of their API keys in a public error log. Tracing where that key actually lived took longer than fixing the leak — and revealed a secrets management problem nobody wanted to own.

A client's payment provider was sending webhook notifications correctly. Their system acknowledged every one. And then quietly threw most of them away.

A client's "graceful degradation" strategy silently served stale pricing data for 11 hours. The outage would have been better.

A client's AI features were burning through their OpenAI budget 3x faster than projected. Adding OpenTelemetry's GenAI semantic conventions revealed the problem wasn't what anyone expected.

Most healthcheck endpoints return 200 OK as long as the process is running. That's not a healthcheck — it's a pulse check. Here's what happened when we confused the two, and what a real healthcheck should verify.

A client's pods were getting OOMKilled during peak traffic, but the team spent days chasing application bugs. The real problem was resource limits that nobody had revisited since the initial cluster setup.

A client was confident about how their services talked to each other. Then we instrumented the system with OpenTelemetry and found out what was actually happening.

A payment provider started responding in 8 seconds instead of 200ms. It wasn't an outage — their status page stayed green. But it took out our client's entire checkout flow because nobody had configured a timeout.

A client had six monitoring tools and still couldn't diagnose a production incident in under an hour. The problem wasn't the tools — it was what happens when observability grows by accretion instead of design.

A startup founder built their MVP almost entirely with AI coding agents. It worked. Then they hired a team, and within two months nobody could ship anything. I got called in to figure out why.

A client's PostgreSQL writes were getting slower every quarter. The table had 57 indexes. Only 14 of them were ever used. Every INSERT and UPDATE was paying a tax nobody had thought to audit.

A client's API was getting measurably slower every week. The dashboards were green, the alerts were silent, and the database looked healthy. The problem was hiding in plain sight — on the container's local disk.

A client's staging environment had drifted so far from production that developers stopped using it. Tests passed in staging and failed in prod. Tests failed in staging and passed in prod. Eventually the team just stopped looking.

A client's API started throwing 500s every weekday afternoon like clockwork. The database was fine. The queries were fast. The problem was a reporting job that quietly hogged every available connection during peak traffic.

A client's notification queue was draining normally and all dashboards showed green. But three weeks of transactional emails had vanished into a catch block nobody thought to monitor.

A consulting engagement where we finally opened the cloud bill and found forgotten dev environments, runaway log storage, and a data pipeline reprocessing everything from scratch every night.

A Node.js service was writing UTC timestamps to a PostgreSQL database configured for Europe/Berlin. Nobody noticed the mismatch until a DST transition made an entire hour of orders vanish from daily reports.

A client's dashboard took 11 seconds to render. Everyone blamed the database. The real problem was an ORM doing exactly what we told it to — we just never looked at what that meant.

A 20-person team was running 14 microservices with three full-time engineers just keeping the infrastructure alive. We consolidated six of them into a modular monolith and cut their deploy time by 70%.

A client's team was drowning in a 200K-line TypeScript monolith. When we finally measured what was actually running in production, we found that almost a third of it was dead code nobody had touched in over a year.

We ran load tests before a big product launch, got green across the board, and watched the system buckle under real traffic two days later. The tests weren't wrong — they just weren't testing reality.

A client's on-call rotation was burning people out — not because of real incidents, but because of noise. Here's how we cut alerts by 87% without missing anything that actually required a human.

A single slow database query triggered aggressive retries across four microservices. Within minutes, the entire order pipeline was down. Here's how we traced it and what we changed.

A client's team tripled their PR output with coding agents. Six weeks later, their cycle time had doubled and nobody could explain why.

A 47-minute build pipeline had become sacred infrastructure. When we finally opened it up, we found cargo-culted steps, redundant checks, and a team afraid of their own tooling.

We ran blameless postmortems after every incident. We wrote detailed action items with owners and deadlines. Six months later, 78% of those items were still open — and the same incidents kept happening.

A tool comparison from a consulting engagement — what Biome actually delivered, where it fell short, and the decision I'd make again (and the one I wouldn't).

A consulting story about a nightly billing job that quietly started double-charging customers after a Kubernetes migration — and the boring lock that finally fixed it.

We added Redis to fix slow API responses. Instead we got stale data, thundering herds, and a system that was harder to debug than the original problem.

A consulting story about a platform engineering initiative that checked every box on paper but collected dust in practice — and the uncomfortable reasons why.

The Axios supply chain attack reminded me of a dependency audit I ran at a client last year. What I found was worse than any vulnerability scanner could flag.

We built a canary deployment pipeline with automated rollbacks. It still let a bad release through to 100% of users. Here's what went wrong.

AI code review tools are everywhere now. After rolling one out on a real project, I learned more about our team than about the tool itself.

How a 45-minute end-to-end test suite trained an entire team to ignore test failures — and what we did about it.

How 340 feature flags turned a codebase into a minefield — and the boring process that dug us out.

A consulting story about a minor field rename in an internal API that cascaded into a production incident, and what we put in place to stop it from happening again.

The latest DORA report confirms what I've seen on consulting engagements — AI tools amplify existing conditions. If your processes are solid, AI accelerates you. If they're not, you just create technical debt faster.

A consulting engagement where we measured where engineering hours actually went — and discovered the biggest productivity killer wasn't what anyone expected.

A debugging deep dive into replacing wall-of-text logs with structured logging and trace IDs — and how it cut our mean time to resolution from hours to minutes.

A consulting war story about a PostgreSQL-to-MongoDB migration that went sideways, what we missed in planning, and the uncomfortable lessons about knowing when not to migrate.

Practical insights from months of using AI-assisted coding in real client projects — what works, what doesn't, and how it changed my workflow.