architecture

The app started with three roles. When I audited it, there were 94 permissions, 17 roles, and a function called checkAccess that was 400 lines of nested conditionals.

A client showed me their "tech debt backlog" with 214 items. Missing tests, outdated dependencies, bad naming, a database nobody understood, and an actual architectural shortcut from 2019. Lumping all of that into one bucket is why none of it gets fixed.

After parachuting into dozens of client projects, I've developed a mental checklist for the first sixty minutes. Most of what matters isn't in the code itself.

A customer clicked "Export to CSV" on 380,000 records. The server loaded them all into memory, OOMed, and took the API offline for every tenant. The fix was straightforward. The real question is why nobody caught it sooner.

Two teams, one PostgreSQL instance, zero ownership boundaries. A consulting story about what happens when microservices share a database — and how we eventually untangled it.

A client was six months into migrating from REST to GraphQL. Half their endpoints were converted, developer experience was arguably worse, and the frontend team was maintaining two data layers. Sometimes the right engineering decision is to stop.

Every 30 minutes, the auth server spiked to 100% CPU and shed requests. The pattern was so regular you could set a watch by it. Turned out every user's JWT expired at the same time.

A client's React class-to-hooks migration had been "90% complete" for fourteen months. The last 10% contained every hard problem the team had been avoiding. Gradual migrations don't fail at the start — they fail when nobody decides to finish.

A critical notification service was processing 50,000 messages a day. When it started silently dropping SMS, it took a week for anyone to notice — because the alerts were going to a Slack channel that belonged to a team that no longer existed.

A client's 45-minute data import gradually became a 14-hour ordeal that bled into business hours, tanked dashboard performance, and made the analytics team distrust their own numbers. The fix wasn't what anyone expected.

A client's real-time notification service kept crashing every few days. Restarts fixed it temporarily. The root cause was 14,000 zombie WebSocket connections from clients that had disconnected days ago.

A client had fourteen microservices, a 280-line Docker Compose file, and an wiki page last updated eight months ago. Their newest hire spent two days just trying to get the app running on her laptop.

We spent months building CI/CD pipelines, canary deployments, and automated rollbacks. Then someone changed a timeout value in an admin panel and took down the payment service for two hours.

A client's team built the core feature in five days. Then came the edge cases, the migration script, the security review, the load test that failed, and the staging bug that took longer to fix than the feature itself.

A client connected 11 MCP servers to their development environment in three months. When I reviewed the OAuth scopes, two of them had write access to production infrastructure. Nobody had checked.

A client's multi-tenant SaaS had menus, routes, and role checks — all in the browser. The API behind it would happily serve any tenant's data to anyone with a valid session token. It took eight months for someone to notice.

A client's internal utility package started as a convenience. Two years later, it had 47 dependents, undocumented side effects, and a change to one date formatter broke invoice generation for three services.

A client's engineering team adopted AI coding tools across the board. Individual velocity soared. But their DORA metrics flatlined — and the reasons had nothing to do with the tools themselves.

A client's API had rate limiting configured and enforced. It still couldn't prevent a single customer from tanking performance for everyone else. The problem wasn't the limiter — it was what we were counting.

A client's platform had 23 AI agents built by a team of 8. Nobody could tell me what half of them did. Agent sprawl is the new microservices sprawl, and the cleanup looks depressingly similar.

A client moved their reads to database replicas for performance. The latency numbers looked great — until customers started getting charged twice and inventory counts drifted from reality.

A client's payment provider was sending webhook notifications correctly. Their system acknowledged every one. And then quietly threw most of them away.

A client's "graceful degradation" strategy silently served stale pricing data for 11 hours. The outage would have been better.

A client was confident about how their services talked to each other. Then we instrumented the system with OpenTelemetry and found out what was actually happening.

A payment provider started responding in 8 seconds instead of 200ms. It wasn't an outage — their status page stayed green. But it took out our client's entire checkout flow because nobody had configured a timeout.

A startup founder built their MVP almost entirely with AI coding agents. It worked. Then they hired a team, and within two months nobody could ship anything. I got called in to figure out why.

A consulting engagement where we finally opened the cloud bill and found forgotten dev environments, runaway log storage, and a data pipeline reprocessing everything from scratch every night.

A 20-person team was running 14 microservices with three full-time engineers just keeping the infrastructure alive. We consolidated six of them into a modular monolith and cut their deploy time by 70%.

We ran load tests before a big product launch, got green across the board, and watched the system buckle under real traffic two days later. The tests weren't wrong — they just weren't testing reality.

A single slow database query triggered aggressive retries across four microservices. Within minutes, the entire order pipeline was down. Here's how we traced it and what we changed.

We added Redis to fix slow API responses. Instead we got stale data, thundering herds, and a system that was harder to debug than the original problem.

A consulting story about a platform engineering initiative that checked every box on paper but collected dust in practice — and the uncomfortable reasons why.

A consulting story about a minor field rename in an internal API that cascaded into a production incident, and what we put in place to stop it from happening again.

A consulting war story about a PostgreSQL-to-MongoDB migration that went sideways, what we missed in planning, and the uncomfortable lessons about knowing when not to migrate.

A journey through implementing efficient sandbox environments using PostgreSQL schemas, exploring different approaches and their trade-offs.