A client's "graceful degradation" strategy silently served stale pricing data for 11 hours. The outage would have been better.
- Published On
Blog
Engineering case studies from the work I do: AI systems in production, and the unglamorous parts that decide whether they survive.
Drafted by an agent pipeline I built. The patterns are real, the incidents are constructed. How it works.
- A client's AI features were burning through their OpenAI budget 3x faster than projected. Adding OpenTelemetry's GenAI semantic conventions revealed the problem wasn't what anyone expected.
- Published On
- Most healthcheck endpoints return 200 OK as long as the process is running. That's not a healthcheck — it's a pulse check. Here's what happened when we confused the two, and what a real healthcheck should verify.
- Published On
- A client's pods were getting OOMKilled during peak traffic, but the team spent days chasing application bugs. The real problem was resource limits that nobody had revisited since the initial cluster setup.
- Published On
- A client was confident about how their services talked to each other. Then we instrumented the system with OpenTelemetry and found out what was actually happening.
- Published On
Tags
consulting (84)architecture (35)debugging (35)developer-experience (26)reliability (20)devops (14)ai (12)performance (11)code-quality (10)observability (9)security (7)engineering-culture (6)microservices (6)database (6)postgres (6)technical-debt (6)databases (6)kubernetes (5)testing (4)ci-cd (4)productivity (4)api-design (4)infrastructure (4)tooling (3)javascript (3)postgresql (3)nodejs (3)dependencies (3)code-review (2)cost-optimization (2)deployment (2)on-call (2)goals (2)monitoring (2)configuration (2)distributed-systems (2)resilience (2)refactoring (2)opentelemetry (2)workflow (1)api (1)contracts (1)cloud (1)typescript (1)caching (1)mdx (1)mongodb (1)migration (1)error-handling (1)feature-flags (1)alerting (1)queues (1)learning (1)journey (1)sonarcube (1)retention (1)orm (1)finops (1)containers (1)platform-engineering (1)incident-management (1)nestjs (1)angular (1)sandbox (1)environments (1)logging (1)secrets-management (1)data-integrity (1)github-actions (1)dns (1)node.js (1)mcp (1)memory (1)data (1)multi-tenant (1)authorization (1)terraform (1)authentication (1)webhooks (1)websockets (1)project-management (1)opinion (1)