Our monitoring said everything was fine. Customers said it wasn't. Turns out, the API returned HTTP 200 for every response — including the ones that failed. The dashboards were blind.
- Published On
api design
- A client's checkout endpoint had idempotency keys. Customers still got double-charged during flash sales. The bug was a three-line race condition between SELECT and INSERT that took two weeks to find.
- Published On
- A client was six months into migrating from REST to GraphQL. Half their endpoints were converted, developer experience was arguably worse, and the frontend team was maintaining two data layers. Sometimes the right engineering decision is to stop.
- Published On
- A client's API had rate limiting configured and enforced. It still couldn't prevent a single customer from tanking performance for everyone else. The problem wasn't the limiter — it was what we were counting.
- Published On
Tags
consulting (89)architecture (39)debugging (37)developer-experience (28)reliability (22)devops (15)ai (13)code-quality (11)performance (11)observability (9)databases (7)security (7)microservices (6)postgres (6)engineering-culture (6)technical-debt (6)database (6)kubernetes (5)api-design (4)ci-cd (4)infrastructure (4)testing (4)productivity (4)distributed-systems (3)nodejs (3)dependencies (3)javascript (3)postgresql (3)tooling (3)typescript (2)migration (2)on-call (2)refactoring (2)configuration (2)resilience (2)cost-optimization (2)code-review (2)deployment (2)opentelemetry (2)monitoring (2)goals (2)developer-tooling (1)build-systems (1)elasticsearch (1)github-actions (1)data-integrity (1)retention (1)authentication (1)node.js (1)data (1)websockets (1)project-management (1)mcp (1)finops (1)multi-tenant (1)authorization (1)terraform (1)dns (1)opinion (1)memory (1)error-handling (1)secrets-management (1)webhooks (1)containers (1)environments (1)queues (1)cloud (1)orm (1)alerting (1)incident-management (1)caching (1)platform-engineering (1)feature-flags (1)api (1)contracts (1)logging (1)mongodb (1)workflow (1)nestjs (1)angular (1)sandbox (1)sonarcube (1)learning (1)journey (1)mdx (1)