We practiced deployments religiously but never tested a rollback. When a release broke checkout and we hit the big red button, we found out half the system couldn't actually go backward.
- Published On
deployment
- We built a canary deployment pipeline with automated rollbacks. It still let a bad release through to 100% of users. Here's what went wrong.
- Published On
Tags
consulting (48)debugging (22)developer-experience (16)architecture (16)reliability (14)devops (9)ai (7)observability (7)performance (7)code-quality (6)database (5)kubernetes (4)postgres (4)technical-debt (3)microservices (3)testing (3)engineering-culture (3)deployment (2)databases (2)security (2)opentelemetry (2)monitoring (2)postgresql (2)productivity (2)ci-cd (2)tooling (2)javascript (2)goals (2)configuration (1)error-handling (1)secrets-management (1)webhooks (1)infrastructure (1)resilience (1)containers (1)environments (1)queues (1)cloud (1)cost-optimization (1)orm (1)refactoring (1)on-call (1)alerting (1)incident-management (1)typescript (1)distributed-systems (1)caching (1)platform-engineering (1)dependencies (1)code-review (1)feature-flags (1)api (1)contracts (1)logging (1)mongodb (1)migration (1)workflow (1)nestjs (1)angular (1)sandbox (1)sonarcube (1)learning (1)journey (1)mdx (1)