The Database Password We Rotated Into an Outage

The security audit was right. The production database password hadn't been changed in 18 months. Same password across all environments, set once during initial setup and never touched again. The auditor flagged it, the client agreed it was overdue, and we scheduled the rotation for the following Thursday during low-traffic hours.

What happened next was a 45-minute partial outage that nobody detected until a customer complained their order wasn't confirming.

The setup

The client ran a mid-size e-commerce platform. Twelve microservices, all talking to a shared PostgreSQL cluster on AWS RDS. They'd set up AWS Secrets Manager a year earlier for new services, but the original six services predated that work. Those six read their database credentials from environment variables baked into the Docker image at build time.

Not injected at runtime by ECS or Kubernetes. Baked in. The Dockerfile had a line like this:

ENV DATABASE_URL=postgresql://app_user:hunter2@prod-db.internal:5432/commerce

The newer services fetched credentials from Secrets Manager on startup, which is the right pattern. But nobody had gone back to retrofit the older ones.

We knew about this. The plan was: rotate the password in Secrets Manager, update the RDS instance, then immediately trigger new builds for the six legacy services so they'd pick up the new password. We estimated 10 minutes of downtime for the legacy services while the builds ran. Acceptable for 2 AM on a Thursday.

What actually happened

The Secrets Manager rotation and RDS password update went smoothly. The six newer services picked up the new credentials within seconds — they fetch on startup and refresh every 15 minutes. No issues there.

Then we triggered the CI pipelines for the legacy services. Four of the six built and deployed in about eight minutes. Clean.

The other two failed. One had a flaky integration test that had been skipped in the last deploy but was re-enabled in the default branch. The other hit a dependency resolution error — a transitive dependency had yanked a patch version three days earlier.

So we had two services running containers with the old password, trying to connect to a database that now only accepted the new one. Every database query from those services failed.

Why nobody noticed for 45 minutes

This is the part that still bothers me.

Both services had health check endpoints. The load balancer pinged /health every 30 seconds, and both services dutifully returned 200 OK. Here's roughly what the health check did:

app.get('/health', (req, res) => {
  res.json({ status: 'ok', uptime: process.uptime() });
});

No database check. No dependency verification. Just "yes, the process is running." The load balancer was happy, the ECS task status was RUNNING, and CloudWatch showed zero alarms because the alarms were based on HTTP 5xx rates from the load balancer — and the services weren't returning 5xx errors. They were returning 200s for cached reads and silently swallowing the database errors on writes, logging them at warn level instead of error.

The order confirmation service was one of the two. When a customer placed an order, the payment processing happened through a different service (which had the new password and worked fine), but the order status update — writing the confirmed status back to the orders table — failed silently. The API returned a 200 to the frontend because the payment succeeded. The status just never updated.

Forty-five minutes later, a customer called support: "I got charged but my order still says pending."

The fix and the argument

The immediate fix was ugly but fast. We SSH'd into the ECS instances, pulled the running container IDs, and exec'd in to update the environment variable manually, then restarted the processes. Not something you'd put in a runbook, but it got the two services talking to the database again within five minutes of us realizing the problem.

Then we fixed the broken CI pipelines and did proper deploys.

The more interesting conversation happened the next day. The engineering lead wanted to retrofit all six legacy services to use Secrets Manager. The platform team pushed back — they wanted to move to a sidecar pattern where a small agent process would handle credential refresh and expose it via a local socket. The security team just wanted confirmation that the password had actually been rotated and didn't care about the mechanism.

We ended up doing the boring thing: retrofitting the six services to read from Secrets Manager at startup, with a fallback to environment variables during the transition. Took about two days. The sidecar idea went into a backlog that, last I checked, still has 40 items ahead of it.

What I took away from this

The credential rotation itself was correct. Letting a production password sit unchanged for 18 months is genuinely bad practice, and several real-world breaches in 2026 alone traced back to static credentials that should have been rotated — the Clerk outage and Resend incident both involved database connectivity failures where operational assumptions about "how things connect" turned out to be wrong.

The failure wasn't in deciding to rotate. It was in treating rotation as a single atomic operation when it's actually a coordination problem across every consumer of that credential. We had a checklist: update Secrets Manager, update RDS, rebuild services. What we didn't have was a verification step — something that confirmed every service could actually reach the database with the new credentials before we declared the rotation complete.

Three patterns I now insist on for any credential rotation:

First, dual-password windows. RDS supports this natively — you can have two valid passwords simultaneously. Rotate by adding the new password, updating all consumers, verifying connectivity, then removing the old password. The rotation becomes a four-step process instead of a two-step one, but there's never a moment where a valid credential stops working.

Second, health checks that check health. If your service needs a database to function, your health check should include a SELECT 1 or equivalent. Separate your liveness probe (is the process running?) from your readiness probe (can this service handle requests?). Kubernetes has this distinction built in. ECS makes you work for it, but it's worth the effort.

Third, rotation dry runs. Before rotating in production, rotate in staging first — but not just the Secrets Manager side. Simulate the full sequence: change the password, see which services recover automatically, see which ones break. We would have caught the baked-in credentials in staging if we'd actually run the rotation end-to-end there. We didn't because "staging uses the same Secrets Manager setup." It didn't, for the legacy services.

The uncomfortable truth is that credential rotation is one of those things that's trivially easy to describe and surprisingly hard to execute safely. The gap between "change the password" and "change the password without breaking anything" is where most of the engineering work lives. And if you haven't done a rotation in 18 months, the first one is going to be the scariest — because you don't know what you don't know about how your services actually get their credentials.

When was the last time you rotated your production database password? And more importantly — do you know, right now, which of your services would survive it?