The Internal Certificate Nobody Was Watching

An internal root CA certificate expired on a Saturday afternoon. Mutual TLS failed across nine services, and the team spent four hours figuring out why everything could reach the internet but nothing could talk to itself.

The alert was vague. "Service order-processor health check failing." It was a Saturday, 2:14 PM. The on-call engineer restarted the pod. It came back, passed its readiness probe, and immediately started failing again. So she restarted it again. Same result.

By 2:30 she'd noticed that five other services were in the same state. All of them could reach external APIs. All of them could resolve DNS. None of them could talk to each other.

I was brought in the following Monday, once the dust had settled. The outage had lasted four hours and twelve minutes. The root cause was an expired certificate that nobody knew existed.

The setup nobody questioned

This was a mid-size logistics company running about 30 services on Kubernetes. Two years earlier, a platform engineer had set up mutual TLS between services using an internal certificate authority. The implementation was solid — a root CA issued intermediate certificates, which in turn signed per-service certificates that rotated automatically every 90 days. A small cert-manager deployment handled the rotation. It worked so well that nobody thought about it.

The root CA certificate had a validity period of two years. That platform engineer had left the company fourteen months ago.

Here's the thing about root CA certificates: they sit at the top of the trust chain, but nothing rotates them automatically. The per-service certs rotate on schedule. The intermediates usually have a longer lifetime. But the root? Someone has to remember. And in this case, nobody did.

Why it was hard to diagnose

The on-call engineer spent the first ninety minutes looking in the wrong places. Here's why.

The services could reach external APIs because those connections used public TLS — certificates signed by well-known CAs that had nothing to do with the internal PKI. Health check endpoints responded on HTTP. Database connections used password auth over private networking. Everything that didn't rely on the internal CA worked perfectly.

The error messages were not helpful. Go services logged x509: certificate has expired or is not yet valid. Node services logged UNABLE_TO_VERIFY_LEAF_SIGNATURE. The Java service logged a stack trace that mentioned sun.security.ssl.SSLHandshakeException buried under forty lines of framework noise. None of them said "your root CA expired."

The team's initial theory was a bad deploy — someone must have pushed a config change that broke mTLS. They checked. Nothing had been deployed since Thursday. Then they suspected cert-manager. They restarted it, re-issued service certificates, and watched the same errors continue. Fresh certs signed by an expired chain are still untrusted.

It took a senior engineer who'd been at the company long enough to remember the original mTLS setup to ask the question: "When does the root cert expire?" The answer, once they found the certificate file and ran openssl x509 -enddate -noout, was two hours ago.

The fix was simple. The lesson wasn't.

Generating a new root CA certificate and re-signing the intermediate took about twenty minutes. Rolling it out to every service took another hour because they had to update the trust bundle in every pod and restart each service in dependency order. Total recovery time: four hours and twelve minutes of downtime for a certificate that took twenty minutes to replace.

The real question was why this happened at all. The team had monitoring for almost everything. Uptime checks, error rate alerts, latency dashboards, even cost alerts on their cloud bill. But nobody had set up a single alert for certificate expiry.

When I asked why, the infrastructure lead said something I've heard in various forms at multiple clients: "It's one of those things where the automation works until it doesn't, and you forget it's there."

That's the core of the problem. Internal PKI is invisible infrastructure. It doesn't show up in application metrics. It doesn't have a line item on the cloud bill. It doesn't slow down over time like a database or spike under load like an API. It just works, silently, until a specific date, and then it stops.

What we actually changed

We spent two days putting safeguards in place. None of it was technically sophisticated.

First, we added certificate expiry monitoring. A small CronJob that runs daily, checks every certificate in the chain, and fires a PagerDuty alert when anything is within 30 days of expiry. Total implementation time: about two hours.

Second, we documented the entire PKI chain in a runbook. Not a wiki page that would gather dust — a runbook linked directly from the on-call handbook, with step-by-step rotation instructions and the location of every key file. The on-call engineer that Saturday was sharp and experienced, but she'd never been told the internal CA existed.

Third — and this is the one the team pushed back on — we shortened the root CA validity to one year. Their argument: a longer validity means fewer rotations and less risk of a botched renewal. My argument: a two-year certificate means you practice the rotation process zero times before it matters. One year means you do it once and build muscle memory. If the rotation process is painful enough that you want to avoid it, that's a reason to automate it, not to defer it.

They went with one year. Their first scheduled rotation, three weeks later, took eleven minutes.

The pattern I keep seeing

This isn't just about certificates. I've watched the same dynamic play out with DNS TTL configurations, API key expiry policies, cloud IAM role session limits, and even database connection pool settings. Any piece of infrastructure that works silently for months or years becomes invisible. And invisible infrastructure doesn't get monitored, documented, or tested.

The uncomfortable truth is that reliability isn't about handling the failures you expect. It's about maintaining awareness of the systems that haven't failed yet. The services that keep running don't get attention. The certificates that haven't expired don't get checked. The backups that haven't been needed don't get tested.

I don't have a clean solution for this. Calendar reminders help. Automated expiry checks help more. But the underlying problem is cultural — teams naturally focus on what's broken, not on what's quietly decaying. The best defense I've found is a periodic "what are we not watching?" review, where the team explicitly asks what infrastructure has been running untouched for more than six months.

What's the longest you've gone without checking a piece of critical infrastructure that you assumed was fine?