The API Key That Outlived Three Engineers

A client found one of their API keys in a public error log. Tracing where that key actually lived took longer than fixing the leak — and revealed a secrets management problem nobody wanted to own.

The ticket said "API key exposed in error log." Severity: high. Assigned to: nobody, because the engineer who created the key had left the company two years ago.

I was four weeks into a consulting engagement with a mid-size logistics company when their security team flagged it. A partner integration was failing, and the error response — including a full API key for their geocoding provider — was being written to an application log. That log was shipped to a hosted logging platform. The logging platform had a dashboard. The dashboard was shared with three external contractors.

The key had been in those logs for nine days before anyone noticed.

The first hour

Rotating the key should have been simple. Go to the geocoding provider, generate a new one, update the config, deploy. Thirty minutes, tops.

Except nobody knew where the key was actually configured. The service that leaked it was a Node.js API that read the key from process.env.GEOCODING_API_KEY. Fine. But where did that environment variable come from?

The Kubernetes deployment manifest referenced a Secret called geocoding-credentials. That Secret was created by a Helm chart. The Helm chart's values file pointed to a sealed secret in a GitOps repo. But when I checked the sealed secret, it hadn't been updated in fourteen months — and the key inside it didn't match the one in the running pod.

Someone had run kubectl edit secret in production and changed it by hand. There was no record of who or when.

The archaeology

Over the next day, I grep'd my way through every repository the team had access to. That one API key — the same literal string — appeared in:

• The Kubernetes secret (hand-edited)
• A .env.example file in the main API repo (committed with the actual key, not a placeholder)
• A GitHub Actions workflow for a batch processing job
• A Terraform variable in their infrastructure repo, stored as a default value in variables.tf
• A Confluence page titled "Onboarding: Setting Up Your Local Environment"
• A Slack message from 2024 in the #backend channel

Six places. Same key. The key had been created by an engineer who left in early 2024. It was copied by a second engineer into the CI pipeline before she moved to another team. A third engineer, now also gone, pasted it into the Confluence doc during an onboarding rewrite.

None of them were around to explain the history. The current team inherited all of it and never questioned it, because the geocoding integration just worked.

The rotation that took a week

Rotating should have been the easy part. It wasn't. When I generated a new key and updated the Kubernetes secret, the main API recovered immediately. But twenty minutes later, the batch processing job started failing. Nobody had remembered it used the same key through a completely separate config path.

After fixing that, the QA environment broke. It turned out QA was pointing at the production geocoding account — not a sandbox — because "the sandbox had rate limits that made testing slow." That decision was made in 2023 and never revisited.

The full rotation — finding every reference, updating every system, verifying every consumer — took six working days. For one key. They had over forty external API integrations.

The pattern I keep seeing

This wasn't unusual. In the past year, I've seen some variation of this at almost every client:

No ownership. Credentials get created in the moment, by whoever happens to be working on the integration. There's no registry, no owner field, no expiration date. When that person leaves, the key becomes an orphan.

Copy-paste as configuration management. Instead of referencing a single source of truth, the key gets copied into each system that needs it. .env files, CI secrets, Kubernetes manifests, documentation wikis. Each copy drifts independently.

Rotation is theoretically possible but practically impossible. Teams tell me they "can rotate keys if needed." When I ask them to actually do it for one key, it takes days. The dependency graph is invisible until you start pulling the thread.

The working system hides the problem. As long as the integration works, nobody audits the credentials. The incentive to fix secrets management is zero right up until the moment it's an emergency.

What actually helped

We didn't adopt a fancy secrets management platform. The team was twelve engineers — they didn't need HashiCorp Vault. What they needed was visibility and process.

We built a simple spreadsheet. Every external credential got a row: the provider, the key identifier (not the value), which services used it, who owned it, when it was last rotated, and where the canonical source of truth was. It took two days to populate and immediately revealed three keys that were no longer used by anything.

Then we established two rules. New credentials go into the cloud provider's secret manager and get referenced from there — no copies. And every key gets a calendar reminder for rotation every 90 days, owned by a specific person.

Not glamorous. No YAML-driven secrets-as-code pipeline. But six weeks later, when they needed to rotate a payment processor key for an unrelated reason, it took forty minutes instead of six days.

The thing that worries me

The Vercel security incident in April put secrets exposure back in the news. But most secrets leaks aren't sophisticated supply chain attacks. They're a console.log someone forgot to remove, a .env file that got committed, a wiki page that's technically internal but shared too broadly.

The tooling for secrets management has gotten genuinely good. AWS Secrets Manager, GCP Secret Manager, 1Password for teams, Doppler, Infisical — pick one. The hard part was never the tool. The hard part is getting a team to admit they don't know where their own credentials live.

If you're reading this and feeling a little uneasy about your own team's situation — good. Open a terminal, pick one API key, and try to answer three questions: who created it, where is it stored, and when was it last rotated?

I'd bet the answers take longer than you'd like.