The Auth Logic That Lived in Twelve Different Places

The engagement started like most security reviews. The CTO of a B2B SaaS company — about 60 engineers, Series C, healthcare-adjacent — wanted a "quick check" on their API security posture before a SOC 2 audit. He expected me to find some missing rate limits, maybe a few endpoints without proper input validation. Standard stuff.

Instead, I spent three weeks untangling how authentication actually worked across their platform. The answer was: differently, everywhere, and sometimes not at all.

Asking the wrong question

My first mistake was asking "how does auth work?" in a meeting with the platform team. I got a clean, confident answer. "All requests go through the API gateway. The gateway validates the JWT, extracts the user context, and forwards it to the downstream service. We use Auth0."

That was true for about eight of their twenty-two services.

When I started actually tracing requests, the picture got more complicated. I set up a proxy, logged headers on every internal call, and mapped out what was really happening. Here's what I found:

• 8 services received traffic through the API gateway, which validated JWTs correctly
• 5 services were called directly by other services over an internal network, bypassing the gateway entirely — these validated JWTs themselves, each using a different library
• 4 services were behind a legacy load balancer from before the gateway migration and used session cookies backed by a shared Redis instance
• 3 services accepted traffic from a partner integration layer that used API keys stored in a Postgres table
• 2 services relied on mutual TLS for auth because "only other internal services call them"

That's twenty-two services and four distinct authentication mechanisms. Five if you count the two services where mTLS was configured but the certificate validation was set to rejectUnauthorized: false in the Node.js config because someone couldn't get the cert chain right during development.

The service with no auth

The real find came on day four. One of the "internal-only" services — a document generation service that created PDFs from patient data — had no authentication at all. No JWT check, no API key, no mTLS. It listened on port 8080 and accepted any POST request with the right JSON shape.

The developers who built it assumed it would only ever be called by the orchestration service. They were right for about eighteen months. Then a team building a new reporting feature needed PDF generation and pointed their service at it directly. Then the partner integration layer started calling it for a different use case. Nobody added auth because nobody realized it was missing — the calls worked, so the service must have been "properly configured."

The document generation service had access to patient records. It was reachable from any service in the VPC. The only thing standing between an attacker who compromised any single service and a full dump of patient PDFs was the assumption that internal traffic was trustworthy.

How this happens

Nobody decided to implement auth twelve different ways. It happened incrementally, the way these things always do.

The company started as a monolith with session-based auth. When they broke out the first few microservices, those services kept using session cookies because the auth code came along with them. A year later, a new platform team introduced the API gateway and JWTs. They migrated the customer-facing services but left the internal ones for "phase two." Phase two never happened.

Meanwhile, a partnership team needed API key auth for external integrations, so they built their own key management. The infrastructure team set up mTLS for a couple of sensitive services because a security consultant (not me) recommended it. Each decision made sense in isolation. Nobody had the full picture.

The real problem wasn't any individual auth mechanism. It was that there was no single place in the codebase — or in anyone's head — that could answer the question: "how does this request get authenticated?"

The JWT library zoo

Even among the services that did validate JWTs, the implementation varied. I found three different JWT libraries across the five services that handled their own validation:

// Service A: jsonwebtoken, checking expiry manually
const decoded = jwt.verify(token, publicKey, { algorithms: ['RS256'] });
if (decoded.exp < Date.now() / 1000) { ... }
 
// Service B: jose, using JWKS endpoint
const { payload } = await jwtVerify(token, JWKS, {
  issuer: 'https://company.auth0.com/',
  audience: 'https://api.company.com'
});
 
// Service C: jsonwebtoken, but with a symmetric key (?!)
const decoded = jwt.verify(token, process.env.JWT_SECRET);

Service C was using a shared secret instead of the RS256 public key that Auth0 was configured to use. This worked because someone had also configured Auth0 to issue tokens with HS256 for "backwards compatibility." That meant any service that knew the shared secret — which was in a .env file committed to the repo — could forge tokens for any user.

When I asked the tech lead about Service C, he stared at the code for about thirty seconds and said "that was supposed to be temporary." It had been running for fourteen months.

What we did about it

The fix wasn't technically complicated, but it required discipline and patience. We spent about six weeks on it:

Week 1-2: Built a shared auth middleware library that all services would use. One library, one configuration source (the JWKS endpoint), one set of validation rules. We resisted the urge to make it "flexible" — it validated RS256 JWTs from Auth0, checked issuer and audience, and that was it. Services that needed API key auth got a second middleware that checked keys against a central service, not a local database table.

Week 3-4: Migrated services one at a time, starting with the document generation service (which went from zero auth to proper JWT validation). Each migration was a separate PR with its own test plan. We ran old and new auth in parallel for 48 hours per service, logging discrepancies.

Week 5-6: Killed the HS256 signing configuration in Auth0. Rotated the shared secret. Removed the legacy session cookie infrastructure. Updated the partner integration to use the centralized API key service.

The mTLS services were the last to migrate. The infrastructure team wanted to keep mTLS as an additional layer, which was fine — defense in depth is good. But we made sure JWT validation happened too, so authentication didn't depend solely on the network layer.

The uncomfortable part

Here's what I keep thinking about. This company had a security team. They had a SOC 2 prep checklist. They had quarterly access reviews. None of that caught the fact that a service handling patient data had zero authentication.

The access reviews checked who had access to what in their identity provider. They didn't check whether services actually enforced that access. The SOC 2 checklist asked "do you use encryption in transit?" and "do you have an authentication mechanism?" — yes and yes, technically true, wildly misleading.

I've seen this pattern at four different clients in the past two years. The auth architecture that's described in the wiki bears no resemblance to what's actually running. The diagram shows a clean gateway pattern. The reality is a patchwork of mechanisms accumulated over three years of microservices migration, team turnover, and deadline pressure.

If you're running microservices and you haven't traced an actual request through every auth boundary in the past six months, you might be in the same spot. Not because anyone was careless. Because distributed systems make it easy to lose track of the answers to simple questions like "who is allowed to call this service, and how do we know?"

That's the thing about auth in a microservices architecture. It's not a feature you build once. It's an invariant you have to keep enforcing, service by service, deploy by deploy, long after everyone's moved on to the next project.