Lk
Lab notes

Case study

The Internal API That Had External Users

A client asked me to refactor their order management API. Nobody mentioned the twelve companies that had been calling it from outside the network for two years.

I got brought in to help a mid-size e-commerce company refactor their order management service. The brief was straightforward: the API had grown organically over three years, the endpoint naming was inconsistent, some responses returned XML for no discernible reason, and the team wanted to clean it all up before building the next round of features.

Standard stuff. I spent the first two days mapping out the endpoints, documenting the request/response shapes, and flagging what could change safely. By Wednesday, I had a migration plan: rename 14 endpoints, consolidate three redundant ones, switch everything to JSON, and add proper pagination to the list endpoints that were returning unbounded result sets.

Then I looked at the access logs.

The traffic that shouldn't have been there

The API sat behind an internal load balancer. It was supposed to handle requests from the company's own frontend, their warehouse management system, and a handful of internal tools. I expected to see traffic from a known set of IPs.

Instead, I found 47 unique external IP addresses making regular requests to six of the endpoints. Some were hitting the API multiple times per minute. The traffic had been there for at least eighteen months.

I asked the engineering lead about it. He stared at the screen for a good ten seconds before saying, "That... shouldn't be happening."

How twelve companies ended up on an internal API

It took us two days of digging through Zendesk tickets and Slack messages to piece together the story.

Twenty-two months earlier, one of the company's fulfillment partners needed order status data. The proper way to handle this would have been to build an external API with authentication, rate limiting, documentation, and a versioning strategy. That would have taken the team a few weeks.

Instead, a support engineer — trying to be helpful and under pressure to unblock the partner — sent them a cURL command pointing at the internal API. No authentication. It worked over the public internet because the load balancer hadn't been configured to restrict source IPs.

The partner built an integration. It worked. They told another partner. A developer at a third company found the endpoint pattern by reading network traffic in their browser dev tools while someone screenshared the admin dashboard in a meeting. Within six months, twelve companies were treating this internal API as a documented, supported integration.

None of them had signed an API agreement. None of them had API keys. None of them had been told about rate limits, because there weren't any.

Why this was worse than it sounds

Those twelve companies had customers. Their integrations were processing real orders. One of them was a logistics provider handling about 2,000 shipments per day through data they pulled from this API.

I couldn't just block the traffic. That would break real supply chains. I couldn't change the endpoint paths or response formats — that would break their integrations silently, and since they had no official relationship with the API, there was no way to send them a deprecation notice.

We were stuck supporting an API contract that nobody had ever intentionally created.

The team's planned refactor? Every one of the 14 endpoint renames would have broken at least one external integration. The consolidation of the three redundant endpoints? Two of them had external callers. The switch from XML to JSON? Three consumers were specifically parsing the XML responses.

If I hadn't checked the access logs, the team would have shipped the refactor and caused outages at a dozen companies they didn't know existed.

What we actually did

The cleanup took six weeks instead of the planned two.

First, we put a reverse proxy in front of the API for visibility without changing the underlying service:

upstream internal_api {
    server api-internal:8080;
}
 
server {
    listen 443 ssl;
    server_name api.example.com;
 
    access_log /var/log/nginx/external_api_access.log detailed;
 
    location /api/ {
        proxy_set_header X-External-Caller $remote_addr;
        proxy_set_header X-Original-URI $request_uri;
        proxy_pass http://internal_api;
    }
}

Second, we contacted every external company individually. This was the most tedious part. We tracked them down through IP addresses, cross-referenced with known partner contacts, and in two cases had to do reverse DNS lookups and cold-email engineering teams at companies we'd never officially worked with.

Third, we built a proper external API — not a new thing, just a thin layer on top of the existing endpoints with authentication, rate limiting, versioning, and actual documentation. We gave each partner an API key and a six-month migration window.

Warning

If you're thinking "just add auth and call it done" — bolting authentication onto endpoints that external systems already call without auth means you break everyone on deploy day. You need a migration path, not a switch.

Fourth, we restricted the internal load balancer to internal IPs only, but only after every external caller had been migrated to the new proxy.

The refactor the team originally wanted finally happened in month three, behind the versioned external API. The v1 endpoints stayed frozen as a compatibility layer while the internal service evolved underneath.

The uncomfortable takeaway

Every HTTP endpoint that's reachable from the internet is a public API. It doesn't matter what you call it internally or that it's not documented. If someone can reach it, someone eventually will — and once they build on it, you own a contract you never agreed to.

The real mistake wasn't the support engineer sharing that cURL command. It was running an unauthenticated API on a public-facing load balancer and assuming "internal" was a property of the endpoint rather than the network configuration.

I've since added a check to my first-week audit for new clients: pull the access logs for every API and look for traffic that doesn't match the expected caller list. In about one out of four engagements, I find something unexpected.

What's the most accidentally-public thing you've found in a codebase?