The Rate Limiter That Let Everything Through

A client's API had rate limiting configured and enforced. It still couldn't prevent a single customer from tanking performance for everyone else. The problem wasn't the limiter — it was what we were counting.

Last quarter I got pulled into a client engagement that started with a familiar-sounding complaint: "Our API gets slow during business hours." The client was a B2B SaaS company — about 200 API consumers, mostly other companies integrating with their data platform. They had all the expected guardrails in place. API keys, rate limiting at the gateway, monitoring dashboards. On paper, everything looked fine.

The rate limiter was set to 120 requests per minute per API key. No customer was exceeding that limit. The 429 response counter was essentially zero. And yet, every weekday between 10 AM and 2 PM Eastern, response times would balloon from 200ms to 8-12 seconds, and occasionally time out entirely.

Not all requests are born equal

The first thing I did was pull request logs broken down by endpoint. The API had about 30 endpoints — standard CRUD stuff for the most part. But one endpoint stood out: /reports/generate. It accepted a set of filters and date ranges, ran aggregation queries across several large tables, assembled a PDF, and returned a download URL.

A typical CRUD call — fetch a record, update a field — hit the database once or twice and returned in under 100ms. A call to /reports/generate could run for 10 to 45 seconds depending on the date range, touching millions of rows across three tables with multiple joins.

The rate limiter treated both of these as "one request."

I pulled the logs for their heaviest API consumer — a large enterprise client that had built an automated reporting pipeline. Every morning at 10 AM Eastern, their system fired off about 60 report generation requests in rapid succession, one for each of their regional offices. Sixty requests per minute. Well within the 120 RPM limit. Absolutely devastating to the backend.

Sixty lightweight requests? No problem. Sixty report generations? That's 60 concurrent queries each scanning date ranges across tens of millions of rows. The database connection pool filled up. CPU on the primary database hit 100%. Every other API consumer experienced the slowdown.

The rate limiter was doing its job perfectly

This is the part that took a minute to land with the client's team. There was no bug. The rate limiter was configured correctly and working as intended. The problem was the mental model behind it.

They'd set up rate limiting as "prevent abuse," where abuse meant "too many requests." But their actual bottleneck wasn't request count — it was compute cost per request. A system that lets you send 120 cheap requests or 120 expensive requests per minute with no distinction is protecting a number, not a resource.

What we changed

The fix wasn't complicated, but it required rethinking what "one request" meant.

We introduced weighted rate limiting. Each endpoint got a cost multiplier based on its typical backend impact. Most CRUD endpoints stayed at cost 1. The report generation endpoint got a cost of 20. The rate limit became 120 cost units per minute instead of 120 requests per minute. That same enterprise customer could still fire off their 60 reports — they'd just get throttled after 6 of them, with the rest queued.

rate_limits:
  default:
    cost: 1
    window: 60s
    max_cost: 120
 
  endpoints:
    /reports/generate:
      cost: 20
    /exports/bulk:
      cost: 15
    /analytics/query:
      cost: 10

We also moved report generation to an async model. Instead of blocking on the response, the endpoint returned a job ID immediately (cost 1), and the client polled a status endpoint until the report was ready. This let us control concurrency on the backend — no more than 5 report jobs running simultaneously — without the API consumer needing to care about our infrastructure constraints.

The enterprise customer's integration needed a small update to handle the async pattern, but they were happy to make the change once we explained that it also meant their reports wouldn't time out during peak hours.

The broader pattern

I've seen this same mistake in three different consulting engagements now. Teams set up rate limiting early, pat themselves on the back, and never revisit it as the API evolves. The rate limiter was designed when every endpoint was a simple database lookup. Then someone adds a report generator, a bulk export, a search endpoint that fans out to six downstream services. The cost profile of the API changes completely, but the rate limiter still counts everything as one.

It's worth auditing this periodically. Pull your slowest endpoints, check their average response time, and compare that to what your rate limiter allows. If one consumer can legally burn 90% of your backend capacity while staying under the rate limit, your rate limiter is a checkbox, not a safeguard.

What does your rate limiting actually protect — request count, or the thing that matters?