The Autoscaler That Made the Outage Worse

During a traffic spike, our autoscaler saw dropping CPU and scaled down. The database was the actual bottleneck. Fewer instances meant more connections per pod, which hit the connection limit and turned a slow page into a full outage.

The first sign was a Slack message from a product manager: "Is the site slow for everyone or just me?" Twenty minutes later, the site wasn't slow anymore. It was down.

I was four weeks into an engagement with a mid-size e-commerce company — about 80 engineers, a Kubernetes-based platform, and a sale event they'd been promoting for a month. The platform had handled similar traffic before. They had auto-scaling configured. They had dashboards. They thought they were ready.

The timeline

At 9:02 AM, traffic started climbing. Marketing had sent the email blast. Request rates doubled within ten minutes, which was expected and within the range they'd load-tested for.

At 9:14 AM, average response times crept from 120ms to 450ms. Not great, but the pages were still loading.

At 9:18 AM, the Horizontal Pod Autoscaler (HPA) scaled the main API service from 12 pods down to 8.

Read that again. Traffic was spiking, latency was climbing, and the autoscaler removed capacity.

By 9:24 AM, response times crossed 4 seconds. Health check failures started cascading. The load balancer pulled three of the remaining eight pods out of rotation. Five pods left. At 9:31 AM, the site was returning 502s for roughly 60% of requests.

Why the autoscaler did the wrong thing

The HPA was configured to scale on CPU utilization, targeting 70%. This is the default everyone copies from blog posts and Stack Overflow answers. It works fine when your service is CPU-bound.

This service was not CPU-bound.

The API was a fairly typical Node.js application talking to PostgreSQL through a connection pool. Under normal load, each pod held about 15 active database connections out of a pool max of 20. The database itself, an RDS instance with a max_connections of 200, was comfortably handling the 180 or so connections from the 12 pods.

When traffic spiked, the database became the bottleneck first. Query times increased as Postgres struggled with lock contention on the inventory tables — every "add to cart" operation was checking and decrementing stock counts. The application threads weren't burning CPU. They were waiting. Waiting on database responses, waiting on connection pool checkouts, waiting on I/O that was never going to get faster by staring at it.

CPU utilization across the pods dropped from 65% to 38%.

The HPA saw 38% and thought: "We're way below the 70% target. Time to scale down." So it did exactly what it was told to do.

The cascade

With fewer pods, the surviving instances had to absorb more requests. Each pod opened more database connections. Eight pods maxing out 20 connections each is 160 — still under the RDS limit. But the queries were already slow, so connections were held longer. The pool checkout timeout started firing. Requests backed up in the event loop. The health check endpoint, which shared the same connection pool, started timing out.

Kubernetes saw unhealthy pods and did what Kubernetes does: it killed them and tried to schedule new ones. But the new pods went through the same cycle — grab connections, get slow responses, fail health checks, get killed. The system was churning.

Meanwhile, the HPA was still looking at CPU. Still below 70%. Still not scaling up.

The fix, in the moment

The on-call engineer manually scaled the deployment to 20 pods and temporarily disabled the HPA. That helped with the connection distribution but didn't solve the database bottleneck. What actually brought things back was a DBA who noticed the lock contention pattern and killed a handful of long-running analytical queries that a reporting dashboard had kicked off at 9:00 AM — its daily scheduled refresh.

The combination of sale traffic hitting the inventory tables and the reporting queries holding locks on the same tables was the root cause. The autoscaler just made the blast radius larger.

The actual fix, after the fire

We changed three things over the following week.

First, we switched the HPA metric from CPU to request latency at the p95, pulled from the Prometheus metrics the ingress controller was already emitting. The scaling policy now said: if p95 latency exceeds 500ms for 60 seconds, scale up. If it stays below 200ms for 5 minutes, scale down. This meant the autoscaler responded to what users actually experienced.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  minReplicas: 10
  maxReplicas: 30
  metrics:
    - type: Pods
      pods:
        metric:
          name: http_request_duration_seconds_p95
        target:
          type: AverageValue
          averageValue: "0.5"
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Pods
          value: 2
          periodSeconds: 120

Second, we moved the reporting queries to a read replica. This should have been done months earlier, but the reporting dashboard had been built as a quick internal tool and nobody revisited the database it pointed at.

Third, we introduced PgBouncer as a connection pooler between the application and RDS. Instead of each pod managing its own connection pool directly against the database, pods connected to PgBouncer, which maintained a smaller number of actual database connections and multiplexed them. This decoupled "number of pods" from "number of database connections" and gave us room to scale the application tier without worrying about hitting the connection ceiling.

The thing that bothers me

Every piece of this setup was individually reasonable. CPU-based autoscaling is the documented default. The connection pool settings were sensible. The reporting dashboard was doing its job. The database connection limit was appropriate for the instance size.

The failure emerged from the interaction between these components — a property that none of them could express on their own. The autoscaler didn't know about the database. The database didn't know about the autoscaler. The reporting dashboard didn't know about the sale.

I keep running into this pattern in consulting work. Each team owns their piece, and each piece works correctly in isolation. The system fails in the gaps between ownership boundaries. The reporting team didn't know their queries would coincide with a traffic spike. The platform team didn't know the autoscaler's metric would invert during an I/O bottleneck. The DBA didn't know the application tier would flood connections during scale events.

There's no single fix for this. Read replicas and connection poolers and better HPA metrics are all good engineering. But the underlying problem — that distributed systems fail in the interactions, not the components — is one of those truths that's easy to say and hard to internalize until you've watched an autoscaler helpfully remove capacity during an outage.

What's your autoscaler actually scaling on?