The API With a Zero Percent Error Rate and a Fifteen Percent Failure Rate

Our monitoring said everything was fine. Customers said it wasn't. Turns out, the API returned HTTP 200 for every response — including the ones that failed. The dashboards were blind.

The client's Grafana dashboard was a wall of green. Error rate: 0.00%. P99 latency: 280ms. Availability: 99.97%. If you'd walked into that office and looked at the big screen, you'd think everything was perfect.

Meanwhile, customer support had 340 open tickets. Almost all of them said some version of the same thing: "I submitted the form but nothing happened."

I was brought in to figure out why a system that looked healthy by every metric was failing its users. It took me about two hours to find the problem, and a full week to fix it, because the root cause was load-bearing.

The investigation that went nowhere

First I did the obvious things. Checked the deployment history — nothing shipped in the last ten days. Checked the infrastructure metrics — CPU, memory, disk all nominal. Checked the database — query times were fine, no locks, no replication lag.

Then I opened the application logs. Thousands of lines per minute, all at INFO level. I searched for ERROR. Zero results. I searched for WARN. Also zero. I searched for "exception" and "fail" and "timeout." Nothing.

I was starting to wonder if the support tickets were a user education problem when a junior engineer named Priya pulled me aside. "Have you looked at what the API actually returns?" she asked.

She showed me a curl request to the order submission endpoint. It came back HTTP 200 with this body:

{
  "success": false,
  "message": "Unable to process order: inventory service unavailable",
  "data": null
}

HTTP 200. Success false. That was the system's idea of an error.

How every endpoint learned to lie

I dug into the codebase — a C# API built about four years earlier. The original architect had created a base controller with a method called SafeExecute. Every controller action was wrapped in it:

protected IActionResult SafeExecute(Func<IActionResult> action)
{
    try
    {
        return action();
    }
    catch (Exception ex)
    {
        _logger.LogInformation("Request handled: {Message}", ex.Message);
        return Ok(new ApiResponse
        {
            Success = false,
            Message = ex.Message,
            Data = null
        });
    }
}

Read that carefully. It catches every exception, logs it at Information level, and returns Ok() — HTTP 200. The original developer had written a universal error suppressor and called it a safety net.

There were 94 endpoints in the API. Every single one used SafeExecute. The entire application was incapable of returning a non-200 response. It had been this way for four years.

The monitoring was configured to alert on HTTP 5xx responses and elevated 4xx rates. Since neither ever happened, the dashboards stayed green. The alerting rules had never fired. Not once. The team had taken that silence as evidence that the system was reliable.

The actual failure rate

Once I knew what to look for, I wrote a quick script to parse the response bodies from the API gateway logs. Over the previous 30 days, 15.3% of all API responses had "success": false. On peak days, it hit 22%.

The failures broke down roughly like this: 40% were transient errors from downstream services (inventory, payment, shipping), 35% were validation errors that should have been 400s, 15% were authentication failures that should have been 401s, and the remaining 10% were genuine 500-level failures — null references, database timeouts, out-of-memory errors — being served as 200 OK with a polite message.

The frontend JavaScript checked response.ok (which is true for any 2xx status) and then checked response.data for content. It never checked response.success. When data was null, the UI just... did nothing. No error toast, no retry, no feedback. The user clicked submit, the spinner stopped, and the page sat there.

The fix that took a week

The fix wasn't technically complex, but it was tedious. I replaced SafeExecute with proper exception-handling middleware that mapped exception types to HTTP status codes. ValidationException became 400. AuthenticationException became 401. NotFoundException became 404. Everything else became 500.

The tedious part was the frontend. Every API call assumed a 200 response. Every single fetch wrapper, every error boundary, every loading state — all of it trusted that the HTTP status would always be 200. Unwinding that took three developers four days. We found 11 places where the frontend was actually parsing the success: false responses and handling them, which meant those flows would break if we suddenly started returning 4xx and 5xx codes. Each one needed to be migrated to use standard HTTP error handling.

The moment we deployed the fix, the dashboard went from a wall of green to a constellation of amber and red. The actual error rate was immediately visible. The team's Slack channel had a brief panic before I could explain that these weren't new errors — they were old errors that had been invisible.

What stayed with me

The original developer wasn't malicious or incompetent. I've seen this pattern enough times to know the thinking behind it: "If we catch everything, the app can never crash." It's a reasonable-sounding instinct that produces a terrible outcome. The app doesn't crash. It just silently fails, and nobody knows until a customer sends an angry email.

Priya, the junior engineer, had actually raised this issue in a retro six months earlier. It was in the Confluence notes — she'd flagged that error handling seemed inconsistent. The ticket that came out of it was labeled "tech debt" and never prioritized.

Sometimes the person closest to the problem is the one furthest from the authority to fix it.