The Test Suite With 90% Coverage That Caught Zero Bugs

A client's codebase had impressive test coverage numbers. Then I introduced a deliberate bug and watched every single test pass. Coverage was measuring execution, not verification.

The tech lead pulled up the coverage report during our first meeting. 91.4% line coverage. He was proud of it, and honestly, he should have been. Getting a team to write tests at all is hard. Getting them above 90% is rare.

But something felt off when I started reading the tests. Two days into the engagement, I understood why. The test suite was a trophy on the shelf — visible, impressive, and doing absolutely nothing useful.

The shape of the problem

The client was a B2B SaaS company with a TypeScript backend — around 60,000 lines of business logic across a dozen services. They'd brought me in because bugs kept reaching production despite what looked like solid engineering practices. Code reviews, CI gates, coverage thresholds. All the boxes were checked.

I started by reading their test files, picking services at random. Here's a simplified version of the kind of test I kept finding:

test("processOrder creates an order", async () => {
  const result = await processOrder({
    userId: "user-1",
    items: [{ sku: "WIDGET-1", quantity: 2 }],
  });
  expect(result).toBeDefined();
});

That test exercises the processOrder function. The coverage tool sees every line of the function execute. Coverage goes up. But the only assertion is that the function returns something. It could return { status: "catastrophic_failure" } and the test would pass.

This pattern was everywhere. Functions called, return values checked for existence but never for correctness. Roughly 40% of their test assertions were some variation of toBeDefined(), toBeTruthy(), or snapshot tests that nobody reviewed when they updated.

The mutation test

I wanted to make this concrete for the team, so I ran a small experiment. I picked their order pricing module — maybe 200 lines — and introduced three deliberate bugs:

1. Changed a discount calculation from price * (1 - discount) to price * (1 + discount) (discounts now increase the price)
2. Swapped a >= to > in the free shipping threshold check
3. Removed a tax rounding step that was supposed to round to two decimal places

All three mutations left the test suite green. Ninety-one percent coverage, zero bug detection.

The room got quiet when I showed this. The tech lead asked, "How is that possible? We test that function." They did. They tested that it ran. They never tested that it calculated correctly.

How this happens

Nobody sets out to write useless tests. This codebase didn't get here because the engineers were lazy. It got here because of three pressures that reinforced each other.

The coverage gate. Their CI pipeline failed any PR below 85% coverage. This is a well-intentioned rule, but it creates a perverse incentive. When you're trying to ship a feature and the pipeline blocks you because your new function dropped coverage to 83%, the fastest fix is a test that calls the function and asserts nothing meaningful. You're writing tests to satisfy a metric, not to catch bugs.

Snapshot creep. About a third of their tests were snapshot tests. Snapshot testing is fine for catching unintended changes to serialized output — API responses, rendered components, that sort of thing. But their snapshots had grown to include entire database query results, full HTTP response bodies, and complex nested objects. When a snapshot broke, the developer would look at the diff, decide it was "expected," and update the snapshot. Over time, updating snapshots became a reflex, not a review.

Copy-paste test patterns. Their test files had a template. New function? Copy an existing test, change the function name and inputs, done. The template had expect(result).toBeDefined() as its assertion. Nobody questioned it because it was "the pattern."

What actually changed

We didn't throw out the test suite. That would have been demoralizing and wasteful — the setup code, the fixtures, the test infrastructure, those were all valuable. Instead, we focused on three changes.

First, we replaced the coverage gate with a mutation testing step on critical paths. We used Stryker for this. Instead of asking "did the tests execute the code?", mutation testing asks "if I break the code, do the tests notice?" It's slower than coverage, so we only ran it on the pricing engine, the permissions module, and the billing integration — the places where bugs actually hurt.

Second, we did a triage of existing snapshot tests. Every snapshot over 20 lines got flagged for review. Most were replaced with targeted assertions on the two or three fields that actually mattered. The team went from 400+ snapshot files to about 60.

Third — and this was the smallest change with the biggest impact — we added a linting rule that flagged bare toBeDefined() and toBeTruthy() assertions. Not banned them outright, because sometimes you genuinely just want to check that a value exists. But the lint warning forced developers to make a conscious choice: "Am I asserting existence because that's all I care about, or because I don't know what to assert?"

The part that surprised me

Within two months, their test suite was actually catching bugs in CI. Real bugs, not flaky failures. The mutation score on their pricing engine went from 12% (meaning 88% of injected bugs went undetected) to 74%. Total coverage dropped to 79% because we deleted a lot of empty-calorie tests. Nobody missed them.

The thing that surprised me was what the team said afterward. They told me writing tests became less annoying, not more. When the goal was "get coverage up," writing tests felt like paperwork — something you did because the pipeline demanded it. When the goal shifted to "would this test actually catch a mistake?", the tests started feeling like they had a point. Developers wrote fewer tests, but each one did something.

The metric trap

Coverage is a useful signal when it's low. If you're at 15%, you probably have entire code paths that have never been exercised by a test. Getting to 50% or 60% is genuinely valuable.

But past a certain point, coverage stops correlating with quality and starts measuring busywork. I've seen this at multiple clients now. The teams with the highest coverage numbers often have the least effective test suites, because the incentive structure rewards quantity of execution over quality of verification.

If I could pick one metric to replace coverage thresholds, it would be mutation score on critical paths. It's harder to game, it answers a better question, and it forces you to think about what each test is actually proving.

But I'd settle for teams just asking themselves one question before every assertion they write: if this code were wrong, would this test tell me?