The Security Scanner We Trained Ourselves to Ignore

A client added a SAST scanner to their CI pipeline after a compliance audit. Six months later, nobody looked at the findings. Then a real SQL injection shipped to production.

The compliance auditor's report was polite but firm: "No evidence of static application security testing in the development workflow." The client — a fintech company processing about $80M in annual transactions — needed SOC 2 Type II certification by Q3, and this was a blocker. So they did the obvious thing. They added Semgrep to CI, pointed it at the monorepo, and opened the results.

827 findings.

The Slack message from the tech lead landed about fourteen minutes later: "Is this thing broken?"

The first week

The team was a dozen engineers maintaining a Django backend and a React frontend. They'd been shipping for three years without any SAST tooling. The codebase wasn't terrible — they had code review, decent test coverage, a linter for style issues. But nobody had ever run a security-focused scanner against it.

The 827 findings broke down roughly like this: about 340 were informational (things like "consider using parameterized queries" on code that already did), around 290 were low-severity style issues the security community argues about endlessly, roughly 150 were medium-severity findings that needed actual review, and 47 were flagged high or critical.

The CTO wanted a plan. The tech lead wanted to ship the feature they'd promised for next sprint. The compliance consultant wanted a dashboard showing findings trending downward.

So they triaged. Two engineers spent a full week going through the 47 critical findings. They fixed 12 genuine issues — a couple of places where user input wasn't being sanitized properly, an overly permissive CORS configuration, a hardcoded API key in a test utility that had accidentally been copied into a service module. The remaining 35 were false positives or findings so theoretical they required an attacker to already have shell access to exploit.

They suppressed the false positives with inline comments, fixed the real ones, and called it done.

The slow slide

Here's what happened over the next five months. The scanner kept running on every PR. Engineers kept seeing findings. And a pattern set in that I've now seen at four different clients.

First, engineers started treating scanner output the way most people treat cookie consent banners — something to click past. The scanner flagged a new finding on a PR? Check if it's in the diff or pre-existing. If pre-existing, ignore. If in the diff, glance at it for three seconds. If it mentions "potential" or "possible," suppress it.

Second, the suppress-comment count grew. By month three, the codebase had 168 # nosec or # nosemgrep annotations. Nobody tracked whether those suppressions were justified. The original two engineers who'd done the triage had moved to different projects within the company.

Third, the CI pipeline got slower. Semgrep added about 90 seconds to each run. Not catastrophic, but enough that engineers noticed. When the build was already taking seven minutes, that extra minute and a half felt personal. Someone opened a ticket to "optimize CI performance" and the proposed solution was to move the scanner to a nightly job instead of running per-PR.

They didn't do that — the compliance requirement kept it in the PR pipeline. But the sentiment was clear. The scanner had gone from "important security tool" to "that thing that slows down CI and tells us stuff we already dismissed."

The finding that wasn't false

Six months after the initial rollout, an engineer added a reporting endpoint. The feature let account admins export transaction data with date range filters. The implementation used Django's ORM for most of the query, but the date range filtering was built with raw SQL because the engineer wanted a specific PostgreSQL date-truncation behavior that was awkward to express through the ORM.

def get_transactions(request):
    start = request.GET.get("start_date")
    end = request.GET.get("end_date")
    query = f"""
        SELECT date_trunc('day', created_at) as day,
               sum(amount), count(*)
        FROM transactions
        WHERE account_id = %s
        AND created_at BETWEEN '{start}' AND '{end}'
    """
    with connection.cursor() as cursor:
        cursor.execute(query, [request.user.account_id])
        return cursor.fetchall()

The scanner flagged it. "Possible SQL injection via string formatting in raw SQL query." The PR reviewer — an experienced engineer who'd been with the company for two years — saw the finding, saw that account_id was properly parameterized, and assumed the scanner was being its usual noisy self. The PR was approved and merged.

The start and end parameters were injected directly into the query string. Classic SQL injection, textbook-grade. The account_id was parameterized correctly, which made the code look safe at a glance. But the date parameters were just string-formatted in.

A penetration tester found it three weeks later during the SOC 2 assessment. The irony was thick enough to taste.

What we actually changed

I got brought in after the pen test report landed. The fix for the SQL injection was trivial — parameterize the date inputs, done in twenty minutes. The harder problem was that the team had a security tool, a security process, and a compliance requirement, and a textbook vulnerability had still walked right through all of it.

We did three things that actually moved the needle.

Shrunk the rule set. The default Semgrep configuration was running every rule in the registry that matched their stack. We stripped it down to about 60 rules — the ones that matched real patterns in their codebase and had a low false-positive rate in practice. Finding count dropped from a rolling average of ~40 per week to about 6. When a scanner tells you something six times a week instead of forty, you actually read what it says.

Made suppressions require a reason. Every # nosemgrep annotation needed a one-line explanation and a second reviewer's approval. Not a heavy process — just enough friction that "I looked at this and it's fine" became the minimum rather than a rubber stamp. The existing 168 suppressions got reviewed in two afternoon sessions. Eleven of them shouldn't have been suppressed.

Separated the signal from the noise in CI. High-severity findings blocked the PR. Medium findings showed as warnings that didn't block but were visible in the review interface. Low and informational findings went to a weekly digest instead of inline. Engineers stopped seeing the scanner as an obstacle because it only interrupted them when something genuinely looked dangerous.

The uncomfortable math

Here's what stuck with me. The scanner correctly identified a real SQL injection vulnerability. It did its job. A human looked at the finding and dismissed it. The scanner had been telling this team about problems for six months, and the team had learned — through entirely rational experience — that most of what it said wasn't worth their time.

This is the same dynamic behind alarm fatigue in hospitals, alert fatigue in monitoring, and notification fatigue in every app on your phone. When a system communicates at the wrong signal-to-noise ratio, humans don't start reading more carefully. They stop reading.

The uncomfortable part is that the vendor's default configuration was the problem as much as the team's behavior. Out of the box, the tool was optimized for coverage — finding every possible issue, including marginal ones. That's great for a security research context. For a product engineering team trying to ship features and stay compliant, it produced exactly the conditions where a real finding gets lost.

I keep thinking about how common this setup must be. Gartner reported this year that 58% of software engineers now carry security responsibilities, but most have never had formal security training. We're asking people to interpret security tooling output the way we used to ask dedicated security engineers to — except those engineers had context for separating real findings from noise, and most product engineers don't.

Tools don't fix processes. A scanner with eight hundred findings and no triage strategy is just a way to feel secure without being secure. The real question isn't whether you have a SAST tool in your pipeline. It's whether anyone still believes what it tells them.