Code coverage policy that survives contact with reality

The 100% coverage badge that shipped a broken checkout

I once inherited a service with a green 96% coverage badge in the README. Beautiful number. Executives loved it. Two weeks after I joined, a refund flow silently double-charged customers in production for six hours before anyone noticed.

The code that caused it? Fully covered. Every line executed by a test. The test just never asserted the amount was correct — it asserted the function returned without throwing. Coverage counted the line as tested. Reality disagreed.

That's the whole problem with coverage in one story. It measures whether code ran during your tests, not whether your tests would notice if that code were wrong. Those are very different questions, and most coverage policies conflate them.

If you're an engineering leader being asked to "set a coverage target," this post is for you. Let's talk about when the number is worth trusting, when it's actively lying to you, and how to write a policy that improves your CI signal instead of decorating a dashboard.

What code coverage actually measures

Coverage tools instrument your code and record which parts executed while your tests ran. The common flavors:

• Line coverage — did this line execute? The weakest and most common.
• Branch coverage — did both sides of every if execute? Meaningfully harder to fake.
• Statement / function coverage — coarser variants of the same idea.

Here's the part that trips people up: coverage is a measure of execution, not verification. Consider this Python test.

def apply_discount(price, pct):
    return price - (price * pct / 100)

def test_apply_discount():
    apply_discount(100, 10)  # no assertion

That test gives you 100% line coverage of apply_discount. It would also pass if the function returned price * 9000. Coverage went up. Confidence should not have.

This isn't a corner case. Assertion-free (or assertion-thin) tests are everywhere, especially in codebases that adopted a coverage gate after the fact and backfilled tests to hit a number. When the incentive is the percentage, people optimize the percentage.

When coverage actually correlates with quality

Coverage isn't worthless. It correlates with quality under specific conditions:

• Low coverage is a reliable negative signal. A module at 12% coverage is genuinely under-tested. You don't need nuance to act on that. Coverage is great at finding the code nobody tests at all.
• Branch coverage on logic-dense code. For a pricing engine, a permissions check, a state machine — branch coverage tells you whether your tests exercise the decision paths. That's where bugs live.
• When paired with real assertions and code review. Coverage plus a culture that reviews what the test asserts is a decent proxy. Coverage alone is not.

The useful mental model: coverage tells you what you definitely haven't tested. It tells you almost nothing about what you have. Treat it as a floor detector, not a quality score.

When coverage lies to you

High aggregate coverage is where leaders get burned. A few failure modes I see repeatedly:

The assertion-free test. Covered in the example above. The line ran; nobody checked the result.

Coverage of trivial code. Getters, setters, DTOs, generated code, and toString() methods are cheap to cover and inflate your average. A team can hit 85% while leaving the gnarly 15% — the concurrency, the retry logic, the money math — completely untested. The average hides exactly the code you should worry about.

Coverage that hides flaky tests. Here's the one nobody puts on the slide. A test can contribute to coverage and be flaky. It executes the code (coverage counts it) but passes or fails nondeterministically. Your coverage number looks stable while your CI signal rots. If half your "passing" suite gets rerun until green, your coverage metric is measuring code exercised by tests you don't actually trust. We've written before about how flaky tests quietly destroy trust in CI — coverage does nothing to catch that, and can actively mask it.

Gaming under pressure. Set a hard 90% gate with no nuance and you'll get 90%. You just won't get better tests. You'll get assert True, disabled edge cases, and # pragma: no cover sprinkled on anything inconvenient.

Diff coverage beats total coverage for policy

If you take one thing from this post: stop gating on total coverage. Gate on diff coverage.

Total coverage is a legacy number. It's dominated by years of accumulated code, moves glacially, and punishes teams for debt they didn't create. Telling a team "get the monolith from 61% to 75%" is a quarter-long slog with no clear owner and no clear payoff.

Diff coverage (also called patch or changed-lines coverage) asks a sharper question: of the lines this PR changed, how many are covered by tests? That's the question you actually care about at review time. New code is where new bugs enter. It's where the author still has context. And it's enforceable per-PR without a migration project.

A sane starting policy:

• Require 80% diff coverage on changed lines to merge.
• Let total coverage drift naturally upward as new code carries its weight.
• Exempt clearly-generated and vendored paths so you're not measuring noise.

Here's a GitHub Actions sketch that fails a PR when new code is under-tested:

name: coverage
on: pull_request

jobs:
  diff-coverage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # need history for the diff
      - name: Run tests with coverage
        run: |
          pytest --cov=app --cov-report=xml
      - name: Enforce diff coverage
        run: |
          pip install diff-cover
          diff-cover coverage.xml \
            --compare-branch=origin/main \
            --fail-under=80

The developer feedback loop is tight: change code, tests must cover the change, done. No one is arguing about the monolith's historical debt in a code review at 5pm.

How to write a coverage policy people won't game

A policy is a set of incentives. Design it like one.

1. Prefer branch coverage over line coverage where your tooling supports it. It's harder to fake and it targets logic. In JS, that's the branches threshold, not just lines.

{
  "coverageThreshold": {
    "global": {
      "branches": 75,
      "functions": 80,
      "lines": 80
    }
  }
}

2. Gate on the diff, report on the total. Enforce diff coverage as a merge requirement. Show total coverage as a trend on a dashboard, not a gate. Trends inform; gates enforce. Don't confuse the two.

3. Exclude what shouldn't be measured. Generated clients, migrations, protobufs, plain data classes. Measuring them inflates or deflates your number without telling you anything. Be explicit and put it in config so it's reviewable.

4. Review assertions, not just coverage. Add one line to your review checklist: does this test fail if the behavior is wrong? A covered line with no meaningful assertion should get the same comment as no test at all. This is the cultural half of the policy, and it's the half that actually moves quality.

5. Don't chase 100%. The cost curve goes vertical past ~85% for most services. The last 15% is error handling for conditions that can't occur, defensive branches, and framework glue. Chasing it burns engineering time and breeds pragma: no cover. Set the bar where marginal effort still buys marginal safety.

6. Watch coverage alongside test reliability, not in isolation. A rising coverage number on top of a flaky suite is a false comfort. Coverage measures reach; reliability measures whether the signal means anything. You want both, tracked together. This is where a tool like BuildPulse fits — it correlates coverage with flake rates and quarantines the tests that are inflating your numbers while eroding trust, so the coverage you report is coverage you can stand behind in a change-management review.

The compliance angle nobody enjoys

If your CI gates are part of a SOC2 or ISO 27001 change-management control, coverage policy stops being purely an engineering preference. An auditor may treat your coverage gate as evidence that changes are tested before merge.

Which makes the assertion-free test a control gap, not just a code smell. "We require 80% coverage" reads well in a controls doc. "Our coverage includes tests that assert nothing" reads badly in an incident postmortem. Diff coverage with real assertion review gives you a control you can actually defend — every change is tested, and the tests would catch a regression. That's a far stronger statement than an aggregate percentage that could be half padding.

The short version

Coverage is a proxy, and proxies drift from the thing they measure the moment you make them a target. Use it for what it's good at:

• Low coverage = a real, actionable gap. Trust it.
• High coverage = necessary, not sufficient. Verify the assertions.
• Gate on diff coverage, report on total, prefer branches over lines.
• Track coverage next to flake rate, because a green badge on a flaky suite is a story about a broken checkout waiting to happen.

Set the policy to reward tested new code and honest assertions, not to hit a round number on a dashboard. The number was never the point. Trusting your CI signal was.