Code coverage metrics that matter: a policy guide for engineering leaders

The 80% mandate that changed nothing

A few years ago I watched a VP of Engineering respond to a painful production incident with a memo: all services must reach 80% code coverage by end of quarter. Reasonable instinct. The incident happened in a service with 31% coverage, the correlation felt obvious, and the board wanted to hear that something measurable was being done.

Six months later the org hit 82%. The incident rate didn't move. What moved was the shape of the test suite: hundreds of new tests that executed code and asserted almost nothing, a pile of snapshot tests that got blindly re-approved on every diff, and a Slack channel where engineers traded tips on which files to exclude from the coverage report.

Nobody was acting in bad faith. They were responding rationally to a metric that was gate-checked but not understood. That's the trap with coverage: it's the easiest quality metric to collect and one of the easiest to corrupt. This post is about when coverage metrics actually tell you something, when they lie, and how to write a coverage policy that improves quality instead of just improving the number.

Coverage measures execution, not verification

Start with what the metric literally is. A line is "covered" if it ran while your tests were running. That's it. Coverage tools have no opinion on whether anything was checked.

This test produces 100% line coverage of processInvoice:

test("processes invoice", async () => {
  await processInvoice(sampleInvoice);
  // No assertions. If it doesn't throw, it passes.
});

Every branch executed, nothing verified. If processInvoice starts silently computing tax in the wrong currency, this test stays green forever. Multiply this pattern across a codebase under a coverage mandate and you get the worst of both worlds: an impressive dashboard and a test suite that can't catch regressions.

So when a director tells me "we're at 85% test coverage," my honest reaction is: that's one input. It tells me code runs under test. It doesn't tell me the tests would fail if the code were wrong, which is the only property that matters.

When coverage correlates with quality

Coverage isn't useless. It's a real signal in specific situations, and knowing which ones is most of the job.

At the low end, it's a screaming alarm. The difference between 20% and 70% coverage is enormous and meaningful. A service at 20% has entire subsystems where no test has ever observed the behavior. Refactoring there is archaeology with a jackhammer. The correlation between coverage and defect risk is strong at the extremes and mushy in the middle: getting from 25% to 60% usually buys real safety, while getting from 82% to 88% usually buys a nicer chart.

Direction matters more than level. A codebase drifting from 74% to 68% over two quarters is telling you that new code is shipping untested. That trend is worth a conversation regardless of what the absolute number is.

Coverage on new code is a strong signal. Legacy coverage reflects a decade of accumulated decisions, half of them made by people who left. Coverage on the code your team wrote this sprint reflects your current engineering culture. That's the number you can actually manage, and it's the foundation of the policy I'll get to below.

As a discovery tool, it's excellent. Pull up the coverage report for your payment reconciliation module or your permissions layer. If the scary parts of the system are the uncovered parts, you've found next quarter's risk register without running a single workshop. Coverage is a great map. It's a lousy trophy.

When coverage lies to you

Now the failure modes, because they're predictable and you should design your policy around them.

Goodhart's law hits coverage harder than almost any metric. The moment a coverage number becomes a gate, engineers optimize for the number, not the outcome. Assertion-free tests, snapshot tests nobody reads, unit tests for trivial getters, /* istanbul ignore */ sprinkled like seasoning. None of this is malicious. It's what a deadline plus a threshold produces.

Aggregate numbers hide the distribution. A repo at 90% overall can be 99% on utility functions and 20% on the billing state machine. The mean tells you nothing about where the risk lives. If you only look at one coverage number per repo, you're reading the weather report for the whole continent.

Coverage says nothing about test quality. It can't distinguish a sharp, behavior-pinning test from a tautology. Mutation testing gets closer to measuring "would this test catch a bug," but it's expensive enough that most teams run it selectively, if at all. In practice, the guardrail for test quality is code review, and reviewers under a coverage mandate learn to wave tests through.

The last 10% has terrible ROI. Defensive error branches, generated code, framework glue, exhaustive enum switches. Chasing 95%+ means writing brittle tests for code paths that are either unreachable or trivially correct. Those tests then break on every refactor and train your team to treat test failures as noise. Which brings us to a theme that should worry you more than any coverage number: whether your team believes CI when it goes red.

Diff coverage: gate the delta, watch the total

Here's the policy shape that actually works, and it's the one I'd recommend to any engineering leader setting this up today: gate diff coverage (coverage of the lines changed in each pull request), and treat total coverage as an observability metric with no hard gate.

Why diff coverage wins:

• It has no legacy tax. A repo-wide gate punishes whoever touches the oldest code. A diff coverage gate only asks: did you test what you just wrote? That's a fair question on every PR in every codebase, no matter how rough the history.
• It aligns the incentive with the moment. The cheapest time to write a test is while the behavior is fresh in the author's head. Diff coverage puts the pressure exactly there instead of spawning a quarterly "coverage sprint" that produces filler tests.
• It fits code review. An uncovered changed line shows up in the PR, where the reviewer can ask "why is this untested?" while context exists. That conversation is where test quality actually gets enforced.
• It compounds. Hold 80–90% diff coverage for a year and total coverage climbs on its own, concentrated in the code that changes most, which is exactly the code most likely to break.

A minimal GitHub Actions setup for a Python service using diff-cover:

name: test
on: [pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # diff-cover needs history to compare against base

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Run tests with coverage
        run: |
          pip install -r requirements.txt pytest pytest-cov diff-cover
          pytest --cov=app --cov-report=xml

      - name: Enforce diff coverage
        run: |
          diff-cover coverage.xml \
            --compare-branch=origin/${{ github.base_ref }} \
            --fail-under=85

JavaScript shops can do the equivalent with jest --coverage plus a diff-coverage reporter; Go teams can wire go test -coverprofile into the same pattern. The tooling is a solved problem. The policy decision is the hard part, and the policy is: 85% on changed lines, exceptions allowed via an explicit label that a reviewer has to apply, total coverage on a dashboard that leadership reviews monthly but never gates on.

For readers in SOC2 or ISO 27001 environments, diff coverage also happens to be a much cleaner change-management control than a repo-wide threshold. "Every change is tested to a documented standard, enforced automatically, with exceptions logged and approved" is a sentence your auditors will like. A repo-wide gate that engineers routinely bypass with admin merges is the opposite: a documented control with documented evidence that you ignore it.

Flaky tests will eat your coverage gate

Here's the part most coverage policy discussions skip entirely. A coverage gate assumes the tests behind it are deterministic. If they're not, the gate rots from underneath you.

It plays out in two ways. First, when a flaky test fails intermittently, engineers rerun the job until it passes. Once rerun-until-green is the culture, every CI gate, coverage included, becomes advisory. The signal you built the control on is no longer trusted, and in a compliance context, a control nobody trusts is a finding waiting to happen.

Second, flakiness distorts the coverage numbers themselves. When a team skips or deletes a flaky test in frustration, the lines it covered go dark. Weeks later, an unrelated PR touches that file and fails the diff coverage gate for code its author never wrote. Now your gate is generating false positives, and false positives are how gates get disabled.

The fix is to treat flaky tests as first-class defects rather than weather: detect them systematically, quarantine them so they stop blocking merges without silently vanishing, and fix them on a real SLA. This is the problem BuildPulse exists for, and it's why we ended up building coverage tracking into the same platform: coverage metrics and test reliability aren't separate concerns. A coverage number computed from a suite you don't trust is a decoration.

A coverage policy you can defend to engineers and auditors

If you want the condensed version to take into your next platform review, here it is:

• Gate diff coverage at 80–90% on changed lines. Enforce it in CI on every PR. Allow exceptions through an explicit, reviewable mechanism, and audit the exceptions monthly.
• Track total coverage as a trend, not a target. Look at direction and per-module distribution. Never set an org-wide absolute number as a goal; you'll get the number and nothing else.
• Set per-module floors only where risk justifies them. Payment paths, authz, data deletion. Name the modules explicitly rather than gesturing at "critical code."
• Exclude generated code, migrations, and vendored dependencies from the denominator so the metric describes code your team actually writes.
• Review tests, not just coverage. Once a quarter, have senior engineers read a random sample of recent tests and grade whether they'd catch a real regression. This is your defense against assertion-free filler, and it costs an afternoon.
• Pair the coverage gate with a flaky-test policy. Detection, quarantine, and a fix SLA. Without it, the gate's credibility has a half-life measured in months.

Coverage is a map of where your tests have been, not a certificate of what they'd catch. Read it like a map: use it to find the dark corners, gate the code you're writing today, and spend the attention you save arguing about repo-wide percentages on the thing that actually protects production, which is a test suite your engineers believe.