Change failure rate: the DORA metric everyone quotes and nobody measures right

The metric that sounds simple and isn't

Here's a number I've watched slide across a dozen leadership decks: "Our change failure rate is 8%." Nods around the room. Somebody writes it down. Nobody asks the two questions that matter — what counts as a change, and what counts as a failure.

Change failure rate (CFR) is one of the four DORA metrics, alongside deploy frequency, lead time for changes, and time to restore. On paper it's the cleanest of the four: what percentage of your deployments cause a problem in production that needs a fix — a hotfix, rollback, or patch. In practice it's the one teams measure worst, because both halves of the fraction are squishy, and because a flaky CI signal poisons it from the source.

If you only internalize one thing: CFR is a quality-of-delivery metric, not a productivity metric. Treat it like a velocity number and you'll optimize for exactly the wrong behavior.

How to actually measure it

The formula is boring:

change failure rate = failed changes / total changes deployed to prod

The hard part is defining both terms so the number means the same thing month over month. Pin these down before you compute anything:

What is a "change"? For most teams the honest unit is a production deployment. Not a merge to main, not a PR — a deploy that reached users. If you deploy ten times a day, your denominator is deploys, and it should tie directly to your deploy frequency metric. If those two numbers come from different systems, you already have a problem.

What is a "failure"? A change that required remediation: a rollback, a hotfix, a forward-fix within some window, a feature flag kill, or an incident. The key word is required. A log line you ignored isn't a failure. A P4 ticket filed three weeks later isn't a failure of that deploy.

The cleanest way to compute this is to tag deploys and correlate them with incidents or rollbacks. If you deploy through GitHub Actions, emit a deployment marker on every prod release:

- name: Record deployment
  run: |
    curl -X POST "$METRICS_URL/deployments" \
      -H "Content-Type: application/json" \
      -d '{
        "sha": "${{ github.sha }}",
        "service": "checkout",
        "env": "production",
        "deployed_at": "'$(date -u +%FT%TZ)'"
      }'

Then every rollback or hotfix references the deploy it's fixing. A hotfix PR title or commit trailer like Fixes-deploy: <sha> gives you a machine-readable link. Now CFR is a join, not a vibe.

Here's a rough version of the join in Python, assuming you've got both tables:

def change_failure_rate(deploys, remediations, window_days=30):
    recent = [d for d in deploys if within(d.deployed_at, window_days)]
    failed_shas = {r.target_sha for r in remediations}
    failed = [d for d in recent if d.sha in failed_shas]
    return len(failed) / len(recent) if recent else None

Note the None when there are no deploys. A team that shipped nothing this month doesn't have a 0% failure rate — it has no data. Reporting 0% there is how you get a great-looking dashboard for a team that's frozen solid.

What CFR does tell you

When you measure it honestly, CFR is one of the better proxies for whether your delivery process is actually safe to run fast. The DORA research pairs it with the throughput metrics on purpose: elite teams deploy often and keep CFR low. That combination is the whole point. Deploy frequency without CFR is a team shipping bugs quickly. CFR without deploy frequency is a team that ships twice a year and calls it quality.

Read together, they tell you:

• Whether your test suite and review process catch real problems before prod.
• Whether small batch sizes are paying off (they usually lower CFR — smaller changes fail less catastrophically and are easier to diagnose).
• Whether a specific service or team is carrying disproportionate risk.

That last one is where CFR earns its keep for an engineering leader. Sliced per service, it surfaces the checkout path that fails one deploy in five while the marketing site cruises at 2%. That's a resourcing conversation, not a shaming one.

What CFR does not tell you

It does not tell you how good your engineers are. I've seen CFR weaponized in exactly this way and it's corrosive. A high CFR on a payments service handling gnarly edge cases is not the same signal as a high CFR on a static docs site, and comparing the two as if engineers were interchangeable is how you teach people to stop deploying.

It does not tell you severity. A 5% CFR made entirely of instant rollbacks with zero customer impact is a healthy system. A 5% CFR where every failure is a four-hour outage is a fire. CFR is a count, not a weight. Pair it with time-to-restore or you're reading half the sentence.

And it does not tell you why things failed. That's a job for the remediation data underneath, not the ratio on top. CFR is a smoke detector. It's not the investigation.

The ways teams quietly ruin it

Counting the wrong denominator. Teams that count merges instead of prod deploys inflate the denominator and artificially deflate CFR. Ten merges bundled into one deploy that breaks becomes "1 failure out of 10" instead of "1 out of 1." Your number looks great and means nothing. Pick deploys, stick with it.

Redefining "failure" until it disappears. The most common form: a team under pressure to improve CFR starts reclassifying incidents. The rollback becomes a "planned config change." The 2 a.m. hotfix becomes "routine maintenance." Nobody lies outright — they just relabel. Six months later the number is beautiful and useless. If your CFR improves while incidents don't, you're measuring your labeling discipline, not your delivery quality.

Comparing teams that shouldn't be compared. CFR is a within-team trend metric. Cross-team leaderboards optimize for the wrong thing every time. The team at the bottom will get the number down by deploying less — which quietly tanks deploy frequency and lead time. You've traded a visible metric for two invisible ones. For more on how these metrics interact rather than trade off, see our post on reading DORA metrics as a system, not a scorecard.

Letting flaky tests corrupt the input. This is the one that keeps me up. CFR depends on a trustworthy CI signal, and a flaky suite makes that signal a coin flip. Here's how it breaks in both directions:

• A flaky test goes red on a good deploy. Someone reruns, it passes, ships. If your remediation tracker counts that rerun-and-hotfix as a failure, your CFR is inflated by noise.
• Worse: teams get so used to red builds meaning "flaky, just rerun" that they rerun through a real failure and ship the bug. Now CFR undercounts, because the failure sailed past a gate everyone stopped trusting.

When your team can't tell a real failure from a flaky one, every downstream metric inherits the ambiguity. CFR isn't measuring your code anymore — it's measuring your suite's honesty. We wrote about that trust erosion in detail in why flaky tests are a CI reliability problem, not a testing problem, and it applies directly here.

Fix the signal before you fix the metric

You can't compute an honest change failure rate on top of a CI pipeline you don't trust. The order of operations matters: stabilize the signal, then measure delivery quality on top of it.

That means quarantining known-flaky tests so a red build reliably means "something is actually broken," and tracking flakiness as its own first-class metric rather than folding it into CFR. This is where a flaky-test management layer earns its place in the pipeline — BuildPulse detects flaky tests from your existing JUnit or test output, quarantines them automatically, and keeps them out of the pass/fail signal your CFR depends on. Once a red build means a real failure again, your change failure rate starts describing your code instead of your test suite's mood.

In a compliance-conscious shop this is more than hygiene. If your CI gate is part of a documented change-management control, "we reran it until it passed" is an audit finding waiting to happen. A clean, explainable failure signal is the difference between a control that works and one that just looks like it does.

A workable starting point

If you're standing this up from scratch:

1. Define a change as a prod deploy and instrument every deploy with a marker.
2. Define a failure as a rollback, hotfix, feature-flag kill, or incident that references a deploy.
3. Quarantine flaky tests so reruns and false reds stop leaking into your failure count.
4. Report CFR alongside deploy frequency and time-to-restore, per service, as a trend — never as a cross-team ranking.
5. Audit your failure classifications quarterly. If CFR improves while incident count doesn't, your definitions drifted.

Done this way, change failure rate stops being a slide-deck ornament and becomes what DORA intended: an early read on whether you can keep shipping fast without lighting production on fire. Done the lazy way, it's just a number that goes down every time your team gets scared to deploy — which is the opposite of what you wanted.