Change failure rate: the DORA metric everyone measures wrong

Every engineering leader I talk to can quote their deploy frequency to two decimal places. Cycle time? They've got a dashboard. Change failure rate? Long pause. Then a number that, when you dig into it, turns out to be measuring something between "incidents we remembered to log" and "whatever Datadog paged us about last Tuesday."

Of the four DORA metrics, change failure rate (CFR) is the one teams most often get wrong. Not because it's conceptually hard, but because the definition has more soft edges than the others, and those soft edges are exactly where the data quality dies. It's also the metric most contaminated by CI you can't trust — which is why I want to spend a whole post on it.

What change failure rate actually measures

The DORA definition: the percentage of deployments to production that result in degraded service and require remediation — a hotfix, a rollback, a patch, a forward-fix. Not the count of failures. The rate. Failures divided by total deployments.

CFR = (deployments that caused a failure) / (total deployments) x 100

So a team shipping 200 times a month with 10 incidents is at 5%. A team shipping 4 times a month with 2 incidents is at 50%. Same two failures, wildly different signal. That ratio is the whole point: it normalizes failures against how much you ship, so a high-throughput team isn't punished for volume.

The DORA benchmarks put elite performers at 0–15% and low performers up around 40–60%. Fine as a north star. Useless as a management target if you don't nail down what "failure" and "deployment" mean in your pipeline first.

The measurement problem: what counts as a failure?

Here's where it falls apart. Ask five engineers what a "change failure" is and you'll get five answers:

• Anything that triggered a rollback
• Anything that opened a Sev1 or Sev2 incident
• Anything that needed a hotfix within 24 hours
• Anything a customer complained about
• Anything that broke a dashboard

These produce completely different numbers from the same deploys. If you want CFR to mean anything over time, you need a single, boring, written-down rule. My recommendation: a deployment is a failure if it required an unplanned remediation to production. Rollback, hotfix, feature-flag kill, config revert — all count. A bug found and fixed in the normal backlog does not. A customer complaint with no code change does not.

Then instrument it so a human isn't deciding case by case. The cleanest signal is usually the remediation event itself:

# Tag remediation deploys so CFR is computed from data, not memory
# In your deploy pipeline, require a reason for out-of-band ships
deploy:
  inputs:
    remediation_for: # optional; SHA of the deploy this fixes
      required: false
  # A deploy with remediation_for set marks the referenced deploy
  # as a failure. Everything else is assumed healthy.

The trick isn't the YAML. It's the discipline: every rollback and hotfix references the deploy it's cleaning up after. Now CFR falls out of your deploy log automatically, and you're not relying on someone remembering to file a retro.

What CFR tells you — and what it doesn't

CFR is a quality signal for your delivery process. Trending up over a quarter means something upstream is degrading: review is getting sloppy, test coverage is rotting, or you're shipping bigger, riskier batches. Paired with deploy frequency, it answers the question leadership actually cares about — are we going faster without breaking more? That pairing is the whole reason DORA insists you never read one metric alone. Speed metrics (deploy frequency, cycle time) and stability metrics (CFR, time to restore) are a system. Optimize one, watch the other.

What CFR does not tell you:

• Severity. A 2% CFR where each failure is a four-hour outage is worse than a 12% CFR of one-line forward-fixes nobody noticed. CFR treats every failure as one tick. Pair it with time-to-restore to get the blast radius.
• Root cause. It tells you failures are happening, not why. Could be bad tests, bad rollout tooling, or one service everyone's afraid to touch.
• Where in the pipeline you're bleeding. A failure that reaches prod is a failure that your CI let through. CFR counts the escape; it says nothing about how many near-misses your pipeline caught or how much of your CI signal you actually believe.

That last point is the one leaders skip. CFR is a measure of escaped defects. Its silent partner is the reliability of the gate that's supposed to stop them.

How teams misuse it

Turning it into an individual scorecard. The fastest way to poison CFR is to attribute failures to engineers and put it in a perf review. You'll get a beautiful downward trend and a team that has quietly stopped shipping anything risky, stopped logging incidents as failures, and started calling rollbacks "planned maintenance." DORA is a team-and-system metric. The moment it's personal, it's fiction.

Redefining failure to hit a target. "That wasn't a change failure, it was a known-issue." "That rollback was precautionary." If your CFR dropped last quarter and nothing about your engineering practice changed, you didn't get better — your bookkeeping did.

Comparing teams with different definitions. Team A counts every hotfix; Team B counts only Sev1s. Team A looks 4x worse and gets "investment." You've just rewarded the team with sloppier accounting. CFR is only comparable within a single, shared definition.

Ignoring the denominator. Teams that batch releases have tiny denominators, so a single bad deploy spikes CFR to 25%. The fix is usually more frequent, smaller deploys — which feels counterintuitive when the metric is already "too high." Read CFR next to deploy frequency or you'll make exactly the wrong call.

The flaky-test tax on your stability metrics

Now the part most CFR write-ups skip. Your CFR is downstream of whether you trust your CI.

When tests are flaky, one of two things happens, and both wreck the metric.

First: the team stops trusting red builds. Once "just re-run it" becomes the reflex, every failure — real or flaky — gets a retry. Real regressions ride through on a green re-run and land in production. Those become change failures you could have caught. Your CFR goes up, and it looks like an engineering-quality problem when it's actually a CI-reliability problem. We wrote about why the retry habit quietly corrodes your signal in our piece on quarantining flaky tests.

Second, and sneakier in compliance-conscious shops: your CI gate is a change-management control. If an auditor asks "how do you prevent bad changes reaching production," the answer is "tests must pass." But if half your failures are flaky and the documented remediation is rerun until green, that control isn't doing what your SOC2 narrative says it does. A flaky gate is a control that fails open. CFR is the metric that eventually surfaces that gap — as escaped defects.

So before you treat a rising CFR as a signal to slow down and add more process, check the health of the gate first:

• What's your test retry rate? If it's climbing, your green builds are increasingly meaningless.
• What fraction of pipeline failures are non-deterministic? If you don't know, you can't tell an engineering regression from CI noise.
• Are the same tests failing intermittently across unrelated PRs? Classic flake fingerprint.

This is exactly the correlation BuildPulse is built to surface: it identifies which failures are flaky versus real, quarantines the flakes so they stop poisoning the signal, and gives you a clean pass/fail gate you can actually put in an audit narrative. When the gate is trustworthy, CFR goes back to measuring what you wanted — real escaped defects, not CI weather.

A workable measurement setup

If you're starting from scratch, here's the order I'd do it in:

1. Write the definition down. One sentence. "A production deployment that required unplanned remediation." Publish it.
2. Instrument the denominator. Count deploys to prod automatically from your CD tool. Don't estimate.
3. Instrument the numerator from remediation events, not human recall. Tag rollbacks and hotfixes with the deploy they're fixing.
4. Clean your CI signal first. Measure retry rate and flake rate. A CFR built on top of a gate nobody trusts is measuring your patience, not your quality.
5. Always show CFR next to deploy frequency and time-to-restore. Never on its own slide.
6. Keep it at the team level. Never on an individual dashboard. Ever.

Do that and CFR becomes what DORA intended: a quiet, honest number that tells you whether your delivery is getting more or less safe as it gets faster. Get lazy about the definition or let a flaky pipeline pollute the inputs, and you've got a vanity metric that goes down right up until the outage that proves it was lying.

Measure the gate before you measure the escapes.