How much CI time flaky test reruns actually burn: a benchmark

Nobody budgets for attempt two

Every engineering org I've worked with can tell you their CI bill to the dollar. Almost none can tell you what fraction of that bill is attempt two. Reruns don't get their own line item. They hide inside the same meter as legitimate work, so a pipeline that runs twice looks exactly like two pipelines that ran once.

So we sat down and priced it. This post is the benchmark version of an argument we've made before: retries are a tax, and the tax rate is higher than you think. If you'd rather skip to the punchline: for a mid-size org with a moderately flaky suite, rerun-driven CI minutes routinely land between 20% and 60% of total test-related compute, and the math that gets you there is embarrassingly simple.

The model, and why it's honest

Here's the core mechanic. If each test in your suite has some small per-run probability of flaking, call it p, and your suite has n tests, then the probability that a given build hits at least one flaky failure is:

P(flaky build) = 1 - (1 - p)^n

That exponent is the whole story. Flakiness compounds with suite size. A per-test flake rate of 0.05% sounds like a rounding error. Across 2,000 tests, it means roughly 63% of your builds contain at least one flaky failure. Your tests are individually reliable and your builds are collectively not.

A few notes on why this model is fair rather than scary marketing math:

• The per-test rates we use are conservative. Google's published data put their overall flaky-test incidence around 16% of tests exhibiting some flakiness; the per-test, per-run rates we model (0.01% to 0.1%) are far below what we see in real suites uploaded to BuildPulse, where a handful of hot tests often flake at 1–5% per run on their own.
• We assume flakes are independent. In reality they cluster (shared fixtures, port collisions, a saturated CI host), which makes bad days worse than the model predicts, not better.
• We only count compute. No engineer-attention cost, no merge-queue latency, no context switching. Just minutes on the meter.

If anything, treat these numbers as a floor.

Benchmark: probability a build hits a flake

First table: the probability that any given CI run contains at least one flaky failure, across suite sizes and per-test flake rates.

Suite size	p = 0.01%	p = 0.05%	p = 0.1%
500 tests	4.9%	22.1%	39.3%
2,000 tests	18.1%	63.2%	86.5%
5,000 tests	39.3%	91.8%	99.3%
10,000 tests	63.2%	99.3%	~100%

Read that middle column again. At a per-test flake rate of one in two thousand, a 5,000-test suite fails spuriously on more than nine out of ten runs. This is why "we'll just clean up the flaky tests next quarter" doesn't survive contact with a growing codebase: you can hold p constant through heroic discipline and still watch build-level flakiness climb, because n keeps growing.

There's a threshold worth marking. Once P(flaky build) crosses 50%, the expected number of full-suite reruns per build, which is P/(1-P) when every retry rolls the same dice, exceeds one. Past that line you are, on average, spending more CI compute on reruns than on the original signal.

Benchmark: CI minutes burned per month

Probabilities are abstract. Minutes are money. Let's fix a reference org: 3,000 CI builds per month (a 150–300 engineer company with normal PR traffic gets there easily), full-workflow retry as the recovery strategy, and test-stage compute that scales with suite size. We'll assume 15 billed CI minutes per full test run at 500 tests, 50 at 2,000, 120 at 5,000, and 240 at 10,000. Billed minutes, not wall clock: sharding makes the pipeline faster and the meter spin exactly as fast.

Extra CI minutes per month from reruns (first retry only):

Suite size	p = 0.01%	p = 0.05%	p = 0.1%
500 tests	2,200	9,900	17,700
2,000 tests	27,200	94,800	129,800
5,000 tests	141,500	330,500	357,500
10,000 tests	455,000	715,000	720,000

And because a full-suite retry can itself flake, the true expected overhead is geometric: P/(1-P) instead of P. At 63% flaky builds, that's not a 63% overhead. It's 172%.

At GitHub's standard Linux runner pricing of $0.008 per minute, the 5,000-test, p = 0.05% cell is about $2,600 a month in pure rerun compute, and over $10,000 if you're on 8-core runners. Real money, but honestly not the number that should scare you. The number that should scare you is the overhead percentage, because it applies to everything downstream: queue depth, runner concurrency limits, merge latency, and how long your engineers sit refreshing a checks page.

One clean way to remember all of this: with a single full retry, your rerun overhead percentage is approximately your flaky-build probability. If a third of your builds contain a flake, you're paying roughly a third more for CI than the work requires.

The retry ladder, priced

Not all retries cost the same. There's a ladder, and most teams are standing on the most expensive rung.

Rung 1: rerun the whole workflow. Someone clicks "Re-run all jobs," or worse, your merge queue does it automatically. You pay the full test-run cost again, including every shard that passed. This is the model above. It's also the default human behavior, because clicking one button is easier than reading a log.

Rung 2: rerun failed jobs only. With gh run rerun --failed or the equivalent UI button, you pay for one shard instead of eight. For our reference org (4,000 tests, 8 shards, 33% flaky builds), that drops rerun compute from roughly 95,000 minutes a month to about 12,000. An 87% reduction from changing which button people click. If you do nothing else after reading this post, do this.

Rung 3: retry failed tests in-process. Jest's jest.retryTimes, pytest-rerunfailures, Go's community retry wrappers. Compute cost: seconds. This rung looks like a free lunch, which is exactly the problem.

Here's rung 3 as most teams deploy it:

- name: Run tests (with retries)
  uses: nick-fields/retry@v3
  with:
    max_attempts: 3
    timeout_minutes: 30
    command: npm test

This makes the job green and makes the flake invisible. No failure ever reaches your test reports, so nothing ever shows up in your flaky-test tracking, so the underlying rate p quietly climbs while your dashboard says everything is fine. We've written before about why blanket test retries hide the problem they're papering over; the short version is that retries without reporting are how a suite gets to that 91.8% cell without anyone noticing the drive.

The defensible position is rung 3 with instrumentation: retry failed tests, but emit every attempt to your JUnit XML and ship it somewhere that tracks pass-on-retry as a first-class signal. That's precisely the failure-then-pass pattern BuildPulse uses to detect flaky tests automatically, and it turns your retry mechanism from a cover-up into a sensor.

The costs the meter doesn't show

Two costs deserve a paragraph each because they don't appear in any table above.

Latency, not just compute. A rerun doesn't start the moment a build fails. It starts when a human notices the failure, decides it's "probably the flaky one," and clicks. In practice that gap is 10 to 45 minutes of a PR sitting red. Multiply by your flaky-build rate and your PR volume and you get days of aggregate merge delay per week. Faster runners shrink the rerun itself (this is half the pitch for BuildPulse's GitHub Actions runners, and cheaper minutes do soften the tables above), but no runner is fast enough to fix the part where a person had to notice first.

Audit posture. If you operate under SOC2 or ISO 27001 and your CI gate is part of change management, every "passed on attempt 3" is a control that fired and was overridden by a click. Your auditors are reasonable people, and "the test is known-flaky, tracked, and quarantined with an owner and a ticket" is a fine answer. "We rerun until green" is a much less fine answer, and it's the honest description of most orgs' process. A documented quarantine workflow converts the first answer from aspiration to evidence.

Run it against your own pipeline

Don't take our reference org's numbers. Price your own. You need three inputs: builds per month, billed minutes per full test run, and your flaky-build probability, which you can estimate directly as the fraction of failed builds that pass on rerun with no code change.

def rerun_cost(builds_per_month, run_minutes, flaky_build_prob,
               strategy="full", shards=1, price_per_min=0.008):
    p = flaky_build_prob
    if strategy == "full":
        # every retry reruns everything and can itself flake
        expected_extra = (p / (1 - p)) * run_minutes
    elif strategy == "failed_jobs":
        expected_extra = p * (run_minutes / shards)
    else:  # in-process test retries
        expected_extra = 0.0
    monthly_minutes = expected_extra * builds_per_month
    return monthly_minutes, monthly_minutes * price_per_min

minutes, dollars = rerun_cost(3000, 96, 0.33, strategy="full")
print(f"{minutes:,.0f} CI min/month, ${dollars:,.0f}")
# 141,672 CI min/month, $1,133

If you don't know your flaky-build probability, that's the first finding. Most teams can't answer it because reruns are launched from five different places (UI clicks, merge-queue automation, in-process retries, a Slack bot someone built in 2022) and nothing aggregates them. Instrumenting test reports so every attempt is recorded is how you get the denominator back.

What to actually do with these numbers

Three moves, in order of effort:

• This week: switch humans and automation from "re-run all jobs" to failed-jobs-only reruns. It's a policy change and a habit, and it cuts the compute tax by whatever your shard count is.
• This month: make retries visible. Every retry attempt lands in your JUnit output; pass-on-retry becomes a tracked metric with a trend line, not folklore.
• This quarter: attack p itself. Quarantine the worst offenders so they stop gating merges, fix them with the reproduction data you're now collecting, and watch the exponent work in your favor for once. Cutting per-test flakiness from 0.05% to 0.01% takes our 5,000-test org from 92% flaky builds to 39%, and from a 330,000-minute monthly tax to 141,000. Same suite, same tests, less than half the burn.

The benchmark's real lesson isn't any single cell in the tables. It's that rerun cost is a function of two variables you control, per-test flake rate and retry strategy, and that both respond quickly to attention. The tax is optional. Most teams just haven't opened the bill.