Every green rerun is a receipt
Here's a scene you can probably timestamp from your own org. A senior engineer opens a PR, CI fails on a test she's never touched, she hits "Re-run failed jobs," gets green, merges, and moves on. Total elapsed time: 40 minutes. Total learning: zero. Somewhere in a config file, someone eventually gets tired of the ritual and automates it: retryTimes(3) in Jest, --reruns 2 in pytest, a retry wrapper around the whole job in GitHub Actions. The failures stop showing up. Everyone agrees CI is "better now."
Nothing got better. You stopped collecting evidence.
I want to make the case that automatic retries on flaky tests are a tax, and like most taxes, the sticker price is not the real price. There are three costs: CI compute, merge latency, and masked regressions. They're listed in ascending order of how much they should scare you, and descending order of how easy they are to see on an invoice. That inversion is exactly why retry-on-failure survives in so many orgs. The cost you can measure is small, and the costs that matter don't show up on any dashboard you currently have.
Cost one: CI minutes, the number on the bill
Let's do the arithmetic with numbers typical of a 300-engineer company. Say you land 150 PRs a day, each PR runs CI 2–3 times before merge (pushes, rebases, merge queue), and a full pipeline consumes 120 runner-minutes across parallel jobs. If 8% of pipeline runs fail flaky, and your answer is "retry the job," that's roughly 30 flaky reruns a day at 120 runner-minutes each: 3,600 wasted runner-minutes daily, call it 75,000 a month.
At typical hosted-runner pricing that's real money but not board-meeting money: a few thousand dollars a month, less if you've moved to cheaper runners. (If CI spend itself is the fire you're fighting, faster and cheaper GitHub Actions runners attack that directly. They do not make retries a good idea; they make the tax rate lower on a behavior that's still a tax.)
Per-test retries look cheaper because you only rerun the failing test, not the pipeline. But they add a quieter cost: every retried test extends the critical path of its shard, and because retries usually fire on your slowest, most integration-flavored tests, they extend the p95 of your suite, not the median. Your pipeline duration graph gets a fat right tail and nobody can say why.
Here's the thing though: if CI minutes were the whole story, I'd tell you to enable retries and go work on something important. They're not the story. They're the distraction.
Cost two: merge latency, the number that compounds
A flaky failure that triggers a rerun doesn't just cost 120 runner-minutes. It costs wall-clock time on the critical path of a human being's day.
Walk through one incident. The pipeline fails 18 minutes in. The author doesn't see it immediately; they've moved on to the next task, so add 15–25 minutes before they notice. They glance at the failure, recognize the test as "that one," click rerun, and context-switch again. The rerun takes another 25 minutes of wall time. Best case, that PR merged an hour later than it should have, and you burned two context switches from an engineer whose focus is the most expensive resource you buy.
Now multiply. At 8% flaky-failure rate and 150 PRs a day, that's 12 PRs daily eating roughly an hour of avoidable latency each, plus the attention fragmentation, which every study on interrupted work prices at 20+ minutes per switch. If you run a merge queue, it's worse: a flaky failure inside the queue ejects the PR, invalidates speculative builds behind it, and can cascade a single bad coin flip into a 90-minute traffic jam for six unrelated changes.
Auto-retry "fixes" this by hiding the failure inside the pipeline instead of surfacing it to a human. Which sounds like an improvement until you notice what you traded: the latency shrinks a little, and the signal disappears entirely. Nobody notices the test is flaky anymore, so nobody fixes it, so the flake population grows, so you raise the retry count. I have watched a team go from retries: 1 to retries: 3 to "retry the whole job twice" over 18 months, each step ratified in a five-minute Slack thread, none of them ever reverted.
Cost three: masked regressions, the number that pages you
This is the one that should end the debate, and it's pure probability.
Retry-on-failure doesn't distinguish between "this test is flaky" and "this code is flaky." A genuine race condition introduced by today's PR often manifests exactly like a flaky test: it fails sometimes. Suppose a new bug causes a test to fail 30% of the time. Under retryTimes(2), the test must fail three consecutive times for CI to report a failure:
P(CI catches the bug on a given run) = 0.3^3 = 2.7%
P(CI stays green) = 97.3%
Your pipeline just gave a 97% pass rate to a real defect. It ships, it hits production traffic at 1,000x test concurrency, and the intermittent failure your test suite was specifically warning you about becomes a Sev2 at 2 a.m. The postmortem will say "our tests didn't catch it." They did. You configured CI to overrule them.
This is why I call blanket retries a confession. The setting says, out loud, in version control: we know our test signal is unreliable, and rather than find out which tests are lying, we've decided to stop listening when any test says no. For readers in SOC2 or change-management-controlled environments, there's a sharper version of that sentence. If your CI gate is part of your change control story, a silent retry policy is an undocumented weakening of that control. Auditors are generally fine with pragmatic engineering decisions; what they're not fine with is decisions nobody can explain, scope, or point to an owner for. "Any test failure is retried three times, globally, since a 2023 Slack thread" is not a control narrative you want to deliver from memory.
Retries also destroy your data. Once a retry passes silently, most JUnit XML output records a clean pass, and your flakiness measurement drops to zero by definition. You can't manage what your own tooling is actively erasing.
When retries are defensible
I'm not an absolutist. There's a narrow class of tests where retries are the honest engineering answer: tests that cross a network boundary you don't control. A third-party payment sandbox that 503s twice a week is not a defect in your code, you can't fix it, and failing a PR over it is noise. Retrying a job that died because a runner failed to provision is likewise fine; that's infrastructure, not signal.
The difference between defensible and confessional retries is scope. Retry the specific failure mode, not the test, and never the world:
import pytest
# pytest-rerunfailures: retry ONLY on connection errors,
# ONLY for tests that legitimately cross the network
@pytest.mark.flaky(reruns=2, only_rerun="ConnectionError|ReadTimeout")
def test_stripe_sandbox_webhook_roundtrip():
...
Compare that to the pattern I see in the wild, which retries everything, including your pure unit tests, including real regressions:
# The confession, as YAML
- name: Run tests
uses: nick-fields/retry@v3
with:
max_attempts: 3
timeout_minutes: 30
command: npm test
If you keep any retries, hold them to four rules. Scoped: attached to named tests or failure classes, never global. Budgeted: one retry, maybe two, never "until green." Visible: retried outcomes are reported somewhere a human reviews, not swallowed. Expiring: every retry annotation has an owner and a review date, like a feature flag. A retry that's been in place for a year isn't a mitigation, it's furniture.
What to do instead
The alternative to retrying flaky tests isn't "tolerate red builds." It's a boring, effective loop: detect, quarantine, fix.
Detect. You need per-test history across thousands of runs to separate "fails randomly on unrelated PRs" from "started failing when this commit landed." This is exactly the analysis BuildPulse does from the JUnit XML you're already producing; if you're rolling your own, our post on how to detect flaky tests covers what the analysis has to get right. Either way, the prerequisite is that retries stop eating the evidence: if a test needed a retry to pass, that fact must be recorded, not laundered into a green checkmark.
Quarantine, loudly. Quarantining a flaky test removes it from the merge-blocking path while keeping it running and reported. Superficially this resembles a retry: the flake stops blocking PRs either way. The difference is accountability. A quarantined test is on a visible list, has an owner, and shows up in review until it's fixed or deleted. A retried test is invisible, owned by no one, and immortal. Same short-term relief, opposite long-term trajectory.
Fix the top offenders. Flakiness follows a power law; in most suites we see, fewer than 10% of flaky tests cause the majority of flaky failures. A recurring, small allocation (one engineer-day per team per sprint aimed at the top of the list) usually cuts flaky failure rates dramatically within a quarter, because you're fixing the tests that actually fire, not the long tail.
Run your own numbers before your next platform-team planning cycle: reruns per day, runner-minutes per rerun, PRs delayed, and one honest estimate of what a masked race condition costs when it reaches production. Then look at your retry config and ask which of those line items it addresses. The answer is the smallest one.
Retries buy you quiet. Quarantine and fixes buy you trust. Only one of those is worth paying for.
