The audit interview you don't want to have
Picture the annual SOC2 audit. Your auditor pulls a sample of twenty merged pull requests and asks you to walk through the change-management evidence for each one. PR number fourteen is a routine change to the billing service. The checks show a failed test suite at 2:14 PM, a rerun at 2:31 PM, another rerun at 2:52 PM, and a green check at 3:07 PM. Nothing in the code changed between attempts.
The auditor asks a reasonable question: "What changed between the failing run and the passing one?"
And the honest answer is: nothing. The test is flaky. Everyone on the team knows it's flaky. Rerunning it is just what you do.
That answer feels harmless in a retro. It lands very differently in an audit interview, because what you've just described is a control that fails nondeterministically and a team that has been trained to keep pressing the button until the control says yes. You didn't verify the change. You outlasted the verification.
I want to make the case that flaky tests are not a developer-experience annoyance you can defer indefinitely. If you're a SOC2 shop, they're a slow leak in one of your named controls, and the longer you tolerate them, the more evidence of that leak accumulates in your own CI logs.
Your CI gate is a CC8.1 control, whether you documented it that way or not
SOC2's CC8.1 is the change-management criterion. Stripped of the framework language, it says: the organization authorizes, designs, tests, approves, and implements changes to systems in a controlled way. When your auditor maps that criterion to your actual engineering process, the mapping almost always includes some version of "all changes pass automated tests in CI before merging to the protected branch."
That sentence probably appears, nearly verbatim, in your system description or your change-management policy. Which means your CI gate isn't just a productivity tool anymore. It's a control. It has the same status as your access reviews and your encryption-at-rest configuration. And controls get evaluated on two axes: design and operating effectiveness.
Here's the uncomfortable part. A test suite with a meaningful flake rate fails on both axes:
- Design: a control that produces different outcomes for identical inputs is not a well-designed control. If the same commit can pass or fail depending on runner load, test ordering, or the phase of the moon, the control isn't actually evaluating the change. It's evaluating the change plus a pile of environmental noise.
- Operating effectiveness: if your engineers routinely rerun failed checks without investigating, the control isn't operating as described. The policy says "changes pass tests." The practice is "changes pass tests eventually, if you're persistent."
No auditor expects zero test failures. Failures that catch real defects are the control working. The problem is failures that everyone, including the person clicking rerun, has already decided to ignore.
Rerun-until-green is override training
The deeper issue isn't any single rerun. It's what a culture of rerunning does to how your engineers relate to the gate.
When a team first encounters a flaky test, the response is investigation. Someone reads the failure, realizes it's a race condition in a test fixture, maybe files a ticket. The tenth time, they skim the failure. The fiftieth time, they don't read it at all. They see red, they click rerun, they go back to Slack. I've watched teams get so fluent at this that rerunning a failed check became muscle memory performed within seconds of the notification, faster than anyone could plausibly have read the failure output.
That's not laziness. It's a rational response to a signal that has stopped carrying information. But notice what the habit actually is: dismissing a failed control without investigation. Your engineers are practicing that motion dozens of times a week. The habit doesn't stay neatly contained to the tests everyone "knows" are flaky, because knowing which failures are flaky requires reading them, and reading them is exactly the step the habit deletes.
Eventually a real failure gets rerun into oblivion. A genuine regression that fails intermittently (because plenty of real bugs are themselves nondeterministic) gets three reruns, goes green on the fourth, and ships. Now you have a production incident whose CI history shows the gate caught the bug and a human overrode it. That's the incident review nobody enjoys writing, and if the change touched anything in scope for your audit, it's also evidence that your change-management control was bypassed in practice.
Normalization of deviance is the textbook term. The CI-specific version is simpler: a team trained to rerun until green is a team trained to override controls, and they will override the one that mattered.
The evidence trail is already written
Here's what makes this different from most process drift: it's fully logged. Every rerun, every admin merge, every branch-protection bypass sits in your CI provider's API, timestamped and attributed. You don't have to speculate about how often the gate gets overridden. You can query it:
# Count workflow runs with multiple attempts on main-bound PRs, last 90 days
gh api "repos/acme/payments-api/actions/runs?per_page=100" \
--paginate \
--jq '[.workflow_runs[] | select(.run_attempt > 1)] | length'
Run that against your busiest repos before your auditor does. If the number is high, you have two options. You can hope the sample of PRs pulled during fieldwork misses the pattern, which is not a strategy. Or you can treat the reruns as what they are: a measurable defect rate in a control you own, with a remediation plan attached.
Auditors are generally reasonable people. They don't expect perfection; they expect that when a control degrades, you notice, you document, and you remediate. "We have flaky tests" is not itself a finding. "We have flaky tests, no inventory of them, no process for handling them, and a rerun rate we've never measured" is how you end up with an exception in the report and an awkward conversation with the customers whose security teams read it.
The defensible alternative: quarantine with a paper trail
The fix is not "delete the flaky tests" (you lose coverage silently) and it's definitely not "add automatic retries everywhere" (that's rerun-until-green with better ergonomics and worse visibility). The fix is a quarantine process with an audit trail, which conveniently is also the thing that makes flaky tests tractable from a pure engineering standpoint. We've written before about why quarantine beats retries for team health; here's the compliance framing of the same process.
A defensible quarantine loop looks like this:
- Detect flakiness from evidence, not vibes. A test is flaky when it produces both pass and fail results on the same commit. That's mechanically detectable from your test reports; it should not depend on an engineer's memory of which tests are "known bad."
- Quarantine the test explicitly: it still runs, its results are still recorded, but it no longer blocks the merge. Crucially, this happens through a tracked change (a PR, a config entry, a dashboard action), not through an engineer clicking rerun at 5:55 PM.
- Ticket and remediate with an owner and an SLA. Quarantine without an exit path is just deletion with extra steps.
- Review the quarantine list on a cadence, the same way you review access grants.
The compliance payoff is that every step produces an artifact. Compare the two stories you could tell in that audit interview:
- Before: "The test failed, so we reran it until it passed."
- After: "The test was flagged as flaky on March 3rd based on conflicting results on identical commits, quarantined via PR #4821 with a linked ticket, fixed on March 11th, and restored to blocking status. Here's the record."
Same flaky test. One story describes a control being overridden ad hoc by whoever was inconvenienced. The other describes a control operating with a documented exception process. Auditors have a deep institutional fondness for documented exception processes.
This is the part of the problem BuildPulse was built for: it watches your test results across builds, identifies flaky tests from actual pass/fail evidence on matching commits, and gives you a quarantine workflow with history you can hand to an auditor instead of a shrug. But the process matters more than the tooling. If you build the loop with a spreadsheet and a weekly review, you're still in vastly better shape than a team with world-class CI and a rerun habit.
What to measure and show your auditor
If you own platform or DevEx at a company where SOC2 is table stakes, a handful of metrics turn this from an anecdote problem into a managed one:
- Rerun rate: percentage of merged PRs that required at least one rerun of a required check. This is your control-override rate, and it should trend down.
- Flaky test inventory: how many tests are currently quarantined, and the age of the oldest. An inventory that only grows is deferred deletion.
- Time to quarantine: how long a test flakes in the blocking suite before someone acts. Every day in that window, it's training your team to ignore red.
- Escape rate: incidents where a rerun masked a real defect. Rare, but each one is a control failure worth a written postmortem.
Put these next to your DORA metrics in whatever leadership review already exists. They belong there: change failure rate and CI trustworthiness are the same conversation viewed from different chairs.
The upside nobody mentions
Here's the part I find genuinely satisfying. Every dollar of this compliance argument buys pure engineering value. A CI gate that engineers trust merges changes faster, because nobody is babysitting reruns. It catches real regressions more reliably, because red means something again. It's cheaper, because reruns are compute you pay for twice (or four times) to learn nothing, and at modern runner prices that adds up faster than most teams track.
So you don't have to choose between the compliance framing and the velocity framing. They're the same fix. The framing just determines who funds it. In my experience, "our flaky tests annoy the engineers" gets a backlog ticket. "Our change-management control has a measurable override rate and here's the query that proves it" gets a quarter of platform time.
Use whichever one works. Then go make your CI gate mean something again, before someone with a sampling methodology asks you to explain why it doesn't.
