How flaky tests are detected and quarantined
BuildPulse flags a test as flaky in one of two ways and can quarantine it automatically. This page gives the exact rules and defaults, what each setting changes, what quarantine does to your CI (nothing, on its own), and how a test leaves quarantine.
The short version
- A test is flagged as flaky if it passes and fails on the same code, with no thresholds, or if its failure rate over the last 30 days is at or above 3% across at least 10 runs, with failures on at least 2 branches. Those three numbers are per-repository settings.
- A test is quarantined automatically only through the failure-rate check: a 30-day failure rate at or above 10%, at least 30 runs in that window, and the test not marked Critical.
- Quarantine is a label in BuildPulse. It does not change your CI result. To keep quarantined tests from failing builds, your pipeline reads the quarantine list from the API. There is a GitHub Actions example below.
- Quarantine does not expire. A test leaves it when someone resolves it, when its linked Jira ticket moves to Done, or when its linked GitHub issue is closed. Passing, however often, never releases it.
What counts as one test
BuildPulse reads the JUnit XML your test runner writes. A test is identified by four values from that file: the suite (the name of the enclosing testsuite), the class name, the test name and the file. If any of the four changes, for example after a rename or a move to another file, BuildPulse treats it as a new test with no history.
Each result is a pass, a failure, an error or a skip. Failures and errors both count as failures. Skipped results are not counted as runs. Detection runs on every upload and only evaluates the tests in that upload, so a test that has stopped running is not re-evaluated.
Detection 1: pass and fail on the same code
Every upload records the git tree SHA of the commit that was tested (git rev-parse <commit>^{tree}), which is a hash of the exact source tree. Two commits with identical contents share a tree SHA, so a re-run of the same commit, or a rebase that changes nothing, counts as the same code.
When a test fails, BuildPulse looks at every result already recorded for that test on the same tree SHA, from any branch, commit, workflow run or upload, including other results in the same upload. If any of them passed, the test is flaky: the code did not change and the outcome did.
- No thresholds apply. One pass and one failure on identical code is enough. Minimum Test Runs and Minimum Branch Count are not used.
- Each failing result caught this way is one disruption. The Disruptions count on a flaky test counts these and nothing else.
- The check runs when the failure arrives. If a test fails first and passes in a later upload of the same code (a workflow re-run that goes green, say), the pair is not linked when the pass arrives. It is linked the next time the test fails on code where it has already passed.
- The upload action reads the tree SHA with git, so the job that uploads needs a checkout of the repository. An upload without a tree SHA skips this check; failure-rate detection still runs.
- This path never quarantines a test automatically, however often it flakes. In the app it is labelled Same-commit conflict.
Detection 2: failure rate over 30 days
For each test in an upload, BuildPulse counts that test's results from the last 30 days:
- Runs are results that passed, failed or errored. Failures are results that failed or errored.
- If runs are fewer than Minimum Test Runs (default 10), stop.
- If failures divided by runs is below the Flaky Detection Threshold (default 3%), stop. The comparison is at or above: a test at exactly 3% is flagged.
- Count the distinct branches with at least one failure in the same 30 days. If there are fewer than Minimum Branch Count (default 2), stop. This passes over a test that fails on one branch only, which usually means a real bug on that branch.
- Otherwise the test is flagged, and its 30-day failure rate is saved as the Failure Rate shown in the app. It is labelled Failure-rate pattern.
A worked example with the defaults: 40 runs in the last 30 days, with 2 failures on main and 1 on feature/login. 3 / 40 = 7.5%, which is at or above 3%; 40 runs is at least 10; 2 branches is at least 2. The test is flagged with a Failure Rate of 7.5%. It is not quarantined, because 7.5% is below 10%.
The window is rolling, so old failures age out of it. It is fixed at 30 days and is not a setting.
Reading the Failure Rate. The number is written each time a test passes the check above. It is not lowered when a test calms down and stops passing the check, so a test that has since become stable can show an older, higher rate. A test caught only by same-code detection shows 0%.
The settings and their defaults
Each repository has its own settings, under Flaky Tests, then Settings. A saved change applies to uploads processed after you save. Nothing already flagged or quarantined is re-evaluated or released when you change a setting.
| Setting | Default | What it changes |
|---|---|---|
| Flaky Detection Threshold | 3% | The 30-day failure rate at or above which failure-rate detection flags a test. Has no effect on same-code detection. |
| Auto-Quarantine Threshold | 10% | The 30-day failure rate at or above which a test flagged by failure rate is quarantined. 0 turns automatic quarantine off. A test has to be flagged first, so a value below the Flaky Detection Threshold behaves like the Flaky Detection Threshold. |
| Minimum Test Runs | 10 runs | Runs inside the 30-day window a test needs before failure-rate detection evaluates it. Set above 30, it also raises the run count automatic quarantine needs. Has no effect on same-code detection. |
| Minimum Branch Count | 2 branches | Distinct branches with at least one failure in the 30-day window that failure-rate detection requires. Has no effect on same-code detection. |
Two numbers are fixed and are not settings: the 30-day window, and the floor of 30 runs before automatic quarantine.
When a test is quarantined automatically
Only failure-rate detection quarantines automatically, and only when all of these hold on an upload that includes the test:
- The Auto-Quarantine Threshold is above 0.
- The test's 30-day failure rate is at or above the Auto-Quarantine Threshold (default 10%).
- The test has at least 30 runs in the 30-day window, or Minimum Test Runs if that is set higher. A test over the threshold with fewer runs stays flagged and is not quarantined until it has them. At 10 runs a single failure is already 10%; at 30 it takes three.
- The test is not marked Critical. Tick Critical Priority on a test's page to keep it out of automatic quarantine. You can still quarantine it by hand.
- The test is not already quarantined.
When that happens, the test is marked quarantined with an entry in its audit log, a Test quarantined Slack alert goes out if that alert is on for the repository, and a Jira ticket is filed if the repository is linked to a Jira project (see Jira).
A released test that still meets these conditions is quarantined again by the next upload that includes it, with a new Jira ticket. To stop that, fix the test, raise the threshold, or mark the test Critical.
Quarantining by hand. Any member of the organization can quarantine a test from its page, several at once from the Flaky Tests list, or from a test run. Manual quarantine has no thresholds and files a Jira ticket under the same conditions as automatic quarantine.
What quarantine does, and what it does not do
Quarantine does:
- Mark the test Quarantined in BuildPulse and record who quarantined it and when.
- Put the test in the quarantined view of the Flaky Tests page and in the API's quarantine list (
GET /api/v1/flaky/tests). - Send the Slack alert and file the Jira ticket, where those are set up.
- Leave everything else running: the test's results keep being recorded, and detection keeps evaluating it.
Quarantine does not:
- Change your CI result. The upload action only sends results; it never reads quarantine state, and BuildPulse never changes the outcome of a job. A quarantined test that fails still fails your build until your pipeline acts on the list.
- Skip the test or stop it running.
- Expire. It lasts until one of the release events below.
Keep quarantined tests from failing CI
Your pipeline does this by reading the quarantine list from the BuildPulse API. There are two ways to use the list:
- Gate on the failures that matter (the example below). Tests run as usual and results are uploaded; the job then fails only if a test that is not quarantined failed. Quarantined tests keep producing results, so you can see in BuildPulse whether they have become stable.
- Skip before running. Turn the same list into your test runner's exclude option, such as
--deselectfor pytest,-skipforgo testor--testNamePatternfor Jest. How test names map onto those options depends on the runner and its JUnit reporter. Skipped tests produce no results, so BuildPulse can no longer tell you whether they still fail.
The endpoint is GET https://platform.buildpulse.io/api/v1/flaky/tests?repository=<repository name>, with an organization API token in an Authorization: Bearer header. It returns quarantined tests by default, all of them however long ago they were quarantined, at most 100 per page with the next page's cursor in metadata.after. Each test carries id, name, suite and file. Create the token under Organization Settings, then API Tokens; the same token works for the upload action. Reading the API needs a plan with API access: Team, Growth, Enterprise, a trial, or a paid runner subscription. Full reference: platform.buildpulse.io/docs.
A complete GitHub Actions job using the BuildPulse upload action and the API:
name: CI
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run tests
id: tests
# Keep going on a red test run: the gate step below decides the result.
continue-on-error: true
run: pytest --junitxml=reports/junit.xml
- name: Upload test results to BuildPulse
if: '!cancelled()'
uses: BuildPulseLLC/test-reporter-action@v3
with:
api-token: ${{ secrets.BUILDPULSE_API_TOKEN }}
path: reports/junit.xml
- name: Fetch quarantined tests
if: steps.tests.outcome == 'failure'
env:
BUILDPULSE_API_TOKEN: ${{ secrets.BUILDPULSE_API_TOKEN }}
REPOSITORY: ${{ github.event.repository.name }}
run: |
echo '[]' > quarantined.json
after=""
while :; do
page=$(curl -sfS -G https://platform.buildpulse.io/api/v1/flaky/tests \
-H "Authorization: Bearer $BUILDPULSE_API_TOKEN" \
--data-urlencode "repository=$REPOSITORY" \
--data-urlencode "quarantine=true" \
--data-urlencode "limit=100" \
${after:+--data-urlencode "after=$after"})
echo "$page" | jq '.tests' | jq -s '.[0] + .[1]' quarantined.json - > next.json
mv next.json quarantined.json
after=$(echo "$page" | jq -r '.metadata.after // empty')
[ -z "$after" ] && break
done
echo "$(jq length quarantined.json) quarantined tests"
- name: Fail on tests that are not quarantined
if: steps.tests.outcome == 'failure'
run: python3 .github/scripts/quarantine_gate.py reports/junit.xml quarantined.jsonAnd the gate script it calls. It needs only Python 3, which GitHub's Ubuntu runners include, as they do jq and curl:
# .github/scripts/quarantine_gate.py
# Exits 1 if any failed test in the JUnit report is not quarantined in BuildPulse.
import json
import re
import sys
import xml.etree.ElementTree as ET
report, quarantine_list = sys.argv[1], sys.argv[2]
def suite_key(name):
# BuildPulse stores an unnamed suite as "default", and RSpec shard suites
# (rspec1, rspec2, ...) as "rspec".
name = name or "default"
return "rspec" if re.fullmatch(r"rspec\d+", name) else name
with open(quarantine_list) as f:
quarantined = {(t["suite"], t["name"]) for t in json.load(f)}
failed = []
for suite in ET.parse(report).iter("testsuite"):
for case in suite.findall("testcase"):
if case.find("failure") is not None or case.find("error") is not None:
failed.append((suite_key(suite.get("name")), case.get("name", "")))
if not failed:
sys.exit("The test step failed, but the report lists no failed tests.")
blocking = [t for t in failed if t not in quarantined]
for suite, name in failed:
label = "FAILED" if (suite, name) in blocking else "quarantined, ignored"
print(f"{label}: {suite} > {name}")
sys.exit(1 if blocking else 0)- The gate fails safe. If the API call fails, the fetch step fails. If the test step failed for a reason other than a test (a compile error, a crash before the report was written), the report has no failed tests and the gate fails the build. A failed test that does not match an entry in the list is treated as not quarantined.
- Matching is on suite and test name, the same values BuildPulse shows for the test. If your runner writes several report files, run the gate on each.
- On a pull request from a fork, GitHub does not pass secrets to the workflow, so neither the upload nor the fetch can authenticate there.
How a test leaves quarantine
Quarantine is a latch. Nothing in result processing releases it, however long the test passes. A test is released only by one of these:
- Resolve. The Resolve button on the flaky test's page, or Bulk Resolve Quarantined Tests in the repository's Settings, which resolves every quarantined test in the repository.
- Its Jira ticket moves to Done. This applies to the ticket BuildPulse filed for the test's current quarantine, whether filed automatically or with the Create Jira Issue button. A Jira link pasted by hand is not tracked, and a ticket from an earlier quarantine of the same test no longer counts. A ticket in Jira's Done category releases the test unless its resolution says the test was not fixed: Duplicate, Won't Do, Won't Fix, Cannot Reproduce, Not a Bug, Incomplete, Declined, Rejected, Obsolete, Invalid or Abandoned (case does not matter). A Done ticket with no resolution, or with Done, Fixed, Resolved or any other resolution, releases it.
- Its linked GitHub issue is closed, as completed or as not planned. The issue has to be in the same repository as the test.
A release clears the issue link and writes an entry in the test's audit log. If the test still meets the automatic quarantine conditions, the next upload that includes it quarantines it again.
Jira
- Connecting. On your organization's Integrations page, connect Jira Cloud with your site's domain, the email of an Atlassian user, and that user's API token. BuildPulse checks the credentials once, when you connect, and stores them encrypted. Tickets are created as that user, so if the user loses access or the token expires, Jira rejects new tickets. Atlassian API tokens have an expiry date.
- Linking a repository. On the same page, link each repository to a Jira project. Each link has Auto-create tickets on quarantine (on by default) and Extra labels. Every ticket gets the label
buildpulseas well. - When a ticket is filed. When a test in a linked repository is quarantined, automatically or by hand, and has no issue linked yet. The Create Jira Issue button on a flaky test files one at any time, even with auto-create off. The ticket is titled
[BuildPulse] Flaky Test: <test name>, uses the project's Story, Task or Bug issue type (the first of those it has), and carries the test's suite and file, its failure rate and disruptions, and links to the failing build and to the test in BuildPulse. - When a ticket can't be filed. The test is still quarantined. Quarantining by hand shows a warning that no ticket was filed, and the Integrations page shows Jira's error with a Reconnect button. Reconnect replaces the API token and keeps your project links and existing tickets. Tests left without a ticket get one later: whenever someone opens the repository's Flaky Tests page, BuildPulse files tickets for up to 10 quarantined tests that have none, in the repository's linked project, and stops at the first rejected request. Each test gets one ticket, even when several filings race.
- Release on Done. If the connecting user is a Jira administrator, BuildPulse registers a Jira webhook and a ticket moving to Done releases the test straight away, under the same resolution rule as above. The Integrations page shows Webhook active. Without the webhook, BuildPulse checks the tickets' status whenever someone opens the repository's Flaky Tests page, so a release waits until then.
- Disconnecting unlinks every project and removes the Jira links from your tests. Tickets already in Jira are not deleted, but BuildPulse stops tracking them, so closing them no longer releases tests. If you connect Jira again, tests that are still quarantined get fresh tickets.
Slack alerts
Set per repository on its Notifications page:
- New flaky test, when a test is flagged for the first time, by either detection. Each test sends it once, however it was caught.
- Test quarantined, when a test is quarantined, automatically or by hand.
- Threshold exceeded, off unless you turn it on. Sent when an already flagged test's 30-day failure rate rises from below the Flaky Detection Threshold to at or above it. It is sent once per crossing: the test has to drop back below the threshold before it can send again. A test that is new to BuildPulse sends New flaky test instead.
- Daily digest, a summary at an hour you choose.
Turning an alert off on the Notifications page stops it for that repository.
There is no alert when a quarantined test starts passing. Use the Flaky Tests page, or the Slack digest, to decide when to resolve one.