A test fails on a pull request that changed a README. You rerun the job. It passes. Nobody touched the test, nobody touched the code it exercises, and the failure is already gone by the time you open the log. That is a flaky test, and "rerun it" is the most expensive debugging strategy your team will ever adopt, because it never tells you which of a small number of root causes you are looking at.
I have spent years reading failure logs for tests like that, first on my own teams and now across the suites that report into BuildPulse. The good news is that flaky tests are not infinitely varied. Almost every one I have ever triaged falls into one of eight causes, and each cause leaves a recognizable fingerprint in the log. This post is the taxonomy: what each cause looks like, how to tell them apart, and where to go for the deep fix. It is deliberately the map, not the territory. Where we have written a full post on one cause, I link to it rather than repeat it.
What a flaky test actually is
A flaky test is one that produces different results on the same code. Same commit, same test, pass on one run and fail on another. That definition matters because two things get mislabeled as flaky all the time.
The first is a real bug with an intermittent trigger. If your checkout code has a race condition and the test catches it one time in twenty, the test is not flaky; it is the only honest thing in the pipeline. The second is an environment problem: a runner ran out of disk, the package registry timed out, the Docker daemon crashed. Those are infrastructure failures that happen to be surfaced by tests.
Both get rerun. Both get ignored. And that is the actual cost of flakiness: once a team learns that red might mean nothing, red stops meaning anything. We worked through the accounting in the real cost of a 5% test failure rate. If you want a gentler introduction, start with what are flaky tests? and come back.
The eight root causes of flaky tests
The order is roughly by how often I see each one in real suites, most common first.
1. Async timing and race conditions
The test kicks off something asynchronous (a promise, a goroutine, a background job, a network request) and then asserts before that something has finished. On a fast laptop the work completes in time. On a loaded CI runner it does not.
The fingerprint: the failure is almost always an assertion on state that "should" exist by now. An element that has not rendered, a record that has not been written, a callback that has not fired. The classic tell is a sleep or setTimeout somewhere near the assertion, which means a previous engineer already met this bug and negotiated with it instead of fixing it.
// The race, in its natural habitat
test('sends the welcome email', async () => {
await signUp({ email: 'a@example.com' })
await new Promise((r) => setTimeout(r, 100)) // "usually enough"
expect(mailer.sent).toHaveLength(1)
})
// The fix: await the thing you are actually waiting for
test('sends the welcome email', async () => {
const sent = waitForEvent(mailer, 'sent')
await signUp({ email: 'a@example.com' })
await sent
expect(mailer.sent).toHaveLength(1)
})
This category is big enough that we split it in two. The async race condition is why your tests are flaky covers the pattern in application code, and why fake timers beat sleep() every time covers the fix when the timing lives inside the code under test.
2. Shared state and order dependence
Test A leaves something behind. Test B assumes a clean world. When A runs before B, B fails. When the runner shuffles the order, or splits files across parallel workers differently, the failure appears and disappears.
The shared thing is usually one of: a database row, a global variable or singleton, an environment variable, a file on disk, a module-level cache, or a mocked function that was never restored. Database state is the most common by a wide margin, because integration tests love a "known" user with id 1.
The fingerprint: the test passes in isolation every single time. Run just that file and it is green. Run the suite and it fails, but only sometimes, and the "sometimes" correlates with which other tests happened to land on the same worker.
The flakiest test in your suite is fighting over a database row walks through this one in depth, including the transaction-per-test pattern that removes most of it.
3. Nondeterministic ordering in the code under test
This is the sneaky cousin of shared state. Nothing leaks between tests. The code itself simply does not promise an order, and the test assumes one.
A SELECT without an ORDER BY returns rows in whatever order the planner feels like today. A hash map in Go or Python iterates in an order you are not supposed to rely on. A Promise.all resolves its inputs in completion order, and a Set built from them will reflect that. The test does expect(results[0].name).toBe('alice') and is right ninety-something percent of the time.
The fingerprint: the assertion is on position or sequence, and the actual and expected values contain the same elements in a different order.
We wrote this one up as flaky tests from nondeterministic ordering: the ORDER BY you never wrote.
4. Time and clock dependence
The test reads the wall clock, directly or through something that does. It computes "tomorrow," "end of month," "30 days ago," or "is this token expired," and the answer depends on when the test runs.
These are the flakes with a schedule. They fail at midnight UTC, on the last day of the month, during the daylight-saving switch, or on February 29th. They pass on the rerun because by then the clock has moved past the boundary.
The fingerprint: look at the timestamps of the failures, not the logs. If they cluster near a boundary (00:00, month end, a DST change), you have a clock bug. The fix is that tests never read the real clock; inject one. Time-dependent flaky tests: why your CI only fails at midnight has the five recurring variants and the CI trick of deliberately running the suite under a hostile timezone.
5. External dependencies and the network
The test talks to something it does not own: a third-party API, a package registry, a DNS resolver, a cloud service, a public sandbox. That thing has its own availability, rate limits, and mood.
The fingerprint: timeouts, connection resets, 429 Too Many Requests, or an assertion on data that changed on the other end. The failure rate tracks the health of the dependency, not the health of your code, which is why these often come in bursts affecting many unrelated PRs at once.
The fix is boring and correct: tests that verify your logic should not cross the network. Record and replay, or use a fake at the boundary. Keep a small number of true end-to-end checks that do hit the real thing, run them separately, and treat their failures as a signal about the dependency rather than the PR.
6. Resource limits and infrastructure
The test needs more than the environment can reliably give it. A port that another parallel job already bound. A file descriptor limit. A container that gets OOM-killed. A test that assumes four cores and gets two. A temp directory shared between two shards.
The fingerprint: failures that correlate with parallelism and with runner size, not with code. The same suite is stable at --workers 2 and flaky at --workers 8. Or it is stable on your laptop and flaky on a small hosted runner. Exit codes like 137 (OOM) and errors like EADDRINUSE are giveaways.
This cause has grown as teams parallelize more aggressively, and it gets its own treatment in your test suite is a distributed system: the race conditions that aren't in your code.
7. Randomness and unseeded data
The test, or a fixture library it uses, generates random input. Most values work. One in a thousand does not: an empty string, a name with an apostrophe, a negative number, a Unicode character that breaks a regex, a UUID that happens to sort first.
The fingerprint: the failure is reproducible only with a specific value, and that value is in the log if you are lucky and nowhere if you are not.
The fix is not to remove randomness. It is to seed it and print the seed on failure, so a flaky failure becomes a reproducible one. Most property-testing frameworks do this for you; most faker calls in test setup do not.
8. UI and browser timing
End-to-end tests in a real browser combine causes 1, 5, and 6 and then add their own: animations that have not finished, elements that exist but are not yet clickable, layout shifts that move the button under the cursor, and a headless browser slower than the one on your desk.
The fingerprint: "element not found," "element not interactable," "detached from DOM," and screenshots that show the page half a second before the state the test expected.
Modern frameworks (Playwright, Cypress) auto-wait for a reason. Almost every flaky browser test I have read either bypassed the auto-wait with a manual sleep, or asserted on something the framework cannot wait for. Fix the wait, not the timeout.
A quick diagnostic table
When a test flakes, the log usually tells you the cause in the first minute if you know what to look for.
| What you see in the failure | Most likely cause |
|---|---|
Assertion on state that "should exist by now"; a sleep nearby | 1. Async timing |
| Passes alone, fails in the full suite; varies with worker assignment | 2. Shared state |
| Same elements, different order in the diff | 3. Nondeterministic ordering |
| Failures cluster at midnight, month end, or DST | 4. Clock dependence |
| Timeouts, 429s, connection resets; many PRs fail at once | 5. External dependency |
Exit 137, EADDRINUSE, worse at higher parallelism | 6. Resource limits |
| Fails only for one specific generated value | 7. Unseeded randomness |
| "Not interactable," "detached from DOM," stale screenshot | 8. Browser timing |
None of this works if you do not keep the failure history. The single most useful artifact for diagnosis is not one log; it is twenty failures of the same test lined up next to each other, with timestamps, runner sizes, and worker assignments. A pattern that is invisible in one run is obvious in twenty.
Fixing versus mitigating
There are two different responses to a flaky test, and healthy teams do both.
Fixing means finding the cause above and removing it. That is the goal, and how to fix flaky tests is the playbook for doing it systematically rather than one heroic afternoon at a time.
Mitigating means containing the damage while the fix is in the queue: quarantining the test so it cannot block merges, tracking its failure rate so it does not quietly rot, and catching new flakes in their first week rather than their first year. How to mitigate and prevent flaky tests covers that side. The thing to avoid is mitigation that pretends to be a fix: untracked automatic retries are how a suite ends up with a third of its tests silently running twice and nobody knowing.
That "nobody knowing" is the part I care most about. Categorizing a flake is easy once you have its history. Getting the history is the hard part, because most CI systems throw it away after the rerun. This is why we built flaky test detection the way we did: it keeps every test result across every run, identifies the tests that pass and fail on the same commit, and lines up the failure timestamps, messages, and frequency side by side. The taxonomy above is much easier to apply when the evidence is already collected.
FAQ
Are flaky tests always the test's fault?
No. Causes 1 through 4 are usually in the test or in test setup, but the same defects can live in the application code, in which case the flaky test is doing its job. Before you "fix" a test, confirm the code it exercises is actually deterministic. A race condition in a checkout flow that a test catches one run in fifty is a production bug with an early warning attached.
Should I just retry flaky tests automatically?
Retries are a mitigation, not a fix, and they are only safe when you record that a retry happened. An untracked retry hides the failure rate, which means you lose the one number that tells you whether things are getting better or worse. If you retry, retry once, log it as a flake, and keep the count visible.
How many flaky tests is too many?
Any number that makes your team rerun jobs by habit. In practice, once more than a low single-digit percentage of runs fail for reasons unrelated to the change, engineers stop reading failures and start clicking rerun, and the whole suite loses its value as a signal. Measure the rate per test and per suite, and treat a rising trend as a bug in the pipeline itself.
