The test that only fails at 16 workers
A staff engineer at a fintech I worked with spent three days hunting a flaky integration test. It never failed locally. It never failed when she ran it in isolation in CI. It failed roughly once in twelve full pipeline runs, always in a different way: sometimes a foreign key violation, sometimes a connection refused, once a timestamp assertion that was off by two seconds.
She read the test six times. The test was fine. The application code was fine. The bug was in the pipeline.
That pattern is worth naming, because I see teams burn weeks on it. When a test fails only under parallel CI and the failure mode keeps changing, stop reading the test. You are not looking at a bug in your code. You are looking at a race condition in the distributed system you accidentally built: your CI pipeline.
You built a distributed system without noticing
Run a suite with eight parallel workers across four CI machines and you have, by any honest definition, a distributed system. Multiple processes. Shared resources. No coordinator. Independent clocks. Partial failure modes. The difference between this and the distributed systems your architects lose sleep over is that nobody drew a diagram for this one, so nobody reasoned about its failure modes.
Every classic distributed-systems bug has a CI equivalent:
- Shared mutable state: two workers writing to the same database.
- Resource contention: two suites binding the same port.
- Ordering assumptions: a test that starts before its dependency container is actually ready.
- Clock skew: parallel jobs on machines whose clocks disagree, or a test that straddles a time boundary.
None of these are visible in a code review of the test file. All of them produce failures that vanish on rerun, which is exactly why teams reach for auto-retries and exactly why retries make the problem invisible instead of gone. Let's take the four one at a time, with reproduction and fix patterns for each, because a race you can't reproduce is a race you'll be arguing about in six months.
Race 1: the shared database
The most common one by a wide margin. Two test workers hit the same Postgres instance. Worker A truncates a table during teardown while worker B is mid-assertion on a row in that table. B fails with a missing record, a deadlock, or a unique constraint violation on a fixture ID both workers thought they owned. I've written about the single-row version of this fight before; the parallel-CI version is the same disease at scale.
Reproduce it: run only your database-touching tests with the worker count doubled, in a loop. If the suite is clean at one worker and dirty at sixteen, you've confirmed it's contention, not logic.
# Crank parallelism and loop until it breaks
for i in $(seq 1 20); do
npx vitest run --pool=forks --maxWorkers=16 tests/integration \
|| { echo "failed on iteration $i"; break; }
done
Fix it: give each worker its own database. Not its own transaction (transactions leak across connection pools and break tests that exercise commits), its own database. Postgres template databases make this nearly free: create one migrated template at suite start, then stamp out a copy per worker in milliseconds.
// globalSetup: create per-worker databases from a migrated template
import { Client } from "pg";
export async function setup() {
const admin = new Client({ database: "postgres" });
await admin.connect();
await admin.query(`CREATE DATABASE app_test_template`);
await migrate("app_test_template");
const workers = Number(process.env.WORKER_COUNT ?? 8);
for (let w = 1; w <= workers; w++) {
await admin.query(
`CREATE DATABASE app_test_w${w} TEMPLATE app_test_template`
);
}
await admin.end();
}
Each worker then reads its worker ID (VITEST_POOL_ID, JEST_WORKER_ID, or the shard index in your CI matrix) and connects to its own copy. Teardown is DROP DATABASE, which never deadlocks with anyone because nobody else is in there.
If a per-worker database sounds heavyweight, measure it first. A template copy in Postgres 15 takes single-digit milliseconds. Your team is currently paying far more than that in rerun minutes and triage time.
Race 2: port collisions
Your integration tests start a server on port 3000. So does the other suite that landed on the same CI machine, or the other worker in the same job, or the leftover process from the previous job on a self-hosted runner that doesn't get a fresh VM. One of them gets EADDRINUSE. The other gets something worse: it silently connects to the wrong server and runs its assertions against someone else's application state. Those failures are baffling because the error is downstream of the actual bug.
Reproduce it: run two copies of your integration suite concurrently on one machine. If either fails, you have a port assumption.
Fix it: never hardcode a port in a test. Bind to port 0 and let the OS hand you a free one, then thread the assigned port to whatever needs it.
// Go: ask the kernel for a free port instead of praying 8080 is open
listener, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
srv := startServer(listener)
baseURL := "http://" + listener.Addr().String()
Every mainstream stack supports this. Node's server.listen(0), Python's socket.bind(("", 0)), Spring's @SpringBootTest(webEnvironment = RANDOM_PORT). The pattern that does not work is picking a "random" port from a range in the test code, because two workers doing that will eventually pick the same number. That's not fixing the race. That's just lowering its frequency until it only fires during your busiest release week.
Race 3: container startup ordering
Your job starts Postgres and Redis containers, then runs tests. The docker compose up command returns when containers are started, not when the services inside them are ready to accept connections. Postgres in particular does an internal restart during initialization, so there's a window where the port accepts a connection and then drops it. Tests that start fast enough fall into that window; on a slower runner they don't, which is why this class of flake often appears right after you upgrade to faster CI machines. Speed exposes races. It doesn't cause them.
Reproduce it: add sleep 5 before your test command. If flakiness disappears, you have a readiness race. Now delete the sleep, because a sleep is a bet on hardware timing, and the bet will lose eventually.
Fix it: health checks with real readiness probes, and make the dependency graph wait on health, not on start.
# docker-compose.ci.yml
services:
db:
image: postgres:16
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 2s
timeout: 2s
retries: 15
app-tests:
build: .
depends_on:
db:
condition: service_healthy
If you're using GitHub Actions services: blocks, the same rule applies: define options: --health-cmd ... on the service container. And be honest about what your health check actually verifies. pg_isready confirms Postgres accepts connections. It does not confirm your migrations ran. If tests race the migration step, gate on a check that queries a table the migrations create.
Race 4: clock skew and time boundaries
The subtlest of the four. Parallel jobs run on different machines, and their clocks disagree by anywhere from milliseconds to a few seconds. A test that writes a record in job A and asserts on its timestamp in job B (via a shared staging database, an artifact, a cache) can see time move backward. Within a single machine you get the sibling problem: a test that computes "start of today," does some work, and asserts, will fail when the suite straddles midnight, because the two halves of the test ran on different days. I've covered the midnight failure class in depth, and parallelism makes it worse: more workers means more wall-clock coverage, which means more chances that some worker is executing during the boundary.
Reproduce it: don't wait for midnight. Fake the clock and set it to 23:59:58, then run the suite.
import { vi, test, expect } from "vitest";
test("report groups by day correctly at a day boundary", () => {
vi.useFakeTimers();
vi.setSystemTime(new Date("2025-03-09T23:59:58Z"));
// run the code that computes "today" ... then cross the boundary
vi.advanceTimersByTime(5_000);
expect(report.bucketFor(record)).toBe("2025-03-09");
});
Fix it: inject the clock. Any code that calls Date.now(), time.Now(), or datetime.now() directly is untestable at boundaries by construction. Pass a clock interface in, use fake timers in tests, and never assert on wall-clock deltas across process boundaries. If two jobs must agree on ordering, use a monotonic sequence you control (a database sequence, a run ID), not timestamps.
The diagnostic that separates code bugs from pipeline bugs
Here's the triage question I give teams: does the failure reproduce with parallelism at 1?
- Fails at
--maxWorkers=1, in isolation, deterministically: it's a code or test-logic bug. Normal debugging applies. - Passes in isolation, fails only under parallel load, failure mode varies: it's one of the four races above. Reading the test harder will not help.
That one bisection saves days, because the two categories demand opposite investigation styles. Code bugs want a debugger. Isolation bugs want a stress harness: crank workers, loop the suite, run two copies at once, fake the clock. You're trying to make the race fire on demand, the same way you'd repro a production race with a load test.
The uncomfortable part for engineering leaders: individual test authors can't fix this class. A developer writing a test has no visibility into what the other fifteen workers are doing, which ports the previous job leaked, or what the runner's clock says. CI test isolation is a platform property. Per-worker databases, port-zero conventions, health-gated startup, and injectable clocks belong in your shared test harness and your pipeline templates, owned by whoever owns the platform, enforced once, inherited by every suite. Teams that treat isolation as each author's personal responsibility get exactly the flake rate that policy deserves.
And you can't manage what you can't see. These races fire probabilistically, so a single rerun hides them and a merge queue amortizes them across everyone's afternoon. Tracking failure patterns across runs is how you find them: a test that fails 3% of the time, only in full-parallel runs, only on certain runners, is waving a flag that says "isolation bug," and that fingerprint is precisely what flaky test detection is built to surface. If your compliance posture depends on CI as a change-management gate, that visibility stops being nice-to-have; a gate that fails randomly is a gate your auditors will eventually ask about.
Your test suite became a distributed system the day you added a second CI worker. It's been one for years. The only question is whether you engineer it like one.
