Your test suite is a distributed system: the race conditions that aren't in your code

The race condition you didn't write

You've stared at the failure for twenty minutes. The test is dead simple — insert a record, query it, assert the count. It passes locally every time. It passes in CI most of the time. But once or twice a week it fails with a count that's off by one, or three, or seven.

The bug isn't in your application code. The bug is in how you're running your tests.

When you parallelize a test suite across workers — whether that's Jest workers, pytest-xdist, parallel RSpec, or a matrix of GitHub Actions jobs — you've built a distributed system. It has shared state, timing dependencies, and no global coordinator. Distributed systems have race conditions. So does your test suite.

This post is about the four categories of CI race condition I see most often, why they're harder to diagnose than application-level races, and what actually fixes them.

Why these are harder to catch than normal race conditions

Application-level races usually leave a trail. A corrupted data structure, a deadlock, an exception with a stack trace. You can reproduce them under load, instrument them, and reason about the fix.

CI race conditions are different in three ways:

• The affected code looks innocent. The test that fails doesn't have a threading bug. Your production code is fine. The failure comes from the environment the test runs in.
• They're sensitive to scheduling, not load. Adding workers makes them worse; reducing workers makes them disappear. That tricks teams into treating parallelism itself as the enemy rather than the missing isolation.
• Reruns mask them. A flaky test that passes on retry gets merged. The failure is real — it just didn't happen to recur when you were watching. Over time these accumulate until your CI signal is genuinely untrustworthy.

If you're not tracking which tests flake and how often, you're flying blind. A platform like BuildPulse surfaces the pattern — test X fails 12% of the time across parallel workers but almost never on sequential runs — which is usually the first clue that the root cause is environmental.

Category 1: shared databases

This is the most common one. Two test workers hit the same database schema, and they both assume they own it.

The failure mode looks like this: Worker A runs a test that seeds three users and expects SELECT COUNT(*) FROM users to return 3. Worker B, running a different test concurrently, has inserted two more users into the same table. Worker A gets 5. The assertion fails.

The naive fix is transactions — wrap each test in a transaction and roll it back. That works until you have tests that use multiple database connections, background jobs, or anything that escapes the transaction boundary.

The correct fix is per-worker database isolation:

# GitHub Actions matrix — give each worker its own database
jobs:
  test:
    strategy:
      matrix:
        worker: [1, 2, 3, 4]
    env:
      DATABASE_URL: postgres://localhost/myapp_test_${{ matrix.worker }}
    steps:
      - name: Create test database
        run: createdb myapp_test_${{ matrix.worker }}
      - name: Run migrations
        run: bundle exec rails db:migrate
      - name: Run tests
        run: bundle exec rspec --format progress

For pytest-xdist, the pytest-django plugin handles this automatically with --reuse-db and the django_db_setup fixture scoped per worker. For Jest with a real database, you want a setup file that reads process.env.JEST_WORKER_ID and builds a unique connection string from it.

// jest.setup.js
const workerId = process.env.JEST_WORKER_ID ?? '1';
process.env.DATABASE_URL = `postgres://localhost/myapp_test_${workerId}`;

The pattern is always the same: namespace the shared resource by worker identity.

Category 2: port collisions

Your integration tests spin up a server. So does the next worker. They both try to bind to port 8080. One of them wins; the other throws EADDRINUSE and the test fails in a way that looks like a connection error, not a port conflict.

The fix here is dynamic port allocation — never hardcode a port in test setup:

import socket
import pytest

def get_free_port() -> int:
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.bind(('', 0))
        return s.getsockname()[1]

@pytest.fixture(scope='session')
def app_port():
    return get_free_port()

@pytest.fixture(scope='session')
def live_server(app_port):
    # Start your server on app_port, yield, then shut it down
    ...

The same logic applies to any bound resource: gRPC servers, mock HTTP servers, Prometheus exporters embedded in test harnesses. If it binds a port, it needs a dynamic one.

For Docker-based setups, use ports: ['0:8080'] in your Compose file and read the assigned port back from the container inspect output. Hardcoded ports in docker-compose.yml files used in CI are a quiet source of parallel test flakiness that only surfaces when you scale your runner pool.

Category 3: container startup ordering

Your test job starts a Postgres container, then immediately tries to connect. The container isn't ready yet. The connection is refused. The test fails.

This one is famous and still gets people. The depends_on key in Docker Compose ensures the container has started — it says nothing about the service inside it being ready to accept connections.

The fix is a proper readiness check, not a sleep:

# docker-compose.yml
services:
  postgres:
    image: postgres:16
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 2s
      timeout: 5s
      retries: 10

  app:
    depends_on:
      postgres:
        condition: service_healthy

Or in a GitHub Actions workflow without Compose:

- name: Wait for Postgres
  run: |
    for i in $(seq 1 30); do
      pg_isready -h localhost -p 5432 && break
      echo "Waiting... ($i)"
      sleep 2
    done

The same pattern applies to Redis, Elasticsearch, Kafka, and any other service your tests depend on. Poll with a real health signal, not a fixed delay. A fixed delay that's too short is a race condition. A fixed delay that's long enough is just waste.

Container startup ordering issues are particularly nasty in flaky-test terms because they're non-deterministic across CI infrastructure. A faster runner makes them less likely to reproduce, which makes them hard to catch in your local Docker environment but present in cloud CI where provisioning time varies.

Category 4: clock skew and time-dependent assertions

This one is subtle. Tests that assert on timestamps, token expiry, cache TTLs, or "created in the last N seconds" are sensitive to how fast the test runs relative to real wall-clock time.

Parallel CI workers are slower than sequential runs — more context switching, more I/O contention, more variance. A test that asserts created_at > Time.now - 1.second and runs in 800ms locally might take 2.1 seconds under load and fail the assertion.

There are two failure sub-modes here:

1. Tests that depend on relative time without controlling the clock.

Fix: freeze the clock in tests. Don't test against Time.now — test against a known, controlled timestamp.

# RSpec with timecop
it 'expires the token after one hour' do
  freeze_time do
    token = Token.create!
    travel 61.minutes
    expect(token.expired?).to be true
  end
end

# pytest with freezegun
from freezegun import freeze_time

@freeze_time("2024-01-15 12:00:00")
def test_token_expires_after_one_hour():
    token = Token.create()
    with freeze_time("2024-01-15 13:01:00"):
        assert token.is_expired()

2. Tests that race against external time-based side effects — a background job that fires after X seconds, a cache that expires, a rate limiter that resets on a clock boundary.

Fix: make the time boundary explicit and injectable, then control it in tests. If your background job fires "after 30 seconds", that's a parameter, not a constant. Tests should be able to pass a very short interval or trigger the job directly rather than waiting for wall-clock time to elapse.

Reproducing these locally

The hardest part of CI race conditions is that they disappear when you look at them. A few approaches that help:

Run tests in random order. RSpec has --order random. pytest-randomly shuffles by default. If a test only passes when it runs after another test has set up state, random ordering will surface that dependency.

Crank up parallelism beyond what's comfortable. Run with 2x or 4x the normal worker count locally. You're trying to increase the probability that two workers collide on a shared resource.

Repeat flaky tests in isolation. Once you've identified a suspect test, run it 50 times in a loop: for i in $(seq 1 50); do pytest tests/test_suspect.py; done. If it fails even once, you've confirmed the flakiness without needing CI.

Read the CI logs from parallel workers side by side. When a failure is in the "shared state" category, you'll often find that two workers were touching the same resource at the same time. The timestamps in the logs are the tell.

For identifying which tests in a large suite are flaky across parallel workers — as opposed to just slow or broken — BuildPulse's flaky test detection tracks pass/fail history per test across every CI run, so you can see whether a failure is correlated with parallelism or with specific worker combinations.

The pattern behind all four categories

Every one of these failure modes has the same shape: a shared resource with no coordination between parallel consumers.

Shared database schema → namespace by worker. Shared port → allocate dynamically. Shared container startup → gate on health, not on start. Shared wall clock → control it explicitly.

The fix is always isolation — giving each worker an independent view of the resource it needs, or making the resource coordination explicit rather than implicit.

The teams I've seen handle parallel test flakiness well treat their test infrastructure with the same rigor they'd apply to a production distributed system. They design for isolation from the start, they instrument the failure modes, and they don't accept "it passed on retry" as a resolution. Because in a SOC2 or ISO 27001 environment, a CI gate that passes on retry is not a CI gate — it's a suggestion.

If you're not sure whether your suite has these problems, look at your flaky test rate across parallel versus sequential runs. A meaningful difference there is almost always environmental, not a bug in your application. That's the good news: the fix is in your test infrastructure, and it's usually a one-time investment.

For more on flaky test patterns and what good CI signal actually looks like, see how to fix flaky tests and the cost of flaky tests on engineering teams.