Merge queues don't fix flaky tests — they industrialize them

The sales pitch vs. the reality

Merge queues are genuinely useful. GitHub's merge queue, Graphite, Aviator, Trunk — they all solve a real problem: the semantic merge conflict that doesn't show up until two approved PRs try to land at the same time. If you've ever had a green-on-branch, red-on-main incident that took two engineers an afternoon to untangle, you understand the appeal.

But there's a quiet assumption baked into every merge queue pitch: your CI signal is trustworthy. If it isn't — and at most companies above a few dozen engineers, it isn't — a merge queue doesn't protect you from flaky tests. It amplifies them.

I've watched teams roll out GitHub's merge queue with genuine excitement, then spend the next three weeks wondering why throughput got worse. The answer, almost every time, is flaky tests hiding in plain sight.

What actually happens when a flake hits the queue

Here's the mechanics. A merge queue batches PRs together and runs CI on the combined diff before merging. When CI fails, the queue has to figure out who's responsible. It does this by bisecting — splitting the batch, retrying subgroups, eventually isolating (or giving up on) the offending PR.

When the failure is deterministic, bisection works fine. The bad PR gets ejected, everyone else moves on.

When the failure is a flake, bisection becomes a tragicomedy.

The queue sees a failure. It splits the batch. It retries. The flake doesn't reproduce. The queue tentatively merges that half — then the flake fires again on the next batch. Or the retry does reproduce and an innocent PR gets ejected. Either way, every PR sitting behind the failure waits through partial reruns before it gets another shot at landing.

Let's put numbers on it.

Quantifying the blast radius

Assume a modest setup:

• CI suite takes 12 minutes to run
• Merge queue batch size: 4 PRs
• One test has a 15% flake rate (not outrageous — I've seen suites with dozens of tests above 20%)
• Team velocity: 20 PRs merged per day across 8 engineers

With a 15% flake rate, that test fires roughly once every 6–7 queue runs. When it does:

1. The batch of 4 fails (~12 min elapsed)
2. Queue bisects: two sub-batches of 2, each reruns (~12 min)
3. One sub-batch passes, one fails again — or the flake clears and the queue has to decide whether to trust it
4. Best case: ~24 minutes of extra CI time before the batch clears, and the 4–8 PRs queued behind it have been waiting the whole time

At 20 PRs/day with a 12-minute base CI time, your theoretical queue throughput is around 100 slots per day (20 PRs × 5 queue positions per hour × 1 hour). A single 15%-rate flake easily consumes 15–20% of that capacity in reruns. Your effective throughput drops — not because the engineers wrote worse code, but because the queue is doing bisection aerobics on a non-deterministic failure.

Now add three flaky tests. Or five. Or the 23 that one of our customers found after their first week of BuildPulse data.

The math compounds fast. And unlike a branch-by-branch CI model where a flake annoys one engineer, a merge queue means that flake annoys everyone in line behind it.

The false comfort of "just rerun"

Before merge queues, the standard response to a flaky CI failure was a manual rerun. Annoying, but contained. One engineer hits the button, gets green, ships.

Merge queues automate reruns — which sounds like a fix until you realize automated reruns at scale just mean you've made the problem faster and more thorough. The queue will dutifully retry, bisect, and retry again, burning compute and burning developer time (waiting for queue position to clear) with perfect consistency.

Automatic retry logic is a painkiller, not a cure. If the retry rate on your merge queue is above, say, 8–10%, that's a diagnostic: your test suite has a flake problem that the queue is papering over at scale.

You can check this directly if you're on GitHub. The merge queue insights in GitHub Actions surface retry counts. Pull that data for a week. If you're seeing more retries than you'd expect, you're not looking at a queue configuration problem — you're looking at a test reliability problem.

Flake detection has to come first

Here's the principle I'd argue for: a merge queue is a force multiplier. It multiplies throughput when CI is reliable. It multiplies pain when CI isn't.

That means the correct order of operations is:

1. Get visibility into flaky tests — which tests flake, how often, and in which jobs
2. Quarantine or fix the worst offenders before turning on queue-based merging
3. Set a flake-rate threshold as an ongoing gate (if test flakiness exceeds X%, the merge queue config tightens or alerts fire)
4. Then roll out the merge queue to capture the throughput wins it actually promises

Skipping step 1 and going straight to step 4 is the pattern I see teams regret.

The good news is that getting visibility doesn't require a heroic effort. Tools like BuildPulse parse your JUnit XML results across CI runs and surface flake rates per test, trend data, and which tests are most disruptive — so you can make a prioritized fix list before the merge queue turns those tests into everyone's problem.

If you want to start manually, the signal is in your CI logs. A test that appears in failure reports more than twice a week but has no corresponding code change in the diff is almost certainly flaking. Track it in a spreadsheet for two weeks. You'll have enough signal to prioritize.

What a pre-queue flake gate looks like

Once you have flake data, you can build a lightweight gate that keeps the queue clean. Here's a simple GitHub Actions job that fails if a known-flaky test is included in a PR without an explicit waiver:

# .github/workflows/flake-gate.yml
name: Flake gate

on:
  pull_request:
    branches: [main]

jobs:
  check-flaky-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Check for quarantined test changes
        run: |
          # quarantined.txt is a list of test file paths with known flake rates > 5%
          # maintained by the platform team, updated weekly from BuildPulse exports
          CHANGED=$(git diff --name-only origin/main...HEAD)
          while IFS= read -r quarantined; do
            if echo "$CHANGED" | grep -qF "$quarantined"; then
              echo "ERROR: $quarantined is in the flake quarantine list."
              echo "Fix the flake or get a waiver from #platform before merging."
              exit 1
            fi
          done < quarantined.txt
          echo "No quarantined tests modified. Gate passed."

This is deliberately simple. The point isn't to block all flaky tests — it's to force a conversation when someone modifies a test that's already known to be unreliable. Combine this with a weekly review of the quarantine list and you have a feedback loop that actually shrinks the list over time rather than letting it grow silently.

For a deeper look at quarantine patterns, this post on test quarantine strategies covers the tradeoffs between hard blocks, soft warnings, and automatic retry with reporting.

The audit angle (if compliance is your context)

If your organization is SOC 2 or ISO 27001 compliant, your merge controls aren't just a developer experience feature — they're part of your change management evidence. When an auditor asks "how do you ensure only reviewed, tested changes reach production," the answer usually includes CI gates and branch protection rules.

A merge queue with a high flake rate quietly undermines that story. If tests are failing and being retried automatically until they pass, the "CI gate" is doing less work than the audit trail suggests. A failed run that eventually passes via retry — without any code change — is a signal worth tracking.

Being able to show an auditor that your flake rate is below 1% and trending down is a meaningfully stronger posture than "we have a merge queue." It's also just true, which tends to hold up better under scrutiny.

The actual fix is boring but it works

Nothing here is glamorous. The path is: measure flake rates, prioritize by impact on queue throughput, fix the top offenders, gate on flake rate, then enjoy the merge queue you paid for.

The teams that get full value from merge queues — consistently fast, rarely blocked — didn't get there by configuring the queue better. They got there by making their test suite trustworthy first.

The queue is a machine. Feed it garbage, get garbage throughput. Feed it reliable CI signal, and it genuinely does what the pitch deck says.