The week the merge queue made everything slower
A platform team I worked with turned on GitHub merge queue on a Monday. Solid reasoning: 60 engineers, a busy monorepo, and a recurring problem where two green PRs merged an hour apart and broke main together. The queue would test every change against the actual future state of the branch. Textbook.
By Thursday, median time-to-merge had gone from 40 minutes to just over three hours. Nobody had pushed a bad change. The suite had a flake rate of about 3% per run, same as the month before. The only thing that changed was the queue.
Here's the part that surprises people: a merge queue doesn't just tolerate flaky tests badly. It structurally amplifies them. A flake that used to cost one developer one rerun now fails a speculative batch, dequeues an innocent PR, and forces a rebuild of every PR stacked behind it. If you're evaluating GitHub merge queue and you haven't measured your flake rate first, you're about to industrialize a problem you could have fixed for cheap.
How speculative batching turns one flake into five failures
A quick refresher on the mechanics, because the amplification lives in the details. When PRs enter a GitHub merge queue, the queue doesn't test them one at a time. It builds speculative merge commits: PR #1 against main, PR #2 against main + #1, PR #3 against main + #1 + #2, and so on, up to your configured parallelism. All of those CI runs execute concurrently. When they all pass, the whole batch merges in order. This is what makes a merge queue fast enough to be usable.
It's also what makes flakes expensive. Suppose the run for PR #3 fails on a flaky test:
- PR #3 is removed from the queue. Its author gets a red X on a change that is perfectly fine.
- Every speculative build behind #3 was computed against a future that no longer exists. PRs #4 and #5 are rebuilt from scratch against the new proposed history.
- PR #3's author, after staring at an unfamiliar failure in a test they've never touched, re-adds the PR. It enters at the back of the queue and waits for a full traversal.
Count the CI runs. Outside a queue, that flake costs one rerun. Inside a five-deep queue, it costs the failed run, two or more rebuilt speculative runs, and a fresh run when the victim requeues. Four to five full suite executions, plus the human time of an engineer investigating a failure that had nothing to do with their code. We benchmarked what plain reruns cost in How much CI time flaky test reruns actually burn; the merge queue multiplies that number by roughly your average batch depth.
The math: your flake rate compounds with queue depth
The intuition is simple. The queue needs k consecutive speculative builds to all come up green. Flakiness gets k chances to ruin the batch instead of one.
If your suite has a probability f of showing at least one flaky failure per run, the probability that a batch of k speculative builds completes without a flake is (1 − f)^k. The probability the batch gets disrupted is:
P(batch disrupted by flake) = 1 - (1 - f)^k
f = 1%, k = 5 -> 4.9%
f = 3%, k = 5 -> 14.1%
f = 3%, k = 8 -> 21.6%
f = 5%, k = 8 -> 33.7%
f = 10%, k = 5 -> 41.0%
Read that middle row again. A 3% per-run flake rate, which most teams would describe as "annoying but livable," means one in five of your eight-deep batches gets torn up by a failure that isn't real. At 10% flakiness, which is not rare in suites nobody has triaged, nearly half your batches fail. Your merge queue is now a machine that converts flaky tests into org-wide latency, running 24/7, with no human in the loop to say "that's just the websocket test again."
And it compounds. Every disrupted batch triggers rebuilds, every rebuild is another run, and every run is another lottery ticket for the next flake. With 40 merges a day, a 3% flake rate, and an average of three extra runs per disruption, you're paying for roughly 10% more CI executions than your merge volume justifies, before counting the requeue runs. That's the compute cost. The latency cost is worse, because a queue failure doesn't just burn minutes, it resets the pipeline: everything behind the failure starts over, and throughput for the whole org stalls while the queue rebuilds its speculative state.
Before the queue, a flake was a private tax on one developer. After the queue, it's a public tax on everyone trying to merge that afternoon. That's the industrialization, and it's why the sticker math on a 5% failure rate understates the damage once a queue is involved.
The failure mode nobody budgets for: trust erosion at the queue
The compute waste is measurable. The behavioral damage is worse, and it follows a predictable arc.
Week one: an engineer gets dequeued by a flake, investigates for 20 minutes, finds nothing, requeues, merges. Mild annoyance. Week three: engineers get dequeued, glance at the failure, recognize the usual suspect, and requeue immediately without looking. Week six: someone writes a script that auto-requeues on failure, and your merge queue, the control you installed to guarantee main is always green, has become an elaborate retry button that occasionally waves real regressions through on the third attempt.
If you're in a SOC2 or ISO 27001 shop, that arc should worry you for a second reason. The merge queue is probably documented as part of your change-management control: no change reaches the protected branch without passing the full suite against the true merge state. Auto-requeue-on-red quietly rewrites that control into "changes pass the suite eventually, on some attempt, and nobody reviews which one." We've written about why flaky tests are a change-management problem, not just a developer-experience one, and the merge queue is where that problem surfaces with an audit trail attached.
Why retry-inside-the-queue makes it worse
The reflexive fix is to configure the queue's CI jobs to retry failed tests automatically. Now the batch survives the flake, the queue keeps moving, problem solved.
Except you've made three trades, all bad. First, you've lengthened the critical path of every batch: a retry inside a speculative build delays every PR stacked behind it, so the whole org waits on your slowest flake. Second, you've added a real-failure blind spot in the one place it matters most. A test that fails intermittently because of an actual race in the code under test will sometimes pass on retry, and the merge queue is the last gate before main. Waving it through there means the race ships. We covered exactly this class of bug in the async race condition is why your tests are flaky; retries don't distinguish it from an infra hiccup. Third, per Retries are a tax, not a fix, you've removed the pain signal that would have gotten the flake fixed, while keeping the cost. The flake is now permanent line-item overhead on every merge, forever.
Retries inside a merge queue aren't a mitigation. They're a subsidy, paid by the whole queue, to avoid fixing a test.
Sequence it correctly: detection first, queue second
None of this is an argument against merge queues. Testing against the true merge state is the right call for any repo with meaningful merge volume; "two green PRs broke main together" is a real failure mode and the queue genuinely eliminates it. The argument is about sequencing. A merge queue assumes your test signal is trustworthy. If it isn't, the queue amplifies the noise at queue-depth scale. So:
1. Measure your flake rate before you flip the switch. Not vibes, numbers. Collect JUnit XML from every CI run for two to four weeks and count how often a test fails and then passes with no relevant code change. This is exactly what BuildPulse does from your existing test output, and the per-run flake probability it gives you plugs directly into the batch math above. If 1 − (1 − f)^k at your intended queue depth comes out above roughly 5%, fix the flakes first, or the queue will hurt more than it helps.
2. Quarantine known flakes out of the queue's blocking path. A quarantined test still runs and still reports, but it can't dequeue an innocent PR or torch a batch. This is the single highest-leverage move before enabling a queue, and unlike blanket retries it's visible, auditable, and reversible per test.
3. Make sure your workflows actually run on the queue's event. GitHub merge queue triggers a separate merge_group event, and a workflow that only listens to pull_request silently doesn't gate the queue at all:
name: CI
on:
pull_request:
merge_group:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npm ci
- run: npm test -- --reporters=default --reporters=jest-junit
- name: Upload test results
if: always()
uses: actions/upload-artifact@v4
with:
name: junit-results
path: junit.xml
Upload results on the merge_group runs too, not just PR runs. Queue runs are where flakes do the most damage, so they're the runs you most need visibility into. If your reporter setup is shaky, start with getting trustworthy JUnit XML out of Jest.
4. Keep queue depth honest. Deeper batching raises throughput on green days and raises the blast radius on flaky ones. If you can't get f below about 1%, run a shallower queue until you can. It's slower on paper and faster in practice.
5. Watch dequeue reasons weekly. Every removal from the queue is either a real regression caught early (the queue working) or a flake taxing the org (the queue amplifying). If you can't tell those apart in your metrics, you can't tell whether the queue is paying for itself.
The queue is a magnifying glass
A merge queue is one of the clearest examples of a general rule: automation multiplies whatever signal you feed it. Feed it a trustworthy suite and it multiplies confidence; you get an always-green main and merges nobody babysits. Feed it a 3% flake rate and it multiplies noise, at batch depth, around the clock.
So before you enable GitHub merge queue, ask the unglamorous question first: do I actually know my flake rate? If the answer is a shrug, measure for two weeks, quarantine what you find, and then turn on the queue. It's the difference between installing a control and installing a very expensive random number generator between your engineers and main.
