A triage playbook for flaky evals: detect, stabilize, quarantine

The eval that failed on a README change

A PR that touched documentation and nothing else failed the eval suite. The engineer reran the job. Green. Merged. Two days later the same eval failed on a PR that did change the prompt, and the engineer did the same thing: rerun, green, merge. This time the rerun masked a real regression, and it shipped.

That's the whole problem with flaky evals in one anecdote. A test that fails randomly doesn't just waste CI minutes. It trains your team to ignore failures, and it does so faster than any flaky unit test, because everyone already half-expects LLM tests to be weird. Non-deterministic LLM tests get a cultural hall pass that a flaky database test never would.

They shouldn't. Eval flakiness is a tractable engineering problem, and the playbook looks a lot like the one for any other flaky test: measure the flake rate, stabilize what you can, and quarantine what you can't fix yet so it stops poisoning your merge signal. The details are LLM-specific. The discipline isn't.

Why evals flake even at temperature zero

Before you can fix eval flakiness, be honest about where it comes from, because "just set temperature to 0" fixes less than most people think.

• Sampling non-determinism. Temperature 0 makes sampling greedy, but it doesn't make inference deterministic. Batched inference, mixture-of-experts routing, and floating-point non-associativity across GPUs mean the same prompt can produce different tokens on different runs, even with identical parameters. OpenAI exposes a seed parameter and a system_fingerprint precisely because determinism is best-effort, not guaranteed.
• Judge non-determinism. If an LLM grades the output, you've stacked a second stochastic system on top of the first. A borderline answer that scores 0.79 one run and 0.82 the next will flap forever across a 0.80 threshold. We covered the failure modes in LLM-as-judge: how to use it without fooling yourself.
• Silent model drift. If you call a model alias like gpt-4o instead of a dated snapshot, the provider can move the target under you. Your eval didn't flake; your dependency changed with no lockfile entry to show for it.
• Retrieval and context variance. RAG evals inherit non-determinism from embedding services, index rebuilds, and top-k ties that break differently run to run.
• Plain old infrastructure. Timeouts, rate limits, and 529s from the provider. These are boring, and they're often the biggest single source of red eval jobs.

Each source needs a different fix, which is why the first step is figuring out which one you actually have.

Step one: measure the flake rate before you touch anything

A single eval failure tells you almost nothing. What you need per eval case is a pass rate at fixed inputs: same commit, same prompt, same model snapshot, N runs. This is the eval equivalent of rerunning a suspect unit test in a loop, and it's the step most teams skip because eval runs cost money. Skip it anyway and you'll spend more money debugging the wrong thing.

# measure_flake.py: run one eval case N times against the same commit
import asyncio, json
from statistics import mean
from my_evals import run_case

N = 20

async def main():
    results = [await run_case("refund-policy-summary") for _ in range(N)]
    passes = [r.passed for r in results]
    scores = [r.score for r in results]
    print(json.dumps({
        "pass_rate": sum(passes) / N,
        "score_mean": mean(scores),
        "score_min": min(scores),
        "score_max": max(scores),
    }, indent=2))

asyncio.run(main())

Twenty runs of one case is cheap compared to one afternoon of an engineer staring at a red check. The output sorts every eval into one of three buckets:

• Pass rate 1.0, tight score range. The case is stable. If it fails in CI, believe it.
• Pass rate 0.85–0.99, scores clustered near the threshold. You don't have a flaky eval, you have a threshold sitting inside the noise band of a stable score distribution. That's a calibration problem, and we wrote a whole post on it: setting pass/fail thresholds for LLM evals in CI without gaslighting yourself.
• Pass rate anywhere with wild score variance. Genuine non-determinism. This is the bucket the rest of this post is about.

Do this once as an audit, then keep doing it passively: emit per-case results as JUnit XML from your eval runner and track pass rates per case over time on your main branch, exactly the way you'd track any flaky test. A case that fails 8% of the time on main, where nothing changed, is flaky by definition. No judgment call required.

Step two: stabilize what you can

Pin everything pinnable

This is the unglamorous 80%. Pin the model to a dated snapshot. Pin the judge model too. Set temperature to 0 and pass a seed where the API supports one. Log the system_fingerprint so that when determinism breaks anyway, you can see whether the provider's backend changed underneath you.

resp = client.chat.completions.create(
    model="gpt-4o-2024-08-06",   # dated snapshot, never the moving alias
    temperature=0,
    seed=42,
    messages=messages,
)
log.info("system_fingerprint=%s", resp.system_fingerprint)

While you're here, separate infrastructure failures from eval failures in your reporting. A provider timeout should surface as an error, not a failed assertion. Retry errors; never silently retry assertion failures. Conflating the two is how teams convince themselves their evals are flakier than they are.

Replace exact match with tolerance bands

Exact-match assertions on free-form LLM output are a flakiness machine. If your assertion is output == expected_summary, you've written a test that fails whenever the model picks a synonym. Assert on properties instead: the output parses as valid JSON against a schema, it contains the three required entities, the judge score clears a threshold.

And when you assert on a score, give it a tolerance band derived from measured noise, not from vibes:

BASELINE = 0.86   # rolling mean over the last 50 main-branch runs
NOISE = 0.04      # ~2 standard deviations of that same window

def test_summary_quality():
    score = judge_score(generate_summary(FIXTURE))
    assert score >= BASELINE - NOISE, (
        f"score {score:.2f} below tolerance band "
        f"(baseline {BASELINE:.2f}, noise {NOISE:.2f})"
    )

The point of the band is that a failure now means something: the score moved further than normal run-to-run variance can explain. If your band has to be so wide that a real regression fits inside it, that's important information too. It means this eval, as written, cannot detect the regressions you care about, and you should redesign the case rather than keep flipping its coin in CI.

Use quorum assertions, not single shots

For genuinely stochastic behavior, one sample is not a measurement. Run the case k times and require m passes:

def quorum_pass(case_id: str, runs: int = 5, required: int = 4) -> bool:
    passes = sum(run_case_sync(case_id).passed for _ in range(runs))
    return passes >= required

A case with a true 90% pass rate fails a single-shot assertion one run in ten. Under a 4-of-5 quorum it fails about 8% of the time; at 7-of-10, under 2%. You're buying signal with inference spend, so do it deliberately: quorum the cases where the flake-rate audit showed real variance, and leave stable cases as cheap single shots. Note what quorum is not: it is not "retry until green." The pass criterion is defined up front and every run counts against it, which is also the version of this story you can defend in a change-management review.

Stabilize the judge

If the audit shows the generation is stable but the score wobbles, your judge is the flaky component. The fixes are the same in miniature: pin the judge model, use a rubric with concrete anchors instead of "rate 1 to 10," force structured output so parsing never fails, and for high-stakes gates, take a majority vote across three judge calls. A judge is a test dependency. You'd never let a randomly failing assertion library into your suite; hold the judge to the same standard.

Step three: quarantine what you can't fix yet

Some evals will still flake after all of that, usually the open-ended ones measuring qualities you can't fully pin down. You have three options: delete them, let them keep blocking merges, or quarantine them. Deleting throws away signal. Letting them block merges teaches your team to rerun red builds, which is the exact reflex that ships regressions, and the reruns themselves aren't free either. We benchmarked how much CI time rerun-driven retries actually burn, and eval reruns are worse because each one bills you for inference on top of compute.

Quarantine means the eval still runs on every commit and still reports results, but it can't fail the build:

jobs:
  evals-blocking:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python -m evals run --suite stable --junit-xml results/stable.xml

  evals-quarantined:
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - uses: actions/checkout@v4
      - run: python -m evals run --suite quarantined --junit-xml results/quarantined.xml

Three rules keep quarantine from becoming a landfill:

• Quarantine is data-driven, both directions. An eval goes in when its main-branch flake rate crosses a line you wrote down (say, any failure at unchanged inputs over the trailing 50 runs), and it comes out when the data says it's stable again. Because your eval runner emits JUnit XML per case, a flaky-test platform like BuildPulse can compute those pass rates and manage the quarantine set for you, the same way it does for conventional tests. Evals don't need special machinery here; they need the machinery you should already have.
• Every quarantined eval gets an owner and a ticket. Quarantine without an exit path is deletion with extra steps.
• Aggregate trends on quarantined evals still gate releases. An individual flaky eval can't block a merge, but if the quarantined suite's rolling pass rate drops ten points after a prompt change, a human looks before the release goes out.

For readers in regulated shops, this framing matters beyond hygiene. "This check is quarantined, tracked in ticket EV-212, trend-monitored, non-blocking by documented policy" is a defensible control. "We reran the pipeline until it passed" is not a sentence anyone wants to say in an audit. We've written about what auditors actually want from your CI evidence, and a documented quarantine process fits that model cleanly where ad-hoc reruns do not.

The triage loop

Run the loop continuously, not as a one-time cleanup. Measure per-case pass rates at fixed inputs so you know which evals are noisy and which thresholds are miscalibrated. Stabilize with pinned snapshots, seeds, property-based assertions, tolerance bands sized from measured noise, and quorums where variance is real. Quarantine the survivors with owners, exit criteria, and trend monitoring, so they inform without blocking.

The standard is simple to state: when your eval suite fails, an engineer's first instinct should be "what did I break," not "I'll rerun it." Every part of this playbook exists to protect that instinct. Lose it, and it doesn't matter how sophisticated your evals are. Nobody's listening to them anymore.