LLM evals are flaky tests: how to build CI for AI you can trust

The eval suite that cried wolf

A team I worked with shipped a RAG-backed support assistant last year. They did the responsible thing: a golden set of 200 questions, an LLM-as-judge scoring faithfulness and relevance, all wired into CI as a required check. Two weeks in, a PR that changed a logging statement failed the multi_doc_citation eval. The engineer reran the job. Green. Merged.

Within a month, "rerun the evals job" was tribal knowledge, right up there with which coffee machine works. And then a prompt refactor that genuinely broke citation formatting sailed through, because the engineer saw a red eval, sighed, and reran it twice until it passed. The suite had trained everyone to ignore it.

If you've run a conventional test suite at any scale, you recognize this disease immediately. It's a flaky test. The symptoms are identical: intermittent failures uncorrelated with code changes, rerun culture, and eventually a gate that gates nothing. The AI engineering world is busy reinventing testing from first principles, and in the excitement it's skipping the chapters the CI world already wrote in blood. Testing LLM applications is not a brand new discipline. It's flaky-test management with a probability distribution bolted on.

Why LLM evals flake, and it's not just temperature

The reflexive answer is "set temperature to 0." It helps, and it is nowhere near enough.

• Temperature 0 is not determinism. Greedy decoding removes sampling randomness, but floating point math on GPUs is non-associative, provider-side batching changes which computations get fused, and mixture-of-experts routing can shift token by token. People have documented byte-different outputs from identical temperature-0 requests to major APIs. You do not control the substrate.
• Pinned models drift anyway. A pinned snapshot pins weights, not the serving stack. Providers change inference infrastructure under stable model names, and your outputs move with it.
• LLM-as-judge doubles the noise. If your assertion is itself a model call, you've stacked one distribution on top of another. A borderline output plus a borderline judge is a coin flip with extra API cost.
• Retrieval drift. Re-embedded documents, a changed chunking strategy, or a rebuilt index can shuffle which passages land in context, and downstream scores wobble with no application code change at all.
• Plain old infrastructure. Rate limits, timeouts, 529s from the provider during peak hours. These are classic flaky-test causes wearing an AI costume, and they deserve classic remedies: retries with backoff, sane timeouts, and failure messages that say "provider timed out" instead of "eval failed."

So when an eval goes red, there are four candidate explanations: a real regression, sampling noise, judge noise, or infrastructure. Your CI renders all four as the same red X. That's the actual hard problem in CI for AI. Writing evals is the easy part. Interpreting a failure is where teams fall over.

Split the suite: assertions versus measurements

The first structural fix is admitting that your "eval suite" contains two fundamentally different kinds of checks, and they should not share a fate.

Deterministic assertions are properties that must hold on every single output: the response parses as valid JSON, the tool call arguments match the schema, the system prompt didn't leak, no email addresses appear in the output, latency stays under budget. These behave like unit tests. They should be hard gates on every PR, and when they fail, someone broke something.

def test_cancel_flow_emits_valid_tool_call(support_agent):
    resp = support_agent.run("Please cancel my subscription")
    assert len(resp.tool_calls) == 1
    call = resp.tool_calls[0]
    assert call.name == "cancel_subscription"
    # Pydantic validation, not vibes. Fails loudly on schema drift.
    CancelArgs.model_validate_json(call.arguments)

Statistical evals are qualities you measure across a distribution: faithfulness to retrieved context, answer relevance, tone. A single sample of "was this answer faithful?" is not a test result. It is one draw from a distribution, and treating one draw as pass/fail is how you end up with the rerun culture from the opening scene.

Most teams I've seen mix these freely in one suite with one pass/fail semantic. That's the original sin. Everything else in this post follows from keeping them apart.

Gate on pass rates, not single samples

For the statistical half, stop asserting on individual generations. Run each case n times and gate on the pass rate, ideally against a baseline you measured on your main branch rather than a number someone picked in a meeting.

import statistics

N_TRIALS = 8
BASELINE = 0.86  # measured weekly on main, stored in repo
SLACK = 0.05     # tolerated regression before we block

def case_pass_rate(case, n=N_TRIALS):
    outcomes = [judge_faithful(run_pipeline(case)) for _ in range(n)]
    return sum(outcomes) / n

def test_faithfulness_smoke_set():
    rates = {c.id: case_pass_rate(c) for c in SMOKE_SET}
    suite_score = statistics.mean(rates.values())
    worst = min(rates, key=rates.get)
    assert suite_score >= BASELINE - SLACK, (
        f"suite {suite_score:.2f} vs baseline {BASELINE:.2f}; "
        f"worst case: {worst} at {rates[worst]:.2f}"
    )

Two things about this pattern matter more than the arithmetic. First, the failure message names the worst case, so a red build starts an investigation instead of a shrug. Second, the baseline lives in the repo and gets re-measured on a schedule, so when the provider quietly shifts the model under you, the drift shows up as a baseline change on main, not as a mystery failure on some innocent PR.

Yes, this costs money. Eight trials across 200 cases is 1,600 pipeline runs per PR, which is absurd. So don't do that. Run a smoke set of 15 to 25 cases with trials on PRs, and the full set nightly. Cache aggressively: if neither the prompt, the retrieval config, nor the relevant code changed, yesterday's numbers are still true.

Your judge is a test dependency, so pin it and calibrate it

An LLM judge is a dependency of your test suite the same way a testing library is, except it has opinions and they change. Treat it accordingly:

• Pin the judge model to a specific snapshot, separately from your product model. Upgrading the judge should be a deliberate PR, never a side effect.
• Calibrate against humans. Keep a small labeled set (50 to 100 examples with human verdicts) and track judge agreement. If your judge agrees with humans 90% of the time, a 4-point swing in eval scores is inside the noise floor, and now you know that instead of guessing.
• Run judge upgrades side by side. Score the same frozen outputs with the old and new judge before switching. If scores move, that's judge drift, not product regression, and conflating the two will send someone on a two-day goose chase through prompt diffs.

The nastiest incidents I've seen in this space were all judge drift misdiagnosed as product regression. The product hadn't changed. The critic had.

Make evals speak JUnit so your CI tooling can actually help

Here's the move almost nobody makes, and it's the one with the best effort-to-payoff ratio: emit your eval results as JUnit XML, one <testcase> per eval case, and feed them into the same reporting pipeline as your regular tests.

<testsuite name="evals.faithfulness" tests="2" failures="1">
  <testcase classname="evals.faithfulness" name="billing_refund_policy" time="4.2"/>
  <testcase classname="evals.faithfulness" name="multi_doc_citation" time="5.1">
    <failure message="pass rate 0.50 below threshold 0.75">
      trials=8 passes=4 judge=gpt-4o-2024-08-06 baseline=0.88
    </failure>
  </testcase>
</testsuite>

Why bother? Because the moment eval results look like test results, twenty years of test-reliability tooling starts working for you. Failure history per case. Flake detection that flags multi_doc_citation as failing 15% of runs with no correlated code change. Quarantine, so a known-noisy eval case gets tracked and fixed instead of silently rerun. This is exactly the workflow we've written about for conventional suites in what flaky tests actually cost you and why quarantine beats rerun-until-green, and it transfers to evals almost verbatim. BuildPulse ingests that same JUnit XML, and it genuinely does not care whether the test behind a result was a Selenium click or an LLM judging a RAG answer. Statistically, a flaky eval and a flaky integration test are the same animal.

Wiring it into GitHub Actions is unremarkable, which is the point:

  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run eval smoke set
        run: python -m evals.run --set smoke --junit-out results/evals.xml
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - name: Upload results to BuildPulse
        if: "!cancelled()"
        uses: buildpulse/buildpulse-action@main
        with:
          account: ${{ vars.BUILDPULSE_ACCOUNT_ID }}
          repository: ${{ vars.BUILDPULSE_REPO_ID }}
          path: results/evals.xml
          key: ${{ secrets.BUILDPULSE_ACCESS_KEY_ID }}
          secret: ${{ secrets.BUILDPULSE_SECRET_ACCESS_KEY }}

PR gates versus the nightly drift watch

Put it together and the CI shape looks like this:

• On every PR: deterministic assertions as hard gates, plus the statistical smoke set with trials and generous slack. Fast enough that nobody routes around it, strict enough to catch a prompt change that torches faithfulness.
• Nightly on main: the full eval set with more trials and tighter thresholds. This is your drift detector. When the provider changes something under a pinned model name, this is where it surfaces, on a schedule, attributed correctly, instead of ambushing a random PR on Friday afternoon.
• Weekly: re-measure baselines, review quarantined eval cases, check judge-human agreement.

One more thing for readers in regulated shops, said without irony: if your eval suite is part of a change-management control, rerun-until-green isn't a workaround, it's a hole in the control. An eval gate that engineers bypass by mashing retry will not survive contact with an auditor, and it shouldn't. A quarantine list with owners and a documented pass-rate policy is something you can actually stand behind in a SOC2 review. "We reran it and it went green" is not.

Stop treating non-determinism as exotic

LLM applications didn't invent non-deterministic tests. Browsers, race conditions, and shared CI runners got there fifteen years ago; models just industrialized the problem. The teams shipping AI features with confidence aren't the ones who found a magic determinism switch, because there isn't one. They're the ones who noticed that an eval is a test, a noisy eval is a flaky test, and the playbook for flaky tests already exists: separate signal from noise, measure against baselines, quarantine what's noisy, and never let "rerun it" become the fix of record.

Your evals are only worth what a red result means. Right now, on most teams, it means "rerun it." That's fixable, and you already know how.