Flaky LLM evals: how to test AI apps without your CI lying to you

The eval that failed for no reason

I shipped a summarization feature last year that had a green test suite and a red on-call channel. The eval suite passed on my machine, passed on the first CI run, then failed on the rerun with no code change. Same prompt, same model, same fixtures. The only thing that changed was the dice.

Welcome to testing LLM applications, where your test subject has opinions and mood swings.

If you come from deterministic software, flaky tests are already the enemy — a test that passes and fails on identical inputs erodes trust until people stop reading the results. We've written before about how flaky tests quietly destroy your CI signal. With LLMs, non-determinism isn't a bug in your test harness. It's a property of the thing you're testing. That changes the whole game.

Why LLM evals are flaky by construction

There are at least four independent sources of non-determinism in a typical LLM eval, and most teams only account for one.

• Sampling. Any temperature > 0 means the model samples from a distribution. Two runs, two outputs.
• Provider-side drift. The vendor silently updates the model behind gpt-4o or claude-sonnet. Your prompt didn't change; the weights did.
• The judge. If you're using LLM-as-judge to score outputs, your grader is itself a stochastic model. You've now stacked two random variables.
• Retrieval. In a RAG pipeline, embedding drift or a reindex changes which chunks get pulled, which changes the answer, which changes the score.

Here's the trap. You write an assertion like this:

def test_summary_quality():
    out = summarize(ARTICLE)
    score = judge(out, rubric="faithful, concise")
    assert score >= 0.8

That >= 0.8 looks like a normal threshold. It's actually a coin flip with a weighted coin. If your true score distribution is centered at 0.82 with a standard deviation of 0.05, you'll fail this test roughly a third of the time. Not because the model regressed. Because you sampled the left tail.

Then someone reruns CI, it goes green, and the PR merges. That's not a passing test. That's a slot machine you eventually walk away from a winner.

Rule one: make the deterministic parts deterministic

Before you reach for statistics, kill every source of randomness you don't actually need to test.

Set temperature=0 and a fixed seed for anything that isn't specifically testing sampling behavior. Yes, temperature=0 isn't perfectly deterministic on most hosted APIs — floating point and batching still leak through — but it collapses the variance by an order of magnitude.

response = client.chat.completions.create(
    model="gpt-4o-2024-08-06",  # pin the exact snapshot, not the floating alias
    temperature=0,
    seed=42,
    messages=messages,
)

Pin the model snapshot, not the alias. gpt-4o is a moving target; gpt-4o-2024-08-06 is a contract. When the vendor ships a new snapshot, that becomes a deliberate upgrade you test, not a surprise your Tuesday deploy inherits.

Cache aggressively. If a fixture's input hasn't changed, you shouldn't be paying for — or re-rolling — a fresh generation on every CI run. Record-and-replay (VCR-style cassettes for LLM calls) turns most of your eval suite into fast, deterministic unit tests that only hit the network when you explicitly re-record.

@llm_cassette("summary_faithfulness.json")
def test_summary_faithfulness():
    out = summarize(ARTICLE)
    assert contains_no_hallucinated_dates(out)

The more of your suite you can push into this deterministic bucket, the smaller and more honest the genuinely stochastic part becomes.

Rule two: for the stochastic part, test the distribution, not the sample

Some behavior you do want to test under realistic sampling — production runs at temperature=0.7, and you care whether quality holds up. For those, a single assertion is statistical malpractice.

Run N samples and assert on an aggregate with a margin that reflects the variance:

def test_summary_quality_distribution():
    scores = [judge(summarize(ARTICLE)) for _ in range(20)]
    mean = statistics.mean(scores)
    p10 = sorted(scores)[2]  # ~10th percentile of 20 samples

    # gate on the floor, not the average — one great run shouldn't hide bad ones
    assert p10 >= 0.75, f"p10 fell to {p10:.2f}"
    assert mean >= 0.82, f"mean fell to {mean:.2f}"

Now a failure means something moved. You're testing "does this feature reliably produce good summaries" instead of "did this one roll come up 6." Pick N based on how much variance you can tolerate and how much you're willing to spend — 20 runs of a cheap model is nothing; 20 runs of a frontier model on 500 fixtures will show up on the bill.

The expensive part is the judge. If you're grading with an LLM, calibrate it once against human labels, pin its snapshot and temperature, and version the rubric like source code. A judge that silently gets stricter is indistinguishable from a model regression, and you will burn a full afternoon before you figure out which one moved.

Rule three: separate "broke" from "drifted"

Deterministic tests answer did this break? Distribution tests answer did quality drift? These are different questions with different failure modes and they belong in different CI stages.

• PR gate (fast, deterministic): schema validation, structured-output parsing, refusal behavior, injection guards, cassette replays. These must be green to merge. No reruns, no excuses.
• Nightly / pre-release (slow, statistical): full eval sets with N samples per case, judge scoring, trend tracking. These gate the release, not the individual PR.

The reason to split them is trust. Your PR gate has to be something engineers believe. The moment a developer sees an LLM eval fail and their reflex is "just rerun it," you've lost — because that reflex doesn't stay confined to the flaky eval. It spreads to the whole suite, including the deterministic tests that were telling the truth. In a change-management or SOC2 context, "rerun until green" isn't just sloppy; it means your CI gate isn't actually a control anymore. It's decoration.

We've made this argument about ordinary flaky tests too — the real cost isn't the wasted minutes, it's the erosion of trust in the signal. LLM evals just make the failure mode arrive faster.

Treat eval flakiness like any other flaky test

Here's the mindset shift. Once you've split deterministic from statistical, any remaining flakiness in your deterministic tier is a real bug — same as anywhere else. A cassette test that fails intermittently means your cache key is wrong, or a timestamp leaked into your prompt, or your parser is fragile. Track it the way you'd track any flaky test.

This is where treating LLM tests as first-class CI citizens pays off. You want the same instrumentation you'd use for a normal suite: per-test failure rates over time, flake detection, quarantine for the ones that are noisy while you fix them. BuildPulse does this by ingesting your JUnit output and flagging tests that pass and fail on the same commit — and an LLM eval emits JUnit XML like anything else:

- name: Run LLM evals
  run: pytest tests/evals --junitxml=results.xml

- name: Upload results to BuildPulse
  if: always()
  run: npx buildpulse-action --results results.xml

The useful signal isn't "this eval failed once." It's "this eval has a 12% failure rate that's been climbing for a week," which almost always means either a provider changed something under you or your fixtures rotted. That's the difference between debugging a specific run and watching the health of your AI test suite as a system.

What good looks like

A team that has this figured out can answer three questions without flinching:

1. Can we tell a real regression from sampling noise? Yes — deterministic tier catches breaks, distribution tier catches drift, and neither one gets rerun into submission.
2. If the vendor ships a new model snapshot, do we find out from a test or from a customer? From a test, because the snapshot is pinned and the upgrade is a PR.
3. When an eval fails in CI, does anyone actually believe it? Yes — which is the whole point.

Testing LLM applications isn't fundamentally harder than testing anything else. It's that the non-determinism you can usually pretend doesn't exist is now sitting in the middle of your test subject, refusing to be ignored. Design for it. Isolate it. Measure it. And stop letting "just rerun it" launder a coin flip into a green check.

CI for AI is still CI. The signal is the product.