Your LLM evals are flaky tests. Run them like it.

The eval that passed on Tuesday

A team I worked with shipped a summarization feature behind a CI gate: every PR touching the prompt had to score at least 0.80 on a faithfulness eval before merge. Reasonable. Two weeks in, a PR that changed a logging statement failed the gate at 0.79. The author reran the job. It passed at 0.84. Nobody changed the prompt, the model, or the test data between those two runs.

Everyone shrugged and merged. Within a month, rerunning the eval job was muscle memory, and the gate was decorative. Three months in, an actual prompt regression sailed through on its second attempt, made it to production, and started confidently inventing refund policies.

If you've spent any time around flaky tests, this story is familiar down to the beats. A gate that fails intermittently for reasons unrelated to the change under review stops being a gate. It becomes a slot machine, and engineers learn to pull the lever. The AI-engineering crowd is currently rediscovering this from first principles, and I'd like to save you the tuition.

The core claim of this post: an LLM eval in CI is a flaky test by construction, and everything we know about managing flaky tests applies. The teams doing testing of LLM applications well aren't the ones who eliminated non-determinism. They're the ones who stopped pretending it wasn't there.

Why your evals are non-deterministic, even at temperature zero

The first instinct is always "set temperature to 0 and the outputs are deterministic." They aren't, and it's worth understanding why, because it changes how you design the gate.

• Floating point and batching. Inference on GPUs is not bitwise reproducible across runs. Batch composition, kernel selection, and reduction order all shift logits by tiny amounts. When two tokens have near-equal probability, a nudge in the sixth decimal place flips which one greedy decoding picks, and the completions diverge from that token onward. Some providers expose a seed parameter; the fine print says best effort, and they mean it.
• The provider changes the model under you. Aliases like gpt-4o or claude-sonnet-latest are moving targets. Even pinned snapshots get infrastructure changes. Your eval scores can drift with zero commits in your repo.
• LLM-as-judge doubles the dice. If a second model scores the output, you've composed two stochastic systems. Judge models show position bias, length bias, and run-to-run variance of their own. A 0.79 vs. 0.84 spread on identical inputs is completely ordinary.
• RAG adds retrieval variance. Index rebuilds, tie-breaking in ranking, and embedding model updates change which chunks land in context, which changes everything downstream.

So when your eval flips from pass to fail with no diff, that isn't a bug in your harness. It's the harness accurately reporting that you built an assertion on a distribution and sampled it once.

A single-sample eval with a hard threshold is a coin flip wearing a lab coat.

Split correctness from quality, and test them differently

Before fixing the eval layer, shrink it. A big fraction of what teams stuff into "LLM evals" is actually deterministic and belongs in ordinary tests:

• Does the output parse as the JSON schema you demanded? Deterministic assertion.
• Did the agent call the tool with valid arguments? Deterministic assertion.
• Does the retrieval layer return the known-relevant chunk for a fixed query against a fixed index? Deterministic, if you pin the index.
• Does the prompt template render correctly with edge-case inputs? Deterministic.

These should run on every PR, fail hard, and never get a rerun pass. Reserve the statistical machinery for the genuinely fuzzy questions: is the summary faithful, is the answer grounded, is the tone acceptable. Mixing the two is how you end up statistically gating things that should be exact, and exactly gating things that are statistical. Both failure modes erode trust in the signal, which is the whole ballgame. If your engineers don't trust the red X, it doesn't matter how sophisticated your eval suite is.

Assert on the distribution, not the roll

For the quality layer, the fix is the same one statisticians would have suggested on day one: sample more than once and gate on an aggregate. Here's the shape of it in pytest:

# evals/test_summary_faithfulness.py
import statistics

from myapp.llm import summarize
from myapp.evals.judges import judge_faithfulness

N_SAMPLES = 6            # completions per document
SCORE_THRESHOLD = 0.75   # what counts as a "good" single sample
MIN_PASS_RATE = 0.80     # fraction of samples that must clear it


def test_summary_faithfulness(golden_docs):
    per_doc_pass_rates = []
    for doc in golden_docs:
        scores = [
            judge_faithfulness(doc.text, summarize(doc.text))
            for _ in range(N_SAMPLES)
        ]
        pass_rate = sum(s >= SCORE_THRESHOLD for s in scores) / N_SAMPLES
        per_doc_pass_rates.append(pass_rate)

    overall = statistics.mean(per_doc_pass_rates)
    assert overall >= MIN_PASS_RATE, (
        f"faithfulness pass rate {overall:.2f} below {MIN_PASS_RATE} "
        f"(per-doc: {[f'{r:.2f}' for r in per_doc_pass_rates]})"
    )

Three design decisions in there matter more than the exact numbers:

1. Sample count is a real knob. With one sample, an eval that's genuinely fine 90% of the time fails one PR in ten for no reason. With six samples and a pass-rate gate, the false-failure rate drops enough that a red result actually means something. Yes, it costs more tokens. Compare that to the cost of every engineer treating your gate as noise.
2. The assertion message carries the distribution. When it fails, the reviewer sees per-document pass rates, not a bare 0.79 < 0.80. That's the difference between "which doc regressed?" and "rerun it and hope."
3. Thresholds are calibrated, not vibes. Run the current production prompt through the harness 50 times before you set the gate. If baseline pass rates range from 0.78 to 0.92, a 0.80 threshold guarantees intermittent failures on unchanged code. Set the gate below the observed baseline floor, then tighten it as the product improves.

If you want to catch regressions smaller than your gate allows, don't tighten the PR gate. Track the trend on a schedule instead, which brings us to CI layout.

Wire it into CI honestly

Two tiers work well. A small, sampled smoke eval on PRs that touch prompts or LLM code, and a full eval suite on a nightly schedule with more samples and the whole golden set.

name: evals
on:
  pull_request:
    paths:
      - "prompts/**"
      - "app/llm/**"
      - "evals/**"
  schedule:
    - cron: "0 6 * * *"   # nightly full run

jobs:
  evals:
    runs-on: ubuntu-latest
    env:
      MODEL_SNAPSHOT: gpt-4o-2024-08-06   # pinned snapshot, never an alias
      JUDGE_SNAPSHOT: gpt-4o-2024-08-06
      EVAL_SEED: "42"                      # best effort, but take it
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements-eval.txt
      - name: Run evals
        run: |
          SUITE=$([ "${{ github.event_name }}" = "schedule" ] && echo full || echo smoke)
          pytest "evals/$SUITE" --junitxml=reports/evals.xml
      - name: Report results
        if: always()
        uses: buildpulse/buildpulse-action@main
        with:
          account: ${{ secrets.BUILDPULSE_ACCOUNT_ID }}
          repository: ${{ secrets.BUILDPULSE_REPOSITORY_ID }}
          path: reports/evals.xml
          key: ${{ secrets.BUILDPULSE_ACCESS_KEY_ID }}
          secret: ${{ secrets.BUILDPULSE_SECRET_ACCESS_KEY }}

A few things that separate a trustworthy setup from a decorative one:

• Pin model snapshots in config, and change them via PR. A model bump should show up in your diff history like any dependency upgrade, with the eval delta attached. When scores move, you want to know whether it was your prompt or their weights.
• Emit JUnit XML. This looks like a boring detail. It's the load-bearing one. The moment each eval case is a named test case in a standard format, your entire test-observability stack applies: history per eval, failure rates over time, ownership, annotations on the PR. Evals stop being a bespoke artifact only one engineer can interpret.
• Record the context. Log model snapshot, prompt hash, and dataset version as test properties. "Faithfulness dropped Thursday night" is a mystery. "Faithfulness dropped Thursday night, same prompt hash, same dataset, pinned snapshot" is a provider incident, and you can respond accordingly.

Track eval flakiness like you track test flakiness

Here's the bridge most AI teams haven't crossed yet. Once evals emit standard test results, you can ask the same question you ask of any test suite: which of these fail intermittently on unchanged code? That's precisely what flaky-test detection does, and it works on eval results without modification. When BuildPulse ingests those JUnit reports, an eval that flips pass/fail across retries or across identical commits surfaces just like a flaky integration test would, with a failure-rate history you can point at in a planning meeting.

And the playbook transfers:

• An eval with an unstable pass rate gets quarantined, not deleted and not rerun into submission. Quarantine keeps it running and reporting while removing its power to block merges, so you keep the data while you recalibrate the threshold or raise the sample count.
• A newly unstable eval is a signal, not an annoyance. If faithfulness variance doubled with no code change, something moved underneath you: the judge, the snapshot, the retrieval index. Variance itself is a monitor.
• Rerun-until-green is the one move you can't afford. This deserves emphasis for anyone running CI as a change-management control under SOC 2 or similar. A required eval check that engineers routinely rerun until it passes is a control that doesn't control anything, and that's an awkward thing to explain in an audit. A documented quarantine process with tracked failure rates and remediation is defensible. A culture of lever-pulling is not. The cost of flaky tests was never just wasted compute; it's the slow death of the signal, and evals die the same way.

The checklist

If you're standing up CI for AI features this quarter, here's the short version:

• Move every deterministic check (schemas, tool calls, parsing, pinned retrieval) into ordinary hard-failing tests.
• For quality evals, sample N completions per case and gate on pass rate, never on a single roll.
• Calibrate thresholds against 50+ baseline runs before enforcing them. Set the gate below the baseline floor.
• Pin model and judge snapshots. Upgrade them through PRs with eval deltas attached.
• Emit JUnit XML and feed it to the same flake-detection and history tooling as the rest of your suite.
• Quarantine unstable evals with a paper trail. Never normalize the rerun button.

None of this makes LLM behavior deterministic. Nothing will. But your CI gate doesn't need deterministic outputs. It needs a known false-failure rate, calibrated thresholds, and a paper trail when something drifts. Testing LLM applications is a statistics problem stapled to a test-reliability problem, and the second half is one we already know how to solve. Treat your evals like the flaky tests they are, and they'll start telling you the truth.