Your LLM evals are flaky, and your CI is lying to you

The build that failed for no reason

A few months ago I watched a teammate rerun a CI job four times to merge a one-line prompt tweak. The change was trivially correct. The eval suite disagreed — twice. Then it agreed. Then it disagreed again with a different assertion. He shipped on the third green, closed his laptop, and never mentioned it in standup.

That's the moment your CI signal dies. Not with a bang, but with a Slack message that says "just rerun it, the LLM thing is flaky."

If you've shipped anything with an LLM in the loop, you know this feeling. You wrote evals because you're a responsible engineer. You wired them into CI because you wanted a gate. And now the gate opens and closes on its own, and nobody trusts it. You've built a very expensive random number generator that occasionally blocks deploys.

Let's fix that. Not by pretending LLMs are deterministic — they aren't, even at temperature=0 — but by treating eval non-determinism as the flaky-test problem it actually is.

Why LLM evals are flaky in ways unit tests never were

A flaky unit test usually has one root cause: a race, a shared fixture, a clock, a real network call that should've been mocked. Annoying, but bounded. You find the shared state, you kill it, the test goes green forever.

LLM evals are flaky along more axes at once:

• Model non-determinism. Even temperature=0 doesn't guarantee identical outputs. Floating-point non-associativity across GPU batches, MoE routing, and provider-side load balancing all leak variance into your "deterministic" call.
• The judge is also a model. If you're using LLM-as-judge, your assertion itself is a probabilistic function. Now you have two dice.
• Provider drift. gpt-4o today is not gpt-4o from six weeks ago. A green suite can go red overnight with zero commits on your side.
• Semantic assertions. "Does this answer mention the refund policy?" is a fuzzy check. Substring matching is brittle; embedding similarity has a threshold you picked by vibes.

So when your eval suite goes red, you genuinely don't know if you broke something or if the dice came up wrong. That ambiguity is the whole problem. A test you can't trust when it fails is worse than no test, because it trains the team to ignore red — and that habit doesn't stay quarantined to the eval suite. It spreads to the tests that actually matter.

Step one: separate the two kinds of failure

Before you can gate on evals, you have to know whether a failure is a regression or variance. These need different treatment, so stop mixing them into one pass/fail bit.

The cleanest lever is to run each eval case n times and score the distribution, not a single roll:

import statistics

def eval_case(case, n=5):
    scores = [score_once(case) for _ in range(n)]
    return {
        "case": case.id,
        "mean": statistics.mean(scores),
        "stdev": statistics.pstdev(scores),
        "min": min(scores),
        "pass_rate": sum(s >= case.threshold for s in scores) / n,
    }

Now you can write assertions that respect the physics. Instead of "this case must pass," you assert "this case passes at least 4 of 5 times" or "mean score >= 0.8 with stdev < 0.1." A single unlucky roll no longer paints the build red, and a genuinely degraded prompt — one that drops from a 0.95 pass rate to 0.6 — still gets caught.

Yes, this costs 5x the tokens per case. Run the full n-sample sweep nightly and a cheaper single-pass smoke set on every PR. Which brings us to the structure.

Step two: tier your suite like you mean it

Not every eval belongs on the PR-blocking path. Testing LLM applications well means being honest about which checks are cheap and deterministic versus which are expensive and probabilistic.

• Tier 0 — deterministic contract tests. No model call. Does the function return valid JSON? Does the tool-call schema validate? Does the RAG retriever return the expected doc IDs for a fixed query and a frozen index? These are ordinary tests and should be flaky-free. Gate hard on them.
• Tier 1 — cheap semantic smoke. A small, curated set of golden cases with a cheap judge (or plain assertions). Runs on every PR. Fast, mostly stable, blocks merge.
• Tier 2 — full eval sweep. The n-sample distribution run across your whole dataset, LLM-as-judge included. Runs nightly or pre-release. Reports trends. Does not block individual PRs — it opens issues.

The mistake I see most: teams shove Tier 2 onto the PR gate because it feels rigorous. It isn't rigor, it's self-harm. You get a slow, expensive, flaky gate that everyone learns to rerun. Rigor is knowing what to block on.

# .github/workflows/llm-evals.yml
jobs:
  contract-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pytest tests/contract -q   # Tier 0, hard gate

  smoke-evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pytest tests/evals/smoke --maxfail=1   # Tier 1

  full-sweep:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python -m evals.sweep --samples 5 --report junit.xml   # Tier 2

Step three: pin what you can, snapshot the rest

You can't make the model deterministic, but you can shrink the surface area that moves.

• Pin model versions explicitly. Use gpt-4o-2024-08-06, not gpt-4o. When you bump the pin, that's a commit — a reviewable, blameable event. Provider drift stops being a mystery and becomes a diff.
• Freeze your RAG inputs. For retrieval tests, snapshot the index and the embeddings. If you're testing retrieval quality, the corpus is part of the test fixture. A moving corpus is a moving test.
• Set a seed where the provider supports it. It reduces variance; it does not eliminate it. Don't build your assertions on the assumption that it does.
• Record judge rationales. When LLM-as-judge fails a case, log why it failed. A raw score of 0.3 tells you nothing at 2 a.m. "Judge: answer omitted the 30-day window" tells you everything.

This is the same discipline that makes any test suite trustworthy: control your inputs, make change explicit, and record enough context to debug the failure without rerunning it. The difference between a flaky test and a real failure is almost always whether you captured enough to tell them apart after the fact.

Step four: measure flakiness instead of arguing about it

Here's where CI for AI stops being folklore. "The evals are flaky" is a vibe. "Case refund_policy_v3 has a 68% pass rate over the last 40 runs with no code changes" is data — and it's actionable.

You want per-case flake tracking across runs: pass rate over time, correlated with model version, prompt version, and commit. Once you have that, quarantine becomes a decision instead of a panic. A case that's genuinely non-deterministic gets moved out of the blocking tier and flagged for redesign — maybe the assertion is too tight, maybe the case is genuinely ambiguous and your product needs to handle both answers.

This is exactly the problem BuildPulse was built for on the traditional-testing side: it ingests your JUnit output, tracks pass/fail history per test across runs, and tells you which tests are flaky versus which are actually broken — then quarantines the flaky ones so they stop blocking merges while staying visible. LLM evals emit JUnit XML like anything else. If you're already reporting eval results in that format, the same flake detection and quarantine workflow applies. Your eval cases become just another test suite with a flake score, which is exactly how you should be thinking about them.

The compliance angle nobody wants to say out loud

If your CI gate is part of a change-management control — and in a SOC 2 or fintech shop it usually is — "just rerun it until it's green" is not a neutral habit. It's a documented control being bypassed by a human who decided the machine was wrong. That's the kind of thing that turns into an audit finding.

A tiered eval suite with explicit, distribution-based thresholds gives you a defensible answer. The gate blocks on deterministic contract tests and a stable smoke set. The probabilistic sweep informs but doesn't block, and its flakiness is tracked and quarantined on purpose, with a record of why. "We reran it four times" becomes "this case is a known non-deterministic eval, quarantined on 2024-11-03, tracked for redesign." One of those survives a review. The other one you explain to an auditor while sweating.

What to do Monday

You don't need to rebuild everything. Start here:

1. Split your one eval job into a hard-gating contract tier and a reporting-only sweep. Do this first; it stops the bleeding.
2. Convert your PR-blocking assertions from single-roll to n-of-m pass rates.
3. Pin your model versions and freeze your RAG fixtures so drift shows up as a diff.
4. Emit JUnit XML from your eval runner and start tracking per-case flake rate over time.

The goal isn't a perfectly deterministic LLM. That doesn't exist and chasing it will burn a quarter. The goal is a CI signal you believe — one where red means something changed and green means ship it. Everything else is a coin flip with extra steps.