Setting pass/fail thresholds for LLM evals in CI without gaslighting yourself

The green check that meant nothing

I shipped a prompt change last spring that passed every eval in CI. Score 0.87, threshold 0.85, green check, merge. Two days later support tickets started rolling in about the assistant confidently inventing refund policies that don't exist.

The evals hadn't lied, exactly. They'd told me the average was fine. What they hadn't told me: on the previous commit that same suite scored 0.91, and the run before that 0.84, and the run before that 0.89. My "passing" 0.87 was inside the noise band of a metric that wandered around like a drunk on a treadmill. The threshold gate wasn't measuring quality. It was measuring luck.

This is the core problem with testing LLM applications in CI. You take a non-deterministic system, score its output with another non-deterministic system, and then draw a bright line at some number you picked because it felt round. Then you're shocked when the line means nothing.

Let's fix the threshold, because the threshold is where most eval pipelines quietly fall apart.

Why a single number and a single threshold is a trap

A traditional unit test is binary and deterministic. Given the same input, assertEqual(add(2,2), 4) passes every time or fails every time. That's the whole reason CI works — the signal is trustworthy. When it goes red, something changed.

LLM evals break that contract twice over:

• The model output varies run to run, even at temperature=0, because of batching, hardware, and provider-side changes you don't control.
• The scorer — whether it's an LLM-as-judge, embedding similarity, or a heuristic — adds its own variance on top.

So your eval score is a distribution, not a point. When you assert score >= 0.85 on a single run, you're sampling that distribution once and pretending the sample is the truth. Sometimes you catch a real regression. Sometimes you flag natural jitter. Sometimes you sail a real regression straight through because the dice came up warm.

We've written before about how flaky evals turn CI into a liar and why non-deterministic LLM tests need different handling than your unit suite. This post is the operational follow-up: given that the score is a distribution, how do you draw a line?

Step one: measure the noise before you set the line

You cannot pick a threshold you haven't earned. Before gating anything, run your eval suite N times on the same commit and look at the spread.

import statistics

scores = []
for _ in range(20):
    result = run_eval_suite(commit="main")  # same code, same data
    scores.append(result.mean_score)

print(f"mean:   {statistics.mean(scores):.4f}")
print(f"stdev:  {statistics.stdev(scores):.4f}")
print(f"min:    {min(scores):.4f}")
print(f"max:    {max(scores):.4f}")

If your mean is 0.87 and your standard deviation is 0.04, then a single run landing at 0.83 tells you nothing. That's one sigma of ordinary noise. A threshold at 0.85 in that world will flag roughly a third of clean runs. Congratulations, you've built a flaky test with extra API costs.

The number that matters isn't the mean. It's the width of the band. A metric with a stdev of 0.002 can support a tight threshold. A metric with a stdev of 0.05 can barely support any threshold at all — and that's a signal to fix the eval, not the gate.

Step two: gate on the delta, not the absolute

Absolute thresholds age badly. You set 0.85, the model improves, everyone's at 0.93, and the gate is now decorative. Or the dataset shifts and 0.85 becomes unreachable and everyone starts adding --skip-evals to their PRs. Either way the gate dies.

What you actually care about is: did this change make things worse than the baseline? Gate on the regression, not the raw score.

def eval_gate(pr_scores, baseline_scores, noise_stdev):
    pr_mean = statistics.mean(pr_scores)
    baseline_mean = statistics.mean(baseline_scores)
    delta = pr_mean - baseline_mean

    # Only fail if the drop exceeds 2x the natural noise band
    tolerance = 2 * noise_stdev
    if delta < -tolerance:
        return "fail", delta
    return "pass", delta

Running both the PR and the baseline in the same CI job cancels out a lot of shared variance — same provider weather, same day, same scorer version. A regression that shows up as a consistent delta against a freshly-measured baseline is far more trustworthy than a raw number compared to a threshold you set in Q2.

This is the same instinct behind measuring change failure rate correctly: the absolute rate matters less than the trend against a stable baseline, and both get poisoned when your underlying signal is noisy.

Step three: run enough samples to see a real effect

One run per eval is malpractice if your stdev is nontrivial. But 20 runs of a 500-case suite through a frontier API is real money and real minutes. So the question is: how many samples do you actually need to distinguish a real regression from noise?

The rough rule from the standard error of the mean: the noise in your averaged score shrinks by the square root of the number of runs.

standard error = stdev / sqrt(n_runs)

stdev 0.04, 1 run   -> SE 0.040   (can't see a 0.03 regression)
stdev 0.04, 4 runs  -> SE 0.020   (marginal)
stdev 0.04, 9 runs  -> SE 0.013   (0.03 regression is ~2 SE, detectable)

Decide the smallest regression you care about — say a 3-point drop in factual accuracy — then run enough samples that a drop that size sits at least two standard errors below the baseline. Anything smaller than your detectable effect, accept that CI won't catch it and stop pretending it will.

This also tells you where to spend. Cutting scorer variance (a more deterministic judge, a stricter rubric) lets you use fewer runs, which is cheaper than brute-forcing sample count. If you're using LLM-as-judge, read how to use it without fooling yourself first — a sloppy judge is the single biggest source of the noise you're trying to sample away.

Step four: separate the flaky eval from the real regression

Here's the failure mode that eats teams alive: an eval fails, someone reruns the job, it passes, they merge. Now you've trained your whole org to treat red evals as noise — including the reds that were real. That's exactly the "just rerun it" spiral that industrializes flakiness, and it's worse for evals because the reruns cost API dollars.

The fix is to make flakiness visible per eval case, not just per suite. Track how each individual assertion behaves across runs:

{
  "eval": "refund_policy_grounding",
  "runs_last_30d": 214,
  "pass_rate": 0.61,
  "score_stdev": 0.11,
  "verdict": "unstable_scorer"
}

A case that swings between pass and fail with a wide score stdev isn't guarding anything — it's a coin flip you're paying for. Quarantine it, fix the rubric or the scorer, and keep it out of the merge gate until it stabilizes. A case with a tight stdev that suddenly drops on one PR is your real signal. Those two things look identical if all you log is a suite-level pass/fail, which is why per-test flakiness tracking matters as much for evals as it does for any flaky test suite.

This is the part that generalizes past AI. CI for AI and CI for everything else share the same disease: when the signal is unreliable, humans stop trusting it, and an untrusted gate is worse than no gate because it still costs time. BuildPulse treats a flaky eval the same way it treats a flaky integration test — it fingerprints the test, tracks its pass/fail history across runs, and tells you whether a red is a genuine regression or the same case that's been unstable for three weeks. That's the difference between a gate you enforce and a gate you route around.

A threshold policy that survives contact with reality

Putting it together, here's the gate I actually run:

• Measure the noise band on a stable commit before setting any threshold. Re-measure when you change models or scorers.
• Gate on the delta against a same-run baseline, not an absolute number.
• Fail only when the regression exceeds ~2x the measured noise — anything inside the band is not signal.
• Sample enough runs that your smallest interesting regression clears two standard errors.
• Track per-case flakiness and quarantine unstable evals out of the merge gate instead of rerunning them into submission.

# .github/workflows/llm-evals.yml
- name: Run eval suite (PR + baseline, N samples)
  run: python eval_gate.py --samples 5 --tolerance-sigma 2 --baseline origin/main
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

None of this makes your LLM deterministic. That's not the goal. The goal is a CI gate whose green check means something — where a pass tells you "no regression larger than our noise floor" and a fail is worth stopping the line for. Get that, and you've turned your evals from a superstition into a control.

And if you're in a regulated shop where the CI gate is part of change management, the same discipline pays double. An auditor asking what evidence backs your test gate is a lot easier to answer when your threshold is a measured statistic instead of a number someone typed in once and forgot.