Testing AI-written code without wrecking your CI signal

The PR that looked perfect

A few weeks ago I watched a team merge a pull request with 412 lines of new test code. It was generated by an AI assistant in under two minutes, covered a session-expiry module that had zero tests before, and every check was green. The reviewer left one comment ("nice coverage bump") and approved. Honestly, I would have too. The tests read well. They asserted sensible things.

Nineteen days later, that same file was responsible for four red main-branch builds in one week, all at times nobody could reproduce locally. Nothing about the tests was wrong in a way a reviewer could see. Everything about them was wrong in a way that only shows up statistically.

This is the actual problem with AI-generated code testing, and it's not the one most posts talk about. The problem isn't that AI writes bad tests. It's that AI writes plausible tests at a volume that breaks your review process, and a meaningful fraction of them are flaky in ways that are invisible on run number one.

It's a volume problem wearing a quality costume

Here's the uncomfortable math. Say your engineers hand-wrote tests with a 2% latent-flake rate (a test that will eventually flake in CI). At 50 new tests a week, that's one new flaky test a week. Annoying, manageable.

Now give everyone Copilot or Cursor and watch test authorship go up 5x, because generating tests is the single most satisfying thing to delegate to an assistant. Even if the per-test flake rate stays identical (it doesn't, more on that below), you're now adding five flaky tests a week to a suite that probably already has a backlog. Your reviewers, meanwhile, did not get 5x more review time. They got the same 30 minutes per PR and 5x more test code to skim.

AI code quality discussions tend to fixate on whether the model writes correct logic. For tests, correctness is almost beside the point. A test that asserts the wrong thing fails immediately and gets fixed. A test that asserts the right thing nondeterministically passes review, passes CI once, and then spends the next six months eroding everyone's trust in your pipeline. If you've read our benchmark on what a 5% failure rate actually costs, you know how that story ends: engineers stop believing red means broken.

The flake patterns AI assistants love

Models are trained on public code, and public code is full of sleeps, real clocks, and shared fixtures. So the assistant reproduces those patterns confidently. Three show up constantly when I audit AI-written test suites.

1. Sleeping instead of waiting. Ask an assistant to test anything time-based and there's a decent chance you get this:

test('expires stale sessions', async () => {
  const session = createSession({ ttlMs: 100 });
  await new Promise((resolve) => setTimeout(resolve, 150));
  expect(session.isExpired()).toBe(true);
});

On your laptop, 150ms is an eternity. On a loaded CI runner sharing a box with three other jobs, the event loop can stall long enough that the timing assumption quietly breaks, or the inverse test ("does not expire early") fails because 99ms of wall time turned into 130ms. The correct tool is fake timers, and we've written up why fake timers beat sleep() every time. Assistants rarely reach for them unless your codebase already uses them heavily, because the training data mostly doesn't.

2. Trusting the real clock. This one is my favorite because it's a time bomb with a literal fuse:

test('invoice is due next month', () => {
  const invoice = createInvoice({ issuedAt: new Date() });
  expect(invoice.dueDate.getMonth()).toBe(new Date().getMonth() + 1);
});

This passes every day of the year except in December, when getMonth() returns 11 and the due date wraps to 0. It also gets weird on the 31st of any month. The assistant wrote a test that is green for weeks and then fails for everyone simultaneously on a specific calendar date. We've catalogued this whole genre in why your CI only fails at midnight.

3. Leaning on shared state. Assistants generate tests file by file, with no awareness that another generated file seeds the same database table or mutates the same module-level cache. Each file passes in isolation. Run them in parallel, or in a different order, and they collide. The individual tests are fine; the suite is flaky. No reviewer catches this by reading one PR, because the conflict lives across two PRs that may have been merged weeks apart.

None of these patterns are new. Humans wrote all of them first. What's new is the rate of production and the confidence of the prose around them. An AI-generated test comes with a tidy name, a clear arrange-act-assert structure, and often a helpful comment. It pattern-matches to "well-written test" precisely because the model optimized for looking like one.

Why review can't be your control

The instinctive response is "review AI-generated tests more carefully." I want to push back on that, not because review is worthless but because it's structurally the wrong layer for this defect class.

Flakiness is a statistical property. A test that fails 3% of the time passes 97% of your reviews of its output, by definition. Your reviewer sees one green run. They cannot see that the sleep is 20ms too tight on a slow runner, that the date math breaks in December, or that a file merged last sprint writes to the same Redis key. Asking humans to catch probabilistic failures by reading code is asking them to simulate a thousand executions in their head. They won't, and you shouldn't want them to spend their time trying.

Review is where you catch wrong assertions. Flakiness gets caught by running the code many times and keeping records. That's a machine's job.

The volume control: gates that scale with generation speed

If AI turned the test-authorship faucet up, your CI needs a proportional drain. Two mechanisms matter, and they work at different timescales.

Gate one: stress-run new tests before they land. The cheapest time to catch a flaky test is before it enters main. New and modified test files should run more than once in PR CI. Here's a minimal GitHub Actions version:

name: stress-new-tests
on: pull_request

jobs:
  stress:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Find changed test files
        id: changed
        run: |
          FILES=$(git diff --name-only "origin/${{ github.base_ref }}...HEAD" \
            | grep -E '\.(test|spec)\.(ts|tsx|js)$' || true)
          echo "files=$(echo "$FILES" | tr '\n' ' ')" >> "$GITHUB_OUTPUT"

      - name: Run changed tests 10 times
        if: steps.changed.outputs.files != ''
        run: |
          for i in $(seq 1 10); do
            echo "=== iteration $i ==="
            npx vitest run ${{ steps.changed.outputs.files }} || exit 1
          done

Ten iterations catches tests that flake often (a 30% flake rate fails at least once in ten runs with 97% probability). It will not catch the 1-in-500 test, and it will not catch cross-file state collisions, because you're running the new files alone. That's fine. This gate exists to filter the loud flakes cheaply, and to send a cultural signal: generated tests get interrogated, not waved through.

One warning: run the iterations with your suite's real parallelism settings. A sleep-based test that only fails under CPU contention will sail through ten sequential runs on an idle runner.

Gate two: longitudinal detection on main. The quiet flakes, the order-dependent ones, the December bombs: these only reveal themselves across hundreds of runs in real conditions. Catching them requires keeping test-level history and flagging tests whose outcomes disagree across identical commits. This is the layer BuildPulse operates at: it watches every result across your CI runs, identifies tests that fail nondeterministically, and quarantines them so a known-flaky assertion stops blocking unrelated merges. The point isn't the tool, it's the architecture. Somebody, or something, has to be counting.

What you should not do is set retries: 2 globally and move on. Blanket retries convert flakiness from a visible problem into an invisible tax, and the tax is bigger than most leaders think. We measured how much CI time flaky reruns actually burn, and if you're feeding a merge queue, retries make it worse, not better; merge queues industrialize flaky tests rather than fixing them.

The compliance angle nobody puts in the AI policy

If you're a SOC 2 or ISO 27001 shop, your CI gate is probably named in a change-management control. That has a sharp implication for AI-written tests that most AI usage policies miss entirely.

When a flaky, AI-generated test fails and an engineer clicks rerun until it's green, you now have a merge that your control says was verified and your run history says was verified on the third attempt with no code change. An auditor who samples that run will ask why. "The AI wrote a bad test" is not an answer that goes well in that meeting.

Quarantine, by contrast, is a defensible control: the test is flagged, excluded from the gate through a documented mechanism, tracked to remediation, and the whole lifecycle leaves an evidence trail. If your team is generating tests at AI speed, the quarantine path needs to exist before the flake backlog does. We've written about what auditors actually want from your CI if you're building that story out.

What I'd actually put in the engineering policy

If you lead a team that's adopting AI assistants (which at this point means every team), here's the short version I'd ship:

• Generated tests are held to the same determinism bar as generated code. No real clocks, no sleeps, no shared mutable fixtures. Put it in the repo's assistant rules file so the model sees it too; assistants follow local conventions surprisingly well when the conventions are written down.
• New and changed tests get stress-run in PR CI. Ten iterations minimum, with production parallelism.
• Flake detection runs on main, with quarantine as the response. Not global retries. Not rerun-until-green.
• Track flake rate as a first-class metric alongside coverage. Coverage going up while trust goes down is a net loss, and AI makes that trade very easy to make by accident.
• Attribute flakes back to their origin. If 40% of your new quarantined tests came from one generation workflow, that's a prompt problem or a template problem you can actually fix upstream.

AI assistants are a genuine gift for test authorship. I mean that without irony: modules that would have stayed untested forever now have suites, and the marginal cost of a regression test rounds to zero. But a test suite is only worth what its signal is worth. If generation speed goes up 5x and your verification machinery stays flat, you haven't bought coverage. You've bought noise with better formatting.

Turn the volume control before the noise turns it for you.