CI audit evidence: JUnit XML retention, traceability, and the flaky rerun problem

The request that starts the scramble

The auditor's ask sounded harmless: "For change ticket PAY-482, deployed March 14th, show us evidence that automated tests ran and passed before release."

The VP of Engineering forwarded it to the platform lead. The platform lead opened GitHub Actions, found the workflow run, and discovered two problems. First, the test artifacts had expired months ago, because the default artifact retention was still set to 90 days and the audit window covers a year. Second, the run was green, but only on attempt two. Attempt one had a failed integration test that someone re-ran without comment, without a ticket, and without anyone deciding whether the failure mattered.

So the honest answer to the auditor was: "Tests passed, eventually, for reasons nobody wrote down, and we can't show you the raw results." That's not a finding yet. It becomes one when it happens on three of the five changes they sample.

If you run engineering at a fintech, healthtech, or any SOC2 or ISO 27001 shop, your CI pipeline is not just a productivity tool. It's a control. And controls need evidence. This post is about what that evidence actually needs to look like, where JUnit XML fits, and how flaky tests and their reruns quietly corrupt the trail.

The three questions auditors actually ask

Auditors don't care about your test framework, your sharding strategy, or your p95 pipeline duration. Across SOC2 Type II, ISO 27001, and most fintech partner audits, the questions about automated testing reduce to three:

• Did the control operate? For this change, did the required tests actually run against the code that shipped?
• What was the result? Not "the check was green" but the actual recorded outcome, retrievable today, for a change from ten months ago.
• Is it traceable? Can you walk from a requirement or change ticket to the tests that cover it, to the specific run that gated the release?

Notice what's not on the list: coverage percentage, test counts, or how clever your suite is. Auditors sample changes and pull the thread. Your job is to make sure the thread doesn't snap at "the artifact expired" or fray at "well, we re-ran it and it passed."

JUnit XML is your evidence format, so treat it like one

Whether you run Jest, Vitest, Go, pytest, or JUnit itself, the common denominator is JUnit XML. It's ugly, it's from 2004, and it is the closest thing CI has to a universal evidence format: structured, timestamped, per-test outcomes that a human or a script can read years later without spinning up your toolchain. If your reporters aren't producing it cleanly yet, start there. We've written up the mechanics for getting trustworthy JUnit XML out of Jest and for Go via gotestsum.

The part most teams get wrong is retention. GitHub Actions artifacts max out at 90 days of retention on GitHub-hosted storage, and many orgs set it lower to control costs. A SOC2 Type II audit period is typically twelve months, and the fieldwork happens after the period ends. Do the math: evidence produced in month one needs to survive at least fourteen or fifteen months. Your CI provider's artifact store was never designed to be your evidence archive.

So split the two jobs. Keep uploading artifacts for day-to-day debugging, and separately ship the XML to storage you control, keyed by the things an auditor will ask about: repo, commit SHA, and run attempt.

- name: Run tests
  run: npx jest --ci --reporters=default --reporters=jest-junit

- name: Upload artifacts for debugging
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: junit-${{ github.run_id }}-attempt-${{ github.run_attempt }}
    path: reports/junit/*.xml
    retention-days: 90  # GitHub's ceiling, not your audit period

- name: Archive test evidence
  if: always()
  run: |
    aws s3 cp reports/junit/ \
      "s3://ci-evidence/${{ github.repository }}/${{ github.sha }}/attempt-${{ github.run_attempt }}/" \
      --recursive

Two details in that snippet do a lot of work. if: always() means you archive failures, not just passes. An evidence trail that only contains green runs is not an evidence trail; it's a highlight reel, and auditors know the difference. And github.run_attempt in the path means reruns land next to the original instead of on top of it. Hold that thought.

Put a lifecycle policy on the bucket (18 to 24 months covers most regimes), turn on object lock or versioning if your framework cares about tamper evidence, and you've converted "we think tests ran" into "here is the XML, here is the SHA it ran against, here is the timestamp."

Traceability: from requirement to test to run

Retention answers "what happened." Traceability answers "so what." In regulated environments, especially healthtech shops adjacent to IEC 62304 or fintechs with change-management commitments, someone eventually asks you to connect a requirement to the tests that verify it and the run that proved it.

You do not need a requirements-management suite to pass this bar. You need a convention, applied consistently, that survives in the XML. JUnit's <properties> element is the underused workhorse here:

<testsuite name="payments/refunds" tests="14" failures="0"
           timestamp="2025-03-14T14:22:07Z">
  <properties>
    <property name="commit_sha" value="9f2c1e4a"/>
    <property name="requirement" value="PAY-482"/>
    <property name="workflow_run" value="14203991"/>
    <property name="run_attempt" value="1"/>
  </properties>
  <testcase name="issues a refund within the settlement window"
            classname="refunds.settlement" time="1.204"/>
  <!-- ... -->
</testsuite>

Most reporters let you inject properties from environment variables, so the CI context flows in without anyone remembering to do it. For requirement IDs, the pragmatic approaches, roughly in order of effort:

• Ticket IDs in test names or describe blocks for the tests that exist specifically to verify a requirement. Greppable, visible in the XML, zero tooling.
• Tags or annotations (@requirement PAY-482 in a docstring, a custom Jest annotation, a Go build tag) extracted into properties at report time.
• A mapping file in the repo connecting requirement IDs to test paths, validated in CI so it can't rot silently.

The goal is that when the auditor samples PAY-482, someone can answer in five minutes with artifacts, not in five days with archaeology. Traceability that requires a senior engineer's memory is not a control. It's a bus-factor risk with a compliance flavor.

How reruns quietly rewrite history

Here's the uncomfortable part, and the reason a flaky-test company is writing about audits. Every mechanism your team uses to cope with flaky tests also edits the evidence trail. Usually in the direction of making failures disappear.

Framework-level retries are the worst offender. Configure jest.retryTimes(2) or pytest's flaky plugin, and a test that fails twice and passes on the third try can produce XML showing a clean pass. The failures happened. Code executed, assertions fired, something was wrong or at least nondeterministic. But the evidence artifact, the thing you just carefully archived for 24 months, says nothing happened. Your archive is now precise, durable, and wrong. We've covered what auto-retries actually cost you in CI minutes and masked defects; add "silently falsified evidence" to the invoice.

Job-level reruns are subtler. GitHub keeps prior attempts, so the data exists, but think about what the record shows: attempt one failed, attempt two passed, the branch protection check went green on attempt two, and the merge proceeded. When the auditor asks "who reviewed the attempt-one failure and determined it was safe to proceed?", the honest answer at most companies is nobody. The rerun button is an undocumented override of your change-management control, exercised dozens of times a week by anyone with write access. Described that way, in an audit report, it reads badly. Because it is bad. As we argued when we called flaky tests a SOC2 problem, "just rerun it" is a control bypass wearing a productivity costume.

And if your evidence archive doesn't key on run_attempt, it's worse: attempt two overwrites attempt one, and the failure is gone from your own records too. You've built a system that destroys the exact evidence an auditor would ask for.

What a defensible evidence trail looks like

You don't have to ban reruns or accept flaky failures blocking every release. You have to make the handling of flaky failures a documented decision instead of a reflex. Concretely:

• Archive every attempt, including failures. Object storage you control, keyed by repo, SHA, and attempt, retained past your audit period. If you did nothing else from this post, do this.
• Disable silent framework retries on the gating pipeline. If you must retry, make the reporter record the retry so the XML reflects reality. A pass-after-retry and a clean pass are different events and your evidence should say so.
• Classify every rerun. When attempt one fails and attempt two ships, there should be a record: this failure matched a known flaky test tracked in ticket X, or this failure was investigated and here's the conclusion. "Re-ran it, went green" is not a classification.
• Quarantine instead of retry. A quarantined test is a beautiful thing in an audit: here's the flaky test, here's when we detected it, here's the ticket, here's the removal from the gating signal, here's the fix and the date it returned. That's a functioning control with a paper trail. A retry loop is the same problem with the evidence shredded.
• Write the one-pager. A short internal doc stating what your CI gate verifies, what evidence it produces, where it's retained, and how flaky failures are handled. Auditors respond well to teams that have already described their own control. It changes the conversation from interrogation to confirmation.

This is also where flaky-test management stops being a developer-experience concern and becomes a compliance asset. A platform like BuildPulse that detects flaky tests from your JUnit XML, tracks each one's history, and manages quarantine gives you the artifact auditors actually want: a longitudinal record showing you knew about the instability, contained it deliberately, and fixed it on a timeline. The same data answers the awkward sampling question. When they pull a change that passed on attempt two, you can show the failing test was a documented, quarante-tracked flake rather than a mystery someone clicked past.

The trail is cheap; the scramble is not

None of this is expensive. Shipping XML to a bucket is twenty lines of YAML. Properties injection is a reporter config change. The quarantine workflow pays for itself in unblocked merges before compliance ever enters the picture.

What's expensive is the alternative: a director spending two weeks before fieldwork reconstructing ten months of test history from expired artifacts and Slack threads, and an auditor writing up "management could not produce evidence of test execution for sampled changes." Your CI already generates the evidence on every run. The only question is whether you keep it, whether it's traceable, and whether your flaky-test coping mechanisms are quietly rewriting it before anyone looks.

Keep the failures. Especially the failures. They're the part of the record that proves the control was real.