Flaky tests that only fail at midnight: hunting clock-dependent CI failures

The failure that only happens at 00:04 UTC

A payments team I worked with had a test that failed roughly once a week. Always in CI, never locally. The failure reports made no sense: a subscription that should have been "active" was "expired." Someone would hit rerun, it would pass, and everyone would move on. It took four months for anyone to notice the pattern, and the person who noticed wasn't an engineer. It was a night-shift SRE in another timezone who saw that every single failure landed between 00:00 and 00:09 UTC.

The test created a subscription expiring "today," then asserted it was still active. Run that at 2 PM and the subscription has ten hours left. Run it at four minutes past midnight, after the fixture computed "today" as yesterday because of a timezone mismatch between the test helper and the database, and the subscription was born dead.

This is the clock-dependent flaky test, and I want to convince you it deserves its own category in how you think about test reliability. Not because it's the most common class of CI flakiness (async race conditions win that prize), but because it's the class your existing defenses are structurally worst at catching.

Why retries can't see clock bugs

Most flaky-test mitigation, formal or informal, is built on one assumption: if a test fails nondeterministically, running it again samples the randomness again. Race conditions, shared-state pollution, resource contention: rerun a few times and the flake reveals itself, or at least gets out of your way.

Clock-dependent tests break that assumption because they aren't random at all. They're perfectly deterministic functions of when they run. Rerun the midnight failure at 00:20 and it might fail again. Rerun it at 9 AM during standup triage and it passes every time, a hundred times out of a hundred. The retry doesn't just fail to catch the bug, it actively manufactures evidence that the test is fine and CI was "just being flaky."

Worse, the failure schedule correlates with your risk calendar. Midnight bugs fire on overnight builds nobody is watching. Month-end bugs fire during the exact window when finance-adjacent teams are shipping billing changes. DST bugs fire twice a year, on a Sunday, and then vanish for six months. If a malicious adversary designed a flaky test to maximize confusion per failure, it would look like a clock bug.

The five usual suspects

Almost every time-based flake I've debugged falls into one of five patterns:

• Midnight boundary. The test computes "today" in one timezone and compares against a system that computes it in another. Anything asserting on date equality is a candidate.
• Month-end arithmetic. Code that adds "one month" to January 31 and gets a date that doesn't exist, or a test fixture that assumes every month has a 30th. These fail three or four days per year and pass the rest.
• DST transitions. "Add 24 hours" and "add one day" are different operations twice a year in most US and EU timezones. Duration math near a transition is quietly wrong.
• TTL and expiry off-by-ones. A token with a 60-second TTL, a test that takes 59.8 seconds on a warm runner and 61 on a cold one. This one masquerades as infra flakiness for months.
• Runner timezone drift. Your laptop is America/New_York, the CI runner is UTC, the staging database is America/Chicago because of a decision made in 2016. The test encodes one of these assumptions without saying so.

Notice what these have in common: none of them are bugs in the test's logic exactly. They're bugs in the test's implicit inputs. The wall clock is a global variable that every test reads and almost no test declares.

The fix: tests never read the wall clock

The durable fix is a policy, not a patch: production code may consult the clock through an injected dependency, and tests always control that dependency. In JavaScript with Jest, that means fake timers plus a pinned system time:

describe("subscription expiry", () => {
  beforeEach(() => {
    jest.useFakeTimers();
    // Pin to a hostile moment on purpose: 2 minutes before
    // month-end midnight UTC, during a US DST transition year.
    jest.setSystemTime(new Date("2025-03-31T23:58:00Z"));
  });

  afterEach(() => {
    jest.useRealTimers();
  });

  it("treats a subscription expiring today as active until 23:59:59", () => {
    const sub = createSubscription({ expiresOn: "2025-03-31" });
    expect(sub.isActive()).toBe(true);

    jest.setSystemTime(new Date("2025-04-01T00:00:01Z"));
    expect(sub.isActive()).toBe(false);
  });
});

Two things matter here. First, the test pins time explicitly, so it produces the same result at 2 PM on a laptop and 00:04 UTC on a runner. Second, it pins time to a nasty value. A test frozen at noon on June 15 proves nothing. Freeze at the boundary you're afraid of.

In Go, the same idea shows up as a clock interface instead of time.Now() scattered through business logic:

type Clock interface {
    Now() time.Time
}

type fakeClock struct{ t time.Time }

func (f *fakeClock) Now() time.Time { return f.t }

func TestSubscriptionActiveAtMonthEnd(t *testing.T) {
    clk := &fakeClock{t: time.Date(2025, 3, 31, 23, 58, 0, 0, time.UTC)}
    sub := NewSubscription(clk, "2025-03-31")
    if !sub.IsActive() {
        t.Fatal("expected subscription to be active before midnight")
    }
}

Yes, threading a clock through your code is mildly annoying. It's also the difference between a test suite whose results depend on the calendar and one that means the same thing every time it runs. If your team pushes back, ask them which async bugs they'd rather debug at quarter-end.

Detect it on purpose: hostile-time CI runs

You can also flip the problem around. Instead of waiting for the calendar to ambush you, make CI simulate the ambush. A scheduled GitHub Actions job that runs the suite under adversarial timezones catches whole classes of clock bugs before they ever flake on a real PR:

name: hostile-time-tests

on:
  schedule:
    - cron: "0 3 * * 1" # weekly, off the critical path

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        tz:
          - "UTC"
          - "Pacific/Kiritimati"   # UTC+14, tomorrow already
          - "Pacific/Pago_Pago"    # UTC-11, still yesterday
          - "America/New_York"     # DST transitions
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npm test
        env:
          TZ: ${{ matrix.tz }}

Pacific/Kiritimati and Pago_Pago bracket the date line: between them, "today" spans three calendar days. Any test that survives both timezones has earned some trust. Run this weekly on a schedule rather than on every PR; it's a detection net, not a merge gate, and it keeps the cost off your critical path.

Timestamps are the fingerprint

Hostile-time runs catch the bugs you can provoke. For the ones already loose in your suite, the tell is in your failure history: clock-dependent tests fail in clusters on the clock, not uniformly across the day. A genuinely random race condition fails at 10 AM about as often as at midnight. A clock bug fails at 00:00–00:15 UTC, or on the 31st, or the Sunday of a DST switch, and almost never otherwise.

Nobody spots that pattern from a Slack channel full of red-X notifications. You need failure history aggregated per test across weeks of builds, with timestamps attached. This is exactly the shape of analysis a flaky test detection platform is built for: BuildPulse ingests your JUnit output from every build and surfaces per-test failure patterns, so a test that fails 14 times in four months, all within ten minutes of midnight UTC, stops looking like noise and starts looking like a diagnosis. Even if you build this in a spreadsheet, build it. The time-of-failure column is the single highest-signal field you're currently throwing away.

The part your auditors will ask about

If you run change management under SOC2 or ISO 27001, your CI gate is probably a named control: changes require passing automated tests before deploy. Here's the uncomfortable question a sharp auditor will eventually ask: when a required check fails and someone reruns it until it's green, what does the green check actually evidence?

For a true infrastructure hiccup, a rerun is defensible. For a clock-dependent test, the rerun didn't resample randomness; it waited out the condition the test was checking. The month-end billing test that failed on March 31 and passed on April 1 may have been correct both times. The rerun didn't clear a false alarm, it suppressed a true one, and your change record now shows a clean gate for a change that a working test tried to block.

The defensible pattern is the one you'd want anyway: a known flaky test gets quarantined through an explicit workflow, with a ticket, an owner, and a record of when it left and rejoined the gate. That gives you a paper trail that says "we identified an unreliable signal, isolated it deliberately, and fixed it," instead of a rerun history that says "we clicked the button until the control agreed with us." Auditors are fine with the first story. The second one generates findings.

What to actually do about it

If you lead an org of a few hundred engineers, here's the short version:

• Add failure timestamps to your triage view. One column. If failures for a test cluster by time of day or day of month, treat it as a clock bug until proven otherwise.
• Set a rule: tests control time, always. Fake timers, injected clocks, pinned system time at hostile values. Enforce it in review for new tests; migrate old ones opportunistically.
• Schedule a weekly hostile-time run. Two or three adversarial timezones, off the PR path. Cheap, and it converts calendar ambushes into Monday-morning tickets.
• Route flakes through quarantine, not reruns. Especially if CI is a compliance control. The rerun button is where evidence goes to die.

Clock-dependent tests are a small slice of overall CI flakiness, but they're the slice that hides longest, fails at the worst possible moments, and turns your retry policy against you. The good news is they're also the most fixable class there is. A race condition takes real engineering to untangle. A clock bug takes a fake timer and the willingness to admit that midnight exists.