Query your CI health from a script: a read API for flaky tests, coverage, and runs

The dashboard you actually want doesn't exist yet

Every CI vendor ships a dashboard. None of them ship your dashboard.

You want the flakiest tests in the payments service, joined against last week's incident tickets, filtered to the two teams that own the release train, and posted to the channel where the on-call actually reads things. Nobody's product page has that. So you either learn to live with the built-in view, or you screen-scrape a UI that changes every quarter.

The better move is a read API. If your CI analytics data lives behind a stable, scriptable endpoint, you stop fighting the dashboard and start building the exact three things that matter: a view your leadership trusts, an alert that fires before a merge, and a release gate that says no on your terms.

This post is a walkthrough of the BuildPulse Platform API from that angle — pulling flaky-test history, coverage, and recent runs from a script, and wiring them into things you'd actually deploy. If you use a different tool, the shapes here map cleanly enough; the point is the pattern, not the vendor.

First principle: read paths should be boring

A good analytics API is boring in the best way. Authenticate with a token, GET a resource, get JSON back, paginate when there's more. No GraphQL schema archaeology, no webhooks you have to stand up a listener for just to answer "how flaky is this test."

Start with a token scoped to read-only. You do not want your Slack bot holding write credentials to your CI config.

export BUILDPULSE_TOKEN="bp_live_..."
export BUILDPULSE_ORG="acme"

curl -s https://api.buildpulse.io/v1/orgs/$BUILDPULSE_ORG/flaky-tests \
  -H "Authorization: Bearer $BUILDPULSE_TOKEN" \
  -H "Accept: application/json" | jq '.data[0]'

A single flaky-test record looks roughly like this:

{
  "id": "ft_9f21",
  "name": "Checkout > applies promo code before tax",
  "suite": "payments",
  "file": "spec/checkout/promo_spec.rb",
  "first_seen": "2024-11-03T08:12:00Z",
  "last_seen": "2025-01-14T22:41:00Z",
  "flake_rate_7d": 0.14,
  "flake_rate_30d": 0.06,
  "total_runs_30d": 812,
  "status": "quarantined",
  "owner_team": "payments"
}

That's the whole game. Two flake rates over two windows, run counts to tell you whether the rate is meaningful, and enough metadata to route the problem to a human. Everything below is joining, filtering, and shipping.

Pull flaky-test history and rank by cost, not by count

The naive question is "which tests are flaky." The useful question is "which flaky tests are costing me the most." A test that flakes 40% of the time but runs twice a week is a footnote. A test that flakes 4% of the time across 3,000 runs is stealing hours of engineer attention every week.

Here's a small Python script that pulls the full flaky-test list, paginates, and ranks by an estimated weekly waste — flake rate times run volume times a rough minutes-per-retry cost.

import os
import requests

BASE = "https://api.buildpulse.io/v1"
ORG = os.environ["BUILDPULSE_ORG"]
HEADERS = {"Authorization": f"Bearer {os.environ['BUILDPULSE_TOKEN']}"}

def paginate(path, params=None):
    params = dict(params or {})
    url = f"{BASE}/orgs/{ORG}/{path}"
    while url:
        r = requests.get(url, headers=HEADERS, params=params)
        r.raise_for_status()
        body = r.json()
        yield from body["data"]
        url = body.get("links", {}).get("next")
        params = None  # next link already carries the cursor

MINUTES_PER_RETRY = 6

tests = list(paginate("flaky-tests", {"window": "30d"}))

for t in tests:
    weekly_runs = t["total_runs_30d"] / 4.3
    t["weekly_waste_min"] = round(
        weekly_runs * t["flake_rate_7d"] * MINUTES_PER_RETRY
    )

top = sorted(tests, key=lambda t: t["weekly_waste_min"], reverse=True)[:10]
for t in top:
    print(f"{t['weekly_waste_min']:>5} min/wk  {t['suite']:<12} {t['name']}")

Output you can actually take to a planning meeting:

  247 min/wk  payments     Checkout > applies promo code before tax
  183 min/wk  auth         SSO > refreshes token near expiry
  119 min/wk  search       Indexer > backfills within timeout

Now "we should fix flaky tests" becomes "this one test is burning four engineer-hours a week." That reframing is the whole reason to have the data in a script instead of a screenshot. We've written before about why flaky tests are an economic problem, not a purity problem — the API is where you make that argument with numbers instead of vibes.

Join coverage against the same slice

Coverage on its own is a vanity metric that engineering leaders have learned to distrust, and rightly. Coverage joined to flakiness is a different animal — it tells you where you have lots of tests that don't work reliably, which is worse than having no tests, because it manufactures false confidence.

The coverage endpoint returns per-suite or per-file numbers with a commit reference so you can trend it:

curl -s "$BASE/orgs/$BUILDPULSE_ORG/coverage?group_by=suite&ref=main" \
  -H "Authorization: Bearer $BUILDPULSE_TOKEN" | jq '.data[]'

{
  "suite": "payments",
  "line_coverage": 0.83,
  "branch_coverage": 0.71,
  "commit": "a1b9c3d",
  "measured_at": "2025-01-15T02:10:00Z"
}

Cross that with the flaky list and you get a quadrant nobody wants to see and everybody needs to:

cov = {c["suite"]: c for c in paginate("coverage", {"group_by": "suite", "ref": "main"})}

by_suite = {}
for t in tests:
    s = by_suite.setdefault(t["suite"], {"flaky": 0, "waste": 0})
    s["flaky"] += 1
    s["waste"] += t["weekly_waste_min"]

for suite, agg in by_suite.items():
    c = cov.get(suite, {})
    print(f"{suite:<12} cov={c.get('line_coverage', 0):.0%} "
          f"flaky={agg['flaky']:>2} waste={agg['waste']:>4}min/wk")

High coverage plus high flake waste is your "looks green, isn't" zone. That's where you spend remediation budget first, because those suites are the ones lying to your release process. Low coverage plus low flakiness might just be code that doesn't need many tests. Don't panic-write specs to hit a number — that's how you create the first quadrant.

Wire recent runs into a release gate

Here's where the read API earns its keep for platform teams: gating a release on the current CI signal rather than on whether the last run happened to pass.

The runs endpoint gives you recent builds for a branch, including whether passes were clean or salvaged by a retry.

curl -s "$BASE/orgs/$BUILDPULSE_ORG/runs?branch=main&limit=20" \
  -H "Authorization: Bearer $BUILDPULSE_TOKEN" | jq '.data[0]'

{
  "id": "run_5512",
  "branch": "main",
  "commit": "a1b9c3d",
  "status": "passed",
  "had_retries": true,
  "flaky_failures": 2,
  "real_failures": 0,
  "duration_sec": 431,
  "finished_at": "2025-01-15T02:08:00Z"
}

had_retries and flaky_failures are the fields that matter. A green run that only went green because CI retried it twice is not the same as a green run that passed cold. If your change-management policy treats a passing pipeline as evidence, you want to know how much of that evidence was manufactured by reruns. This is exactly the kind of thing auditors ask about when reruns become part of your control story.

A gate that fails when main has become retry-dependent:

runs = list(paginate("runs", {"branch": "main", "limit": "20"}))
recent = runs[:20]

retry_dependent = sum(1 for r in recent if r["status"] == "passed" and r["had_retries"])
real_failures = sum(1 for r in recent if r["real_failures"] > 0)

if retry_dependent / len(recent) > 0.25:
    print("BLOCK: >25% of recent green runs on main needed retries")
    raise SystemExit(1)

if real_failures > 0:
    print(f"BLOCK: {real_failures} runs had real failures on main")
    raise SystemExit(1)

print("CI signal healthy — release gate open")

Drop that into a release workflow and you've built a gate that reasons about trend, not the last data point. One retry is noise. A quarter of your green builds leaning on retries is a signal that main is quietly rotting, and you want to catch that before you cut a release, not during the postmortem.

# .github/workflows/release-gate.yml
jobs:
  ci-health-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install requests
      - run: python scripts/release_gate.py
        env:
          BUILDPULSE_TOKEN: ${{ secrets.BUILDPULSE_READ_TOKEN }}
          BUILDPULSE_ORG: acme

Alerts that fire on a delta, not a threshold

Static thresholds age badly. "Alert if flake rate > 10%" is fine until the day someone quarantines a batch of tests and your baseline shifts. The alerts worth keeping fire on change — a test that was stable last week and started flaking today, a suite whose coverage dropped after a merge.

Because the API gives you windowed rates, you can diff them cheaply:

regressions = [
    t for t in tests
    if t["flake_rate_7d"] > t["flake_rate_30d"] * 2
    and t["total_runs_30d"] > 100
    and t["status"] != "quarantined"
]

for t in regressions:
    post_to_slack(
        f":warning: *{t['name']}* newly flaky — "
        f"7d {t['flake_rate_7d']:.0%} vs 30d {t['flake_rate_30d']:.0%} "
        f"(owner: {t['owner_team']})"
    )

The run-count guard matters. Without it you'll page the payments team about a test that ran nine times and failed twice, which is statistically meaningless and a great way to get your alerts muted. Alert on regressions that have volume behind them. Route them by owner_team so the notice lands on the people who can act, not in a firehose channel everyone learned to ignore in Q2.

Why build this instead of clicking around

A dashboard answers the questions its designer anticipated. A read API answers the questions you have this week — the ones tied to your services, your teams, your release cadence, and whatever your last incident review turned up.

The three artifacts here — a cost-ranked flaky-test report, a coverage-versus-flakiness join, and a retry-aware release gate — are each maybe forty lines of code. They exist because the data is queryable, versioned, and stable. That's the difference between a metric you present once and a metric that lives in your pipeline and quietly does its job.

If you want the full endpoint reference, our Platform API docs cover pagination, rate limits, and the token scopes for read-only access. Start with the flaky-tests endpoint, get one number your team argues about into a script, and grow from there. The best CI dashboard is the one you didn't have to ask a vendor to build.