MCP for CI: ask your test suite a question from Claude or Cursor

The tab you don't want to leave

Here's the workflow I actually live in: I'm in Cursor, mid-refactor, and CI goes red on payments_test.go. Before, that meant a context switch — open the CI dashboard, find the run, scroll the log, cross-reference the last three runs to figure out if this failure is real or if the test just hates me. Ten minutes of tab-hopping to answer a yes/no question.

What I want is to stay in the editor and ask: "Is this test flaky, or did I actually break something?" And get an answer grounded in the last 200 runs of that test, not the model's imagination.

That's the pitch for MCP for CI. The Model Context Protocol gives your AI assistant a typed, permissioned door into external systems — and "the historical behavior of your test suite" turns out to be an extremely good thing to put behind that door.

What MCP actually is, minus the hype

MCP is an open protocol from Anthropic that standardizes how LLM clients (Claude Desktop, Cursor, Claude Code, Windsurf, whatever ships next quarter) talk to external tools and data. Think of it as LSP but for AI context instead of code intelligence.

A server exposes three kinds of things:

• Tools — functions the model can call (get_flaky_tests, explain_failure).
• Resources — data the model can read (a test's run history, a coverage report).
• Prompts — reusable templates the client can surface.

The client handles the conversation; the server handles the truth. The model doesn't get raw database access — it gets a menu of specific, auditable calls. That last part matters more than it sounds, and we'll come back to it.

Why is CI data a natural fit? Because the questions you ask about CI are lookups against structured history, and the answers are boring facts the model is bad at guessing but great at explaining once you hand them over. "How many times did this test fail in the last week and on which branches" is not a reasoning problem. It's a query. MCP lets the model make the query and then do the part it's actually good at: summarizing and contextualizing.

Wiring up the BuildPulse MCP server

The BuildPulse MCP server exposes your org's test-reliability data — flaky test rankings, per-test failure history, quarantine status, CI cost — as MCP tools. Setup is a config block and a token.

First, grab an API token from your BuildPulse org settings (scoped read-only is fine for querying). Then point your client at the server.

For Claude Desktop or Claude Code, edit claude_desktop_config.json:

{
  "mcpServers": {
    "buildpulse": {
      "command": "npx",
      "args": ["-y", "@buildpulse/mcp-server"],
      "env": {
        "BUILDPULSE_API_TOKEN": "bp_xxxxxxxxxxxx"
      }
    }
  }
}

For Cursor, drop the same shape into .cursor/mcp.json at your project root (so it's per-repo and doesn't leak into unrelated projects):

{
  "mcpServers": {
    "buildpulse": {
      "command": "npx",
      "args": ["-y", "@buildpulse/mcp-server"],
      "env": {
        "BUILDPULSE_API_TOKEN": "${BUILDPULSE_API_TOKEN}"
      }
    }
  }
}

Use the ${VAR} interpolation and keep the actual token in your shell env or a secrets manager. Committing bp_... into .cursor/mcp.json is the kind of thing that shows up in an incident retro.

Restart the client, and you should see the BuildPulse tools register. In Cursor they show up under the MCP settings pane with a green dot; in Claude you'll see the tool icon appear in the composer.

Questions worth asking

Once it's live, the fun starts. A few that earn their keep:

"Why is checkout_flow_test flaky?" The model calls the failure-history tool, pulls the pattern — say, 14 failures in 30 days, all clustered on the main branch between 2pm and 4pm UTC — and hands you back something like: "This test fails predominantly during peak CI load, and the failures correlate with a 30s timeout on an external payment sandbox. Likely a network/timeout flake, not a logic bug." That's a real hypothesis you can act on, sourced from data, not a guess.

"What's my flakiest repo this quarter?" Ranked list, failure rates, retry counts. Great for the Monday planning meeting where someone asks where the CI pain actually lives instead of where it feels like it lives.

"Which tests are quarantined right now, and why?" Because quarantine is only useful if it's visible. A test that's been quarantined for 90 days isn't quarantined — it's deleted with extra steps, and someone should decide to either fix it or delete it honestly.

"Did my last PR make any test flakier?" This one's underrated. Ask it before you merge. If your change bumped the failure rate of three tests in the touched module, that's a signal the model can surface in seconds.

Here's the thing that makes this more than a novelty: the model can chain these. Ask "why is my pipeline slow and flaky," and a decent client will pull the flakiest tests, cross-reference their retry counts against CI minutes burned, and tell you that 60% of your wasted runner time is three tests deep in the auth suite. That's a report you'd normally build by hand.

Don't let the model 'fix' the flake

A warning, because I've watched this go sideways. When you tell an AI assistant a test is flaky and ask it to fix the test, its favorite move is to make the flake invisible — add a retry wrapper, bump a timeout, wrap the assertion in a try/except that swallows the failure. It'll do it cheerfully and call it done.

That's not a fix. That's the model bribing your CI to stop complaining. A retry that hides a real race condition is worse than the flake, because now the signal is gone and the bug is still there — it'll surface in production where there's no retry button.

So scope the question. Use MCP to diagnose — "what's the failure pattern, what's the likely root cause category" — and keep the human in the loop for the fix. The data-grounded diagnosis is where the value is. We wrote more about why retrying isn't remediation in the case against auto-retrying flaky tests, and it applies double when an LLM is the one holding the retry button.

The compliance angle nobody mentions

If you're in a SOC2 or ISO 27001 shop, you might be twitching at the phrase "give the AI access to CI." Good instinct. Let's be precise about what's actually happening.

The BuildPulse MCP server is read-only by default and scoped by token. The model can query test history; it cannot re-run a pipeline, override a gate, or mark a failing check green. Those are exactly the actions your change-management controls exist to protect, and MCP keeps them out of the model's reach.

Every tool call is a discrete, loggable event — not an opaque blob of database access. When your auditor asks "who or what queried the deployment-gating test data and when," you have an answer, not a shrug. That's a materially better posture than handing someone a psql prompt and hoping.

The rule I'd write into your policy: AI can read CI signal, humans act on it. Diagnosis is automatable. Merging a change that touches a compliance-gating pipeline is not. Keep that line bright and MCP-for-CI stays firmly on the right side of your controls.

Why this beats a dashboard

Dashboards answer the questions you knew to ask when you built them. The value of MCP for CI is the long tail — the weird, specific, one-off question you have at 4pm on a Thursday that no dashboard tile anticipated. "Of the tests I touched in this branch, which have failed on macOS runners but not Linux in the last month?" You're not building a chart for that. You're just asking.

And because the answer is grounded in your actual run history through a typed tool call, you get the thing LLMs are usually bad at: a factual, checkable answer instead of a confident hallucination. The model brings the language; your CI data brings the truth.

That's the whole trick. Model context protocol testing workflows aren't magic — they're just putting your reliability data one question away, in the tool where you already work. Wire it up, ask it something real, and stop leaving the editor to find out whether you're chasing a bug or a ghost.