We moved 13 CI repos to our own runners in one afternoon. Here is everything that broke.

Yesterday we migrated all thirteen of our public example repositories, one per test framework, from GitHub-hosted runners to BuildPulse runners. We expected to learn about our own gaps. We did. But that is not mostly what we found.

Final score: 13 of 13 green, running and uploading test results on BuildPulse runners. Getting there took three rounds of fixes, and here is the honest ledger of every failure along the way, because the breakdown surprised us:

Repo	What broke	Whose fault
jest, pytest, junit, rspec, cypress, elixir	nothing, green on first run	nobody
phpunit	actions/cache@v2, hard-disabled by GitHub platform-wide in 2024	time
mocha	old setup-node calling the shut-down cache API	time
go	pinned to Go 1.16 (2021), which cannot parse modern go.mod files	time
testng	resolved dependencies from JCenter, a repository that no longer exists, and pinned TestNG 7.2.0, a version never published to Maven Central	time
ruby-minitest	EOL Ruby 2.7 pin with a Bundler 2.2 lockfile	time
phpunit (round 2)	needed composer, not on our runner image	us
scalatest	needed sbt, not on our runner image	us
selenium	Selenium 3-era test code, plus Chrome not on our image	both

For ten of the thirteen repos, the entire migration was this diff:

 jobs:
   build:
-    runs-on: ubuntu-latest
+    runs-on: bp-ubuntu-latest-x64-2x

The other three needed one more line each, and we will get to why.

Two findings worth your time whether or not you ever use our runners.

1. Your CI is probably rotting, and green checkmarks are hiding it

Nine of our thirteen repos had not run CI meaningfully in months. In that time, GitHub deprecated actions out from under them, JCenter died, Ruby 2.7 went EOL, and a pinned Go version quietly became unable to install modern tools. None of this was visible, because nothing was forcing the workflows to run.

A runner migration is not really a migration. It is the first full CI health check most repos have had in years. Almost everything that "broke when we switched runners" was already broken, waiting for any change at all to surface it.

2. The "minimal image" problem is real, and the fix already exists

GitHub-hosted runners ship roughly 40 GB of preinstalled toolchain. Our image ships a tiny fraction of that, on purpose, because it is most of why cold starts are fast. The fear is that every migration dies on a missing tool.

Measured across thirteen frameworks: the missing tools were composer, sbt, and Chrome. Every one was fixed with the standard setup action the ecosystem already uses, the same way you would pin a Node or Python version. Here is the scalatest repo's entire fix:

       - name: Set up JDK 17
         uses: actions/setup-java@v2
         with:
           java-version: "17"
           distribution: "temurin"

+      - uses: sbt/setup-sbt@v1
+
       - name: Run tests
         run: sbt test

Same shape for the other two: shivammathur/setup-php for composer, browser-actions/setup-chrome for Chrome. If your workflow already uses setup actions instead of assuming preinstalled tools, it is portable across every runner vendor, including back to GitHub.

The one real rewrite

One repo needed actual code changes: our Selenium example, written against Selenium 3 in 2020, driving Chrome through a chromedriver-autoinstaller and xvfb dance that Selenium 4 made obsolete. Twenty lines of test code later it runs headless Chrome and passed in 2m06s (the run).

Why we did this

Every one of these repos now runs its tests on BuildPulse runners and uploads results to BuildPulse flaky-test detection on every push. When we say the product is built on itself, the receipts are public: click any badge in any of the repos below and read the run.

The repos: jest · pytest · junit · go · rspec · mocha · cypress · elixir · phpunit · ruby-minitest · scalatest · selenium · testng.

Numbers we observed and you can check: first job pickup on a freshly provisioned tenant was 16 seconds; typical pickup after that ranged 15 to 55 seconds (example: 53s on a later run).

FAQ

Do I have to change my workflows? For most repos the migration is the one-line runs-on swap above. If your workflow assumes a preinstalled tool instead of using a setup action, you add the setup action, which also makes the workflow portable back to GitHub-hosted runners.

What if my job needs a tool that is not on the image? Use the ecosystem's setup action for it, exactly as you would to pin a version. Across thirteen frameworks we hit three missing tools, and each was one line.

Can I switch back? Yes. Change runs-on back. Nothing else in the workflow is BuildPulse-specific except the test upload step, which runs anywhere.

How fast is pickup really? First job on a freshly provisioned tenant: 16 seconds, linked above. Steady state we observed 15 to 55 seconds. Those are our public repos; click the runs and check.

Runners that know whether the failure is your code or a flaky test. Try BuildPulse