Self-hosted GitHub Actions runners vs GitHub-hosted: where queue time actually goes

The 22-minute check with six minutes of work

Last month I watched a platform lead screen-share a pull request whose required check took 21 minutes and 40 seconds, wall clock. The test suite inside it ran for 6 minutes and 38 seconds. The other 15 minutes were queue wait, VM boot, a multi-gigabyte image pull, actions/setup-node, a Docker layer pull, and an actions/cache restore crawling across the network. Nobody on the team had ever itemized this, because the GitHub UI reports "duration" starting from the moment a runner picks the job up. The slowest part of their CI was invisible by design.

That's the trap in almost every self-hosted runners vs GitHub-hosted debate I've sat through. Teams argue about per-minute compute price — which is real, and we'll do that math — while the thing engineers actually feel, and the thing that changes merge behavior, is queue time. So let's take queue time apart, measure it properly, and then compare architectures with numbers instead of vibes.

What "queue time" actually contains

When an engineer says "CI is slow," they mean push-to-verdict. That interval hides four different buckets, and they're owned by different systems:

• Scheduler wait. The job exists but no runner has claimed it. On GitHub-hosted, you're waiting on GitHub's shared pool. On self-hosted, you're waiting on your own capacity.
• Provisioning. A VM boots or a pod schedules, the runner image gets pulled, the runner registers with GitHub. On hosted runners this is amortized into GitHub's pool management. On scale-to-zero self-hosted setups, it lands squarely on your job.
• Environment setup. setup-node, setup-go, docker pull, apt-get install. This is billed and reported as run time, but from the developer's chair it's pure waiting — none of it is their code.
• Cache restore. actions/cache pulling a 2 GB node_modules tarball over the network, versus finding it already sitting on local disk.

GitHub's API gives you bucket one (job created_at to started_at), sometimes with bucket two smeared into it. Buckets three and four get counted as "the build," which is exactly how a six-minute suite becomes a twenty-one-minute check without anyone noticing.

Measure it before you argue about it

Before you migrate anything, get your actual queue distribution. Job-level created_at is set when the job becomes eligible to run (after its needs are satisfied), so started_at - created_at isolates queue plus provisioning cleanly:

#!/usr/bin/env bash
# p50/p90/p99 job queue time across recent completed runs
REPO="your-org/your-repo"

gh api "repos/$REPO/actions/runs?status=completed&per_page=50" \
  --jq '.workflow_runs[].id' | while read -r run_id; do
  gh api "repos/$REPO/actions/runs/$run_id/jobs?per_page=100" \
    --jq '.jobs[] | select(.started_at != null) |
      ((.started_at | fromdateiso8601) - (.created_at | fromdateiso8601))'
done | sort -n | awk '{a[NR]=$1} END {
  printf "n=%d  p50=%ds  p90=%ds  p99=%ds\n",
    NR, a[int(NR*.5)], a[int(NR*.9)], a[int(NR*.99)]
}'

Two things to look at once you have this. First, the p90 and p99, not the median — queue pain is bursty and correlated (everyone pushes Monday morning, everyone's matrix fans out at once). Second, run a separate pass summing the durations of setup-ish steps (Set up job, anything starting with Set up, cache restores). That's your bucket-three-and-four number, and on hosted runners with network caches it's routinely 90–150 seconds per job. Multiply by your daily job count and sit with that for a minute.

The numbers

These are representative figures from migrations we've run and from operating our own fleet — mid-size orgs, 30–80 concurrent jobs at peak, Node and Go monorepos with Docker builds. Your distribution will differ; the shape won't.

Setup	Queue p50	Queue p95	Setup overhead (typical job)
GitHub-hosted standard (ubuntu-latest)	10–40s	2–6 min at peak	60–150s
GitHub-hosted larger runners (8-core)	30–90s	3–8 min	60–150s
Self-hosted, scale-to-zero (ARC or EC2 ASG)	90–240s	5–12 min	20–120s
Self-hosted, warm per-tenant pool	under 5s	15–60s	5–30s

Three observations worth calling out.

Larger hosted runners often queue worse than standard ones. Each job gets a dedicated VM booted for it, and the capacity pool is per-org rather than GitHub-global. You paid for faster hardware and bought slower pickup. I've seen teams upgrade to 16-core runners, watch wall-clock time barely move, and conclude their tests were the problem. The tests were fine. The queue ate the speedup.

Naive scale-to-zero self-hosting is the worst of both worlds at p50. More on that below.

The warm-pool setup wins twice. Queue collapses to seconds, and setup overhead collapses too, because dependency caches and Docker layers are already on local disk. That second effect is frequently bigger than the first.

Cost math, including the humans

Now the part everyone Googles: GitHub Actions runner cost. Hosted Linux is $0.008/min for 2 cores and scales linearly — $0.032/min for 8 cores, call it ~$1.92/hour. A c6i.2xlarge (8 vCPU) is about $0.34/hour on-demand and half that on spot. Raw compute is 5–10x cheaper self-hosted. That's the slide every migration deck leads with.

It's also incomplete. Honest self-hosted accounting includes idle warm capacity, EBS and network egress, and — the big one — platform engineering time: patching runner images, upgrading ARC, debugging the incident where a webhook secret rotation silently stalled the fleet at 9 a.m. My rule of thumb: below roughly $3–5k/month of hosted spend, self-managing runners loses money once you price the humans running them. Above that, the compute arbitrage is real.

But the line item that never makes the spreadsheet is engineer wait. Forty engineers, six CI-triggering pushes a day, an extra two minutes of p50 queue each: eight person-hours per day of your most expensive payroll watching a yellow dot. And that undercounts, because a wait over about two minutes isn't a wait — it's a context switch, and the engineer is now reading Slack.

The autoscaling trap

Scale-to-zero doesn't eliminate cold-start cost. It reassigns it from your AWS bill to your engineers' queue time, which is a worse ledger to put it on.

Walk the cold path for actions-runner-controller: webhook fires, listener requests a runner, pod is created, no node has room, Karpenter or the ASG provisions one (60–120s), the node pulls a multi-gigabyte runner image, the runner registers, and only then does the job start. Two to five minutes — and it lands precisely during bursts, when a fan-out matrix outruns your warm capacity. Your p50 looks fine in the dashboard; your p95 is why people are complaining.

The fix is unglamorous: a warm floor.

apiVersion: actions.github.com/v1alpha1
kind: AutoscalingRunnerSet
metadata:
  name: linux-8core
spec:
  githubConfigUrl: https://github.com/your-org
  minRunners: 4    # warm floor ~= business-hours p75 concurrency
  maxRunners: 40
  template:
    spec:
      containers:
        - name: runner
          image: ghcr.io/actions/actions-runner:latest

Size the floor from data, not feel: pull your concurrent-jobs distribution during business hours, set minRunners around p75–p90, and schedule it down to near zero on nights and weekends. The cost of being wrong is small. Four warm 8-core spot instances for ten business hours is about six dollars a day. One engineer waiting four minutes for a cold node costs more than that, and they'll do it a dozen times before lunch.

The case for warm-pooled per-tenant runners

Two words in that phrase are doing all the work: warm and per-tenant.

Warm is the performance story. Pickup in seconds, and — because the machine ran your jobs an hour ago — Docker layers, toolchains, and dependency caches are already on local NVMe. A node_modules restore that took 90 seconds through actions/cache over the network becomes a four-second disk read. This is why warm self-hosted runners routinely halve wall-clock time even when the CPU is no faster than GitHub's.

Per-tenant is the risk story, and if you're a SOC 2 or ISO 27001 shop it's not optional garnish. Dedicated instances mean no shared cache tenancy, no "your job ran on a VM another org just used," and a clean isolation answer when an auditor asks how your CI control environment is bounded. When a required status check is part of your change-management evidence, "ephemeral VM from a shared pool, provenance unknown" is a harder conversation than "dedicated warm fleet, this account, this VPC." Auditors don't hate speed; they hate ambiguity.

Full disclosure: this architecture is what we run. BuildPulse runners keep a warm, per-tenant pool sized to your concurrency, which is where the "2x faster at half the cost" numbers come from — not exotic CPUs, just deleting queue and setup time while charging closer to raw compute prices. If you'd rather build it yourself with ARC and a warm floor, genuinely, the sections above are the playbook. The architecture wins either way.

Queue time is a CI-trust problem wearing a cost costume

Here's why a flaky-test company cares this much about queue time. Slow queues change engineering behavior in ways that quietly corrupt your CI signal. Engineers batch commits to avoid round-trips, so diffs get bigger and riskier. They merge on a stale green rather than re-validate. And they stop clicking re-run — which means retry-based flake detection gets starved of data, and flaky tests get force-merged past instead of identified and quarantined. Cheap, fast reruns are what make flakiness data honest. A 4-minute queue makes every rerun a small act of penance, and people stop doing penance fast.

So, the short version to take into your next infra review:

• Measure job-level queue percentiles this week — the script above takes five minutes.
• Itemize setup and cache-restore time separately. It's probably bigger than you think.
• If you're self-hosted with scale-to-zero, add a scheduled warm floor before you buy bigger instances.
• Compare costs per developer-visible minute saved, not per compute minute — and include idle capacity, ops time, and the humans in the queue.

The cheapest runner minute is the one nobody spent waiting for.