GitHub Actions has taken the devtools world by storm over the past few years - if you’re a software engineer at a startup, there's a high chance you're relying on GitHub Actions to run your CI/CD pipelines. Although convenient at first, as teams and codebases grow, a few shortcomings and become painfully obvious.

The Pitfalls of GitHub-Hosted Runners

High Costs, Modest Returns GitHub Actions seems reasonably priced—until your build minutes skyrocket. Suddenly, you're paying premium prices for a service running on general-purpose hardware. A quick price comparison between the standard GitHub-hosted runner (2 CPU, 8GB memory) at $0.016 per minute vs m7i.large on AWS (2CPU, 8GB memory) is $0.00168 per minute - GitHub actions is almost a 10x premium!

Limited Cache Size GitHub enforces a 10 GB cache size limit. As your project grows, the limited cache capacity can quickly become a bottleneck, increasing build duration.

Limited Customization Need specific tooling or CLI for your workflow? GitHub-hosted runners don’t support pre-baked images, meaning you have to install these tools from scratch on every build - that adds extra minutes spent every time a job runs.

No Analytics GitHub Actions doesn’t have a way of telling you which parts of your pipeline are the most expensive, or take the longest.

So the natural solution would be to turn to GitHub self-hosted runners: GitHub offers an open source agent that connects to GitHub, monitors the job queue, and runs jobs (executes workflow files). You can manage agents individually, or have them dynamically provisioned via GitHub auto-resource controller (arc). This way, you can run your builds in your own cloud environment.

"So let's just host builds ourselves"

Self-hosted runners offer control, cost-savings, and customization:

• Cheaper Compute by running builds on a cloud provider, where compute cost is much cheaper.
  
  

• Control & Flexibility with choices in compute platforms and machines configurations to fit your needs.
  
  

• Security through the ability to lock down your CI infrastructure as you need.

However, great power comes with great responsibility — it's not all sunshine and rainbows. Here's some immediately unobvious challenges.

The Gotcha's

Complexity of Setup and Maintenance Managing your own infrastructure involves significant upfront work. You need to plan everything from IP address management and instance rightsizing, to baking custom images - since GitHub's default runner images don’t contain all the tooling that ships with the Github-hosted runner.

When running jobs, to avoid the cost of a fleet of hot instances, one common solution is autoscaling instances dynamically as jobs are queued. Resource-based autoscaling (cpu/memory saturation) will not work as job concurrency grows, and unless the autoscaler understands job queue ordering, some jobs could be starved - queue early but start later. To keep job times and costs low, a custom scheduler is needed. To top it off, observability is needed to debug any issues - so that infrastructure or vendor must be provisioned and configured.

Maintenance involves everything from making sure images are up to date for security, to right-sizing infrastructure as demands change from the application. If any new tooling is needed, that request must be handled. Because CI stability is paramount to developer productivity, any fires require an infrastructure engineer to act immediately - often at a cost to planned work.

Unforeseen Costs that aren’t immediately obvious:

• Network Costs: Pulling dependencies, docker layers, and using GitHub’s cache or setup actions that involve caching result in additional network costs - which can exceed the cost of compute.
  
  

• Compute Costs: Booting new instances, pulling images or tooling, and instance cooldown periods contribute to idle time your instances are running, but no work is being done. This inflates your bill.
  
  

• Observability Costs: Storage cost of storing logs and metrics, compute cost of aggregation, or vendor costs.
  
  

• Performance Costs: Standard EBS volumes are networked and by default (free tier) offer low IOPS, low bandwidth, and high latency - which could actually increase your build times compared to GitHub-hosted runners. Tuning these variables comes at a cost.

• Security and Compliance Ensuring your self-hosted runners meet security and compliance standards involves constant vigilance. Regular security audits, vulnerability assessments, patch management, and strict access control policies become mandatory responsibilities, demanding considerable time and resources.

These learnings often come after starting implementation, and sunk cost fallacy encourages additional investment or distractions, depending on how you see it. Not to worry, we've taken care of this for you.

BuildPulse Runners: Your builds at half cost, 2x performance

We’ve built optimized infrastructure that runs your GitHub Actions, CircleCI, and SemaphoreCI builds at half cost, 2x performance – with a 1-line change. We do this by leveraging self-hosted runners to run your builds on our infrastructure.

• Integrations: Seamlessly integrates into (without replacing) your existing CI system – GitHub Actions, CircleCI, and SemaphoreCI – with minimal configuration.
  
  

• Onboarding Time: Faster than your build takes to finish.
  
  

• Tuned Infrastructure: Infrastructure that uses resources more efficiently to accelerate your builds and reduce cost.
  
  

• Customizable Images and Machine Configurations: Bake your own images or choose optimized templates tailored specifically for your builds.
  
  

• Toolkit: Building blocks that speed up your pipelines.
  
  

• In-Cluster Caching: Caching at every level – Docker image layers, application dependencies, and artifacts.
  
  

• Fast Job Starts: Fast, optimized instance booting, reducing wait times dramatically.
  
  

• Instant Observability: Built-in Prometheus endpoints providing seamless integration with your existing monitoring tools.
  
  

• Security: We’re SOC 2 Type 2-compliant, and your builds are network- and compute- isolated.
  
  

• Whiteglove Support: Dedicated, proactive assistance ensuring smooth operations.

We're turning devops teams into heroes, book a meeting for a free demo!