Buildkite vs RunsOn
Buildkite is a whole CI platform: you write Buildkite pipelines, and its hosted agents run each job on a VM it hosts in the US East Coast. RunsOn keeps GitHub Actions as your CI and runs each job on a fresh EC2 machine in your own AWS account, billed by AWS at cost.
The companies and products
Buildkite
Buildkite is a CI and software delivery platform, and hosted agents are the build machines it runs for you.
You write Buildkite pipelines, and each job runs on a fresh virtual machine that Buildkite starts for that job and destroys when it ends; only cache volumes carry files to the next job. Hosted agents come in Linux and macOS on Apple M4, with Windows in private preview, and you can also run Buildkite's agent on your own machines. It is a CI service, not an agent sandbox, so this page compares the part where they overlap: a machine where an agent or a job does its work.
- Where it runs
- Hosted agents run only in US East Coast data centers
- Compliance
- Maintains a SOC 2 Type II report
RunsOn
RunsOn is self-hosted GitHub Actions runners on AWS: one CloudFormation or Terraform stack in your account, a yearly licence, and compute billed by AWS at cost.
You install RunsOn in your own AWS account, change the runs-on: label in your workflows, and every job then starts on a new EC2 instance that is terminated when the job ends. It covers Linux on x64 and arm64, Windows Server and GPU instances, with caches in S3 and sticky EBS disks. It is a CI runner, a different kind of product from boat, so this page compares the part where they overlap: a machine where an agent or a job does its work.
Which use cases each one fits
For each use case, Battleships prices every sandbox provider on the same workload and sorts them from the cheapest monthly bill up. “43rd cheapest of 367” means 42 of the other 366 providers run that workload, with every feature it needs, for less. “With limits” means the provider runs it but misses something, so it is listed after every full match. Open a use case for its whole ranking and what each provider misses.
| Use case and its workload | Buildkite: price rank | RunsOn: price rank | boat: price rank |
|---|---|---|---|
| Software factory 100 at once · 4 vCPU / 8 GB · 2,200 sessions of 8 h a month A coding agent needs its machine to keep its work; Buildkite keeps only cache volume paths and RunsOn only the sticky disk, while boat keeps the whole disk. | 43rd of 367, with limits $11,648 a month | 122nd of 367, with limits $1,114 a month | 3rd cheapest of 367 $634 a month |
| Always-on AI employees 50 machines 24/7 · 2 vCPU / 4 GB An assistant that never stops is a compromise on both: a Buildkite job ends after 8 hours and a RunsOn job after 5 days, while boat has no session cap on paid plans. | 351st of 366, with limits $17,564 a month | 188th of 366, with limits $1,724 a month | 5th cheapest of 366 $657 a month |
| Code interpreter bursts 100 at once · 1 vCPU / 2 GB · 200,000 sessions of 1 min a month | 315th of 345, with limits $1,587 a month | 229th of 345, with limits $147 a month | 186th of 345, with limits $2,000 a month |
| Sandboxes for your users 300 at once · 2 vCPU / 4 GB · 30,000 sessions of 23 min a month Neither CI service starts machines for your users or gives them a URL; boat gives each sandbox HTTPS on up to 50 ports and raw TCP. | 326th of 361, with limits $4,984 a month | 216th of 361, with limits $585 a month | Cheapest of 361 $209 a month |
| RL rollouts 1,000 at once · 2 vCPU / 4 GB · 1,000,000 sessions of 3 min a month | 316th of 360, with limits $23,881 a month | 245th of 360, with limits $2,714 a month | 34th of 360, with limits $2,000 a month |
| Agent evals 250 at once · 4 vCPU / 8 GB · 15,000 sessions of 20 min a month | 324th of 364, with limits $3,884 a month | 251st of 364, with limits $389 a month | 10th cheapest of 364 $200 a month |
| CI runners 40 at once · 8 vCPU / 16 GB · 6,000 sessions of 12 min a month Both are built for CI: Buildkite brings its own pipelines and cache volumes, RunsOn plugs into GitHub Actions with S3 caches and sticky disks, and boat runs CI jobs on full VMs with Docker and /dev/kvm. | 203rd of 377, with limits $2,318 a month | 58th cheapest of 377 $432 a month | 4th cheapest of 377 $86 a month |
| 24/7 dev boxes 1 machine 24/7 · 4 vCPU / 8 GB A dev box has to stay up with its packages installed; both CI services throw the machine away after each job, and a boat machine stays up as long as you want. | 354th of 368, with limits $715 a month | 158th of 368, with limits $86 a month | 18th cheapest of 368 $26 a month |
| Computer-use agents 50 at once · 4 vCPU / 8 GB · 20,000 sessions of 15 min a month Buildkite's desktop access is for a person debugging a macOS job and RunsOn runners are headless; boat streams a 60fps Linux desktop with computer use. | 70th of 369, with limits $3,500 a month | 142nd of 369, with limits $353 a month | 6th cheapest of 369 $200 a month |
| Browser-use agents 50 at once · 2 vCPU / 4 GB · 30,000 sessions of 10 min a month | 142nd of 367, with limits $1,772 a month | 206th of 367, with limits $224 a month | 7th cheapest of 367 $200 a month |
| Scraping & bots (startup scale) 50 at once · 1 vCPU / 2 GB · 100,000 sessions of 3 min a month | 142nd of 346, with limits $2,354 a month | 228th of 346, with limits $205 a month | 6th cheapest of 346 $90 a month |
| Mass scraping & bots (undetectable) 1,000 at once · 1 vCPU / 2 GB · 3,000,000 sessions of 2 min a month | 317th of 338, with limits $46,814 a month | 122nd of 338, with limits $3,539 a month | 71st of 338, with limits $1,800 a month |
| Parallel red teaming 200 at once · 2 vCPU / 4 GB · 5,000 sessions of 30 min a month | 85th of 360, with limits $1,184 a month | 178th of 360, with limits $157 a month | 2nd cheapest of 360 $200 a month |
| GPU eval jobs 10 at once · 8 vCPU / 32 GB · 500 sessions of 30 min a month None of the three fits in our model, though RunsOn can launch NVIDIA GPU instances for CI jobs; Buildkite's hosted shapes and boat are CPU only. | Can't run it | Can't run it | Can't run it |
| macOS agents 5 at once · 4 vCPU / 8 GB · 1,000 sessions of 20 min a month Only Buildkite fits here: it runs macOS on Apple M4 with Xcode, while RunsOn does not support macOS and boat runs Linux only. | 35th of 36, with limits $2,310 a month | Can't run it | Can't run it |
Key features
| Feature | Buildkite | RunsOn | boat |
|---|---|---|---|
| Kind of product | A CI platform with its own pipelines; hosted agents run the jobs | Self-hosted GitHub Actions runners, one stack in your AWS account | Agent sandboxes you create and drive by API, CLI or SDK |
| Where machines run | Buildkite's US East Coast data centers only | Your own AWS account, in the region you pick | Managed by boat in the EU (Germany, Finland, France), with EU data residency |
| Isolation | A VM per job, destroyed with its data after the job | A fresh EC2 instance per job, terminated when the job ends | Full VM with its own kernel on dedicated AMD Ryzen 9 9950X hosts |
| How long a machine runs | Up to 8 hours per job on Linux and 4 on macOS, longer on request | One job; GitHub ends self-hosted jobs after 5 days | No session cap on paid plans; you set the time to live |
| Files after a run | Cache volumes keep chosen paths after successful jobs, best effort, up to 14 days | One sticky EBS disk per job is snapshotted and restored next job | The whole disk is saved every minute and on stop; a stopped sandbox costs nothing |
| Fork | Each job gets its own copy of the last committed cache volume | No; every job starts from an AMI | Fork a running sandbox with its disk |
| Docker and nested virtualization | Docker on Linux; macOS has the Docker CLI but no daemon; /dev/kvm not documented | Docker and root; KVM on compatible Intel x64 families, not arm64 | Docker, root, systemd and /dev/kvm on every machine |
| Operating systems | Linux (ARM64 on Enterprise), macOS on Apple M4 with Xcode, Windows in private preview | Linux x64 and arm64, Windows Server 2022 and 2025; no macOS | Linux only |
| Biggest machine and GPUs | Linux up to 8 vCPU / 32 GB on Pro and 64 vCPU / 256 GB on Enterprise; no GPUs | Any EC2 size up to 96 vCPU, plus NVIDIA or AMD GPU instances | Up to 16 vCPU / 32 GB on the $100 plan and above; CPU only |
| Reaching the machine | No exposed ports; a terminal into a running job through Buildkite's UI or bk job ssh | SSH from a CIDR range you allow, and SSM; no HTTPS URLs | HTTPS URLs for up to 50 ports, token-protected; raw TCP; dedicated IPv4 or IPv6 |
| Desktop and computer use | Browser-based desktop access to debug a running macOS job; no computer-use API | Headless runners | 60fps Linux desktop with Chromium, computer use and browser control |
| Coding agents and keys | Install Claude Code as a pipeline step; an Anthropic proxy sets the API key for the job | Bedrock access through the instance role; Copilot's coding agent can use a runner | Claude Code, Codex and more preinstalled with a prompt, steer and events API; credentials proxy for any HTTPS API |
| Outbound network rules | Static egress IPs; no provider-side allowlist | Your own AWS Network Firewall policy or security groups; static egress IPs through NAT | Open internet, no provider-side allowlist |
Every feature Battleships tracks (43)
| Feature | Buildkite | RunsOn | boat |
|---|---|---|---|
| Full VM (own kernel) | Yes | Yes | Yes |
| Docker inside | Yes | Yes | Yes |
| Nested virtualization | No | Yes | Yes |
| Root | Not published | Yes | Yes |
| Systemd | No | Not published | Yes |
| Desktop GUI | Yes | No | Yes |
| Browser | Yes | No | Yes |
| Computer-use API | No | No | Yes |
| SSH | Yes | Yes | Yes |
| Persistent disk | Yes | Yes | Yes |
| Pause/resume | No | Yes | Yes |
| Automatic snapshots | Yes | Yes | Yes |
| Fork/clone | Yes | No | Yes |
| Live fork (no pause) | No | No | Yes |
| Memory fork | No | No | No |
| Custom image (Docker or snapshot) | Yes | Yes | Yes |
| ≥24 h sessions | No | Yes | Yes |
| Idle auto-stop | Yes | Yes | No |
| Wake on request | No | Yes | No |
| HTTPS preview URLs | No | No | Yes |
| Raw TCP inbound | No | No | Yes |
| Public IPv4 | No | Yes | No |
| Egress allowlist | No | Yes | No |
| Secret proxy | Yes | No | Yes |
| Secret proxy for any API | No | No | Yes |
| Preinstalled agents | No | No | Yes |
| Agent harness API | No | No | Yes |
| MCP server | Yes | No | No |
| GPU | No | Yes | No |
| Windows | Yes | Yes | No |
| Arm64 | Yes | Yes | No |
| Self-hosting | Yes | Yes | No |
| BYOC | No | Yes | No |
| Open source | No | No | No |
| Scoped API keys | Not published | Not published | Yes |
| Spend limits | Not published | Not published | Yes |
| Usage per sandbox (API) | No | No | Yes |
| Audit logs | Not published | Not published | No |
| SSO | Yes | No | No |
| SOC 2 | Yes | No | No |
| HIPAA | No | No | No |
| EU data residency | No | Yes | Yes |
| Zero data retention | Not published | Not published | Yes |
Performance
We have no comparable measurement for Buildkite hosted agents or RunsOn: neither was part of the StarSling hpc-sandbox-benchmarks run. RunsOn's speed also depends on the EC2 family you pick. On that Node.js build-and-test benchmark at 4 vCPU / 8 GB, boat did 26.69 runs per second on dedicated AMD Ryzen 9 9950X hosts.
Node.js build-and-test loop at 4 vCPU / 8 GB, higher is better. Measured with the open hpc-sandbox-benchmarks suite. Buildkite and RunsOn have no published run in this suite yet, so there is no like-for-like number to compare.
Pricing in five scenarios
Each bill is priced from the providers' public price lists by the open Battleships engine, with the cheapest plan that runs the scenario.
| Scenario | Buildkite, a month | RunsOn, a month | boat, a month |
|---|---|---|---|
| One coding agent, workdays 4 vCPU / 8 GB, 50 GB disk, 8 hours a day, 22 days a month, files kept between days | $166 · Pro · with limits | $61 · Starter · with limits | $20 · $20/mo |
| One agent that never stops 4 vCPU / 8 GB, 50 GB disk, running 24/7 all month | $715 · Pro · with limits | $159 · Starter · with limits | $26 · $20/mo |
| 50 agents in parallel 50 machines of 4 vCPU / 8 GB at once, 8-hour sessions, 1,100 sessions a month, Docker inside | $7,677 · Pro · with limits | $1,597 · Starter · with limits | $317 · $20/mo × 3 seats |
| Sandboxes for your users 2 vCPU / 4 GB, 30,000 sessions of 10 minutes a month, up to 300 at once | $2,204 · Pro · with limits | $515 · Starter · with limits | $200 · $100/mo × 2 seats |
| 1,000 short jobs at once 2 vCPU / 4 GB, 1,000 at once, 3 minutes each, 200,000 jobs a month | $4,787 · Pro · with limits | $1,103 · Starter · with limits | $2,000 · $2000/mo · with limits |
Buildkite: Pro costs $30 per active user a month and includes 4,000 Linux vCPU minutes; after that Linux is $0.004 per vCPU minute and macOS $0.02, metered to the second. Enterprise pricing is custom.
RunsOn: AWS bills the EC2 instances and EBS disks on your own AWS bill at spot or on-demand prices with no markup, and RunsOn charges one licence a year per company, from $350 for under 50,000 jobs a month. S3 cache traffic and data transfer are extra.
RunsOn: You pick a monthly plan from $20, and all of it becomes usage credit spent per second on running sandboxes. A stopped sandbox costs nothing and keeps its disk.
Limits in those scenarios
Neither fits the agent scenarios cleanly, because both are CI runners whose machines end with the job: after 8 hours on Buildkite and 5 days on RunsOn, keeping only cache volume paths or the sticky disk. Buildkite Pro also shares one pool of 256 Linux vCPU and 128 concurrent jobs, so 300 user sandboxes or 1,000 short jobs at once mostly wait in the queue. RunsOn's scale limits are your own AWS quotas and launch pacing. On boat the limit for 1,000 short jobs is starts: the $2,000 plan queues bursts and allows fewer starts per day than the scenario needs.
| Scenario | Buildkite | RunsOn | boat |
|---|---|---|---|
| One coding agent, workdays |
|
| None that bite |
| One agent that never stops |
|
| None that bite |
| 50 agents in parallel |
|
| None that bite |
| Sandboxes for your users |
|
| None that bite |
| 1,000 short jobs at once |
|
|
|
Which one should you use?
Use Buildkite when
you want a hosted CI platform with its own pipelines, macOS on Apple M4 for iOS builds, and no cloud account of your own to run.
- macOS on Apple M4. Useful if you build iOS or Mac apps: Buildkite runs macOS with current Xcode on Apple M4, which RunsOn does not offer.
- A whole CI platform, not tied to GitHub Actions. Useful if you want managed CI with its own pipelines, test tooling and cache volumes, mixing hosted and self-hosted agents.
- Nothing to run in your cloud. Useful if you have no AWS account to manage: Buildkite hosts the machines and bills one monthly price per user plus minutes.
Use RunsOn when
your CI is GitHub Actions and you want faster, cheaper runners in your own AWS account, including Windows, arm64 or GPU jobs.
- Keep GitHub Actions, pay AWS at cost. Useful if your CI already lives in GitHub Actions: change the runs-on: label and jobs run on EC2 with no markup.
- Your own AWS account, any region. Useful when code and logs must stay in your AWS account, or jobs must run outside the US East Coast.
- GPUs, Windows Server and bigger machines. Useful for jobs that need NVIDIA GPUs, Windows Server 2022 or 2025, or EC2 sizes up to 96 vCPU.
Where boat fits
Buildkite and RunsOn are both CI services: a machine starts for one job and is thrown away when it ends. If what you are really running is agents that work for hours or days, keep their files, and need to be reached or watched, boat gives each one a full Linux VM you start from code. boat does not sell a CI pipeline or a GitHub runner service, so for plain CI the two are built for the job.
One machine that keeps its state
A Buildkite job stops after 8 hours and a RunsOn job after 5 days, and only cache volume paths or one sticky disk come back. An agent has to reinstall its packages each run; boat saves the whole disk every minute.
Coding agents you drive from outside
On both services an agent is a CI step you start and read through logs, with no way to steer it mid-run. boat ships Claude Code, Codex and more, with a prompt, steer and events API and a credentials proxy for any HTTPS API.
A computer you can reach and see
Buildkite agents expose no ports and RunsOn runners are reached over SSM or SSH, so a preview server or browser task needs your own tunnels. boat gives HTTPS on up to 50 ports, raw TCP and a 60fps desktop.
Use boat when you need machines for agents rather than CI jobs: a full VM you start by API, keep with its whole disk, fork, reach over HTTPS and drive with built-in coding agents. Compare it one to one: boat vs Buildkite and boat vs RunsOn.
Questions people ask
Is Buildkite cheaper than RunsOn?
In the scenarios we priced, RunsOn costs less: one agent on workdays comes to $61 a month on RunsOn and $166 on Buildkite, and 50 agents in parallel to $1,597 against $7,677. RunsOn adds a yearly licence, while Buildkite charges $30 per active user a month on Pro. boat comes to $20 and $317 for the same two.
Which is faster, Buildkite or RunsOn?
We have no comparable CPU measurement for either. RunsOn's speed depends on the EC2 machine you pick, up to 96 vCPU, while Buildkite Pro goes up to 8 vCPU and Enterprise up to 64. boat did 26.69 runs per second at 4 vCPU / 8 GB on the StarSling benchmark.
Can Buildkite or RunsOn run Docker?
Yes, both run Docker on Linux. RunsOn also supports KVM on compatible Intel x64 machines, while Buildkite does not document /dev/kvm and its macOS agents have no Docker daemon. boat has Docker, systemd and /dev/kvm on every machine.
Does RunsOn support macOS like Buildkite?
No. RunsOn covers Linux x64 and arm64, Windows Server and GPU instances, but not macOS. Buildkite runs macOS on Apple M4 with Xcode, and boat runs Linux only.
How long can a job run on Buildkite and RunsOn?
A Buildkite hosted job runs up to 8 hours on Linux and 4 on macOS, longer on request. A RunsOn job runs under GitHub's limit of 5 days for self-hosted jobs. boat has no session cap on paid plans.
Where does my code run?
Buildkite hosted agents run only in US East Coast data centers. RunsOn runs in your own AWS account and region, and your code and logs are never sent to RunsOn. boat runs only in the EU, with EU data residency.
Try boat
Install the CLI, then run boat new.
7-day free trial