CircleCI vs RunsOn
CircleCI is a whole CI and CD platform: you describe pipelines in its config, and its machines in the US run each job, including macOS. RunsOn is not a CI service of its own: it makes your GitHub Actions jobs run on fresh EC2 machines in your own AWS account, billed by AWS at cost plus a yearly licence.
The companies and products
CircleCI
CircleCI is a CI and CD platform that runs your pipeline jobs on machines it hosts, or on runners you host.
You describe jobs in a config file, and each job runs on a fresh Docker container or a machine VM (Linux, Arm, Windows, macOS on Apple M4 Pro, or GPU on Scale), then the machine is discarded. Files move between jobs only through caches, workspaces and artifacts. It is a CI service, not an agent sandbox, so this page compares where they overlap: a machine where a job or an agent does its work. CircleCI also has Chunk, a toolkit that starts its own agent tasks in CircleCI Cloud from a prompt.
RunsOn
RunsOn is self-hosted GitHub Actions runners on AWS: one CloudFormation or Terraform stack in your account, a yearly licence, and compute billed by AWS at cost.
You install RunsOn in your own AWS account, change the runs-on: label in your workflows, and every job then starts on a new EC2 instance that is terminated when the job ends. It covers Linux on x64 and arm64, Windows Server and GPU instances, with caches in S3 and sticky EBS disks. It is a CI runner, a different kind of product from boat, so this page compares the part where they overlap: a machine where an agent or a job does its work.
Which use cases each one fits
For each use case, Battleships prices every sandbox provider on the same workload and sorts them from the cheapest monthly bill up. “136th cheapest of 377” means 135 of the other 376 providers run that workload, with every feature it needs, for less. “With limits” means the provider runs it but misses something, so it is listed after every full match. Open a use case for its whole ranking and what each provider misses.
| Use case and its workload | CircleCI: price rank | RunsOn: price rank | boat: price rank |
|---|---|---|---|
| Software factory 100 at once · 4 vCPU / 8 GB · 2,200 sessions of 8 h a month A coding agent that builds a whole app needs one machine for hours with its files in place; CircleCI stops a job after 3 hours on Performance, RunsOn keeps only the sticky disk between jobs, and boat keeps the whole disk. | 333rd of 367, with limits $8,692 a month | 122nd of 367, with limits $1,114 a month | 3rd cheapest of 367 $634 a month |
| Always-on AI employees 50 machines 24/7 · 2 vCPU / 4 GB An assistant that lives on its machine all month fits neither: CircleCI jobs end after 5 hours even on Scale, GitHub ends RunsOn jobs after 5 days, and boat has no session cap on paid plans. | 340th of 366, with limits $13,137 a month | 188th of 366, with limits $1,724 a month | 5th cheapest of 366 $657 a month |
| Code interpreter bursts 100 at once · 1 vCPU / 2 GB · 200,000 sessions of 1 min a month | 285th of 345, with limits $572 a month | 229th of 345, with limits $147 a month | 186th of 345, with limits $2,000 a month |
| Sandboxes for your users 300 at once · 2 vCPU / 4 GB · 30,000 sessions of 23 min a month Neither starts machines for your users: both run jobs from your own pipelines, while boat gives each sandbox HTTPS on up to 50 ports and raw TCP. | 316th of 361, with limits $3,702 a month | 216th of 361, with limits $585 a month | Cheapest of 361 $209 a month |
| RL rollouts 1,000 at once · 2 vCPU / 4 GB · 1,000,000 sessions of 3 min a month | 310th of 360, with limits $17,882 a month | 245th of 360, with limits $2,714 a month | 34th of 360, with limits $2,000 a month |
| Agent evals 250 at once · 4 vCPU / 8 GB · 15,000 sessions of 20 min a month | 318th of 364, with limits $2,877 a month | 251st of 364, with limits $389 a month | 10th cheapest of 364 $200 a month |
| CI runners 40 at once · 8 vCPU / 16 GB · 6,000 sessions of 12 min a month Both are built for CI: CircleCI brings its own pipelines, caches and test splitting, RunsOn speeds up GitHub Actions on EC2 at AWS cost, and boat runs CI jobs on full VMs with Docker and /dev/kvm but has no pipeline service. | 136th cheapest of 377 $4,302 a month | 58th cheapest of 377 $432 a month | 4th cheapest of 377 $86 a month |
| 24/7 dev boxes 1 machine 24/7 · 4 vCPU / 8 GB A dev box has to stay up and keep what you installed; both CircleCI and RunsOn throw the machine away when the job ends. | 197th of 368, with limits $508 a month | 158th of 368, with limits $86 a month | 18th cheapest of 368 $26 a month |
| Computer-use agents 50 at once · 4 vCPU / 8 GB · 20,000 sessions of 15 min a month Both run headless jobs with no desktop to drive; boat streams a 60fps Linux desktop with Chromium and computer use. | 320th of 369, with limits $2,589 a month | 142nd of 369, with limits $353 a month | 6th cheapest of 369 $200 a month |
| Browser-use agents 50 at once · 2 vCPU / 4 GB · 30,000 sessions of 10 min a month | 116th of 367, with limits $1,293 a month | 206th of 367, with limits $224 a month | 7th cheapest of 367 $200 a month |
| Scraping & bots (startup scale) 50 at once · 1 vCPU / 2 GB · 100,000 sessions of 3 min a month | 70th cheapest of 346 $860 a month | 228th of 346, with limits $205 a month | 6th cheapest of 346 $90 a month |
| Mass scraping & bots (undetectable) 1,000 at once · 1 vCPU / 2 GB · 3,000,000 sessions of 2 min a month | 274th of 338, with limits $17,532 a month | 122nd of 338, with limits $3,539 a month | 71st of 338, with limits $1,800 a month |
| Parallel red teaming 200 at once · 2 vCPU / 4 GB · 5,000 sessions of 30 min a month | 80th of 360, with limits $852 a month | 178th of 360, with limits $157 a month | 2nd cheapest of 360 $200 a month |
| GPU eval jobs 10 at once · 8 vCPU / 32 GB · 500 sessions of 30 min a month Neither is priced here: CircleCI's GPU classes are on its custom-priced Scale plan, RunsOn can launch NVIDIA GPU instances for CI jobs, and boat has no GPUs. | Can't run it | Can't run it | Can't run it |
| macOS agents 5 at once · 4 vCPU / 8 GB · 1,000 sessions of 20 min a month Only CircleCI can do this here: it runs macOS on Apple M4 Pro with Xcode, while RunsOn does not support macOS and boat runs Linux only. | 34th of 36, with limits $2,262 a month | Can't run it | Can't run it |
Key features
| Feature | CircleCI | RunsOn | boat |
|---|---|---|---|
| What it is | A managed CI and CD platform with its own pipelines and config file | Self-hosted GitHub Actions runners: one EC2 machine per job in your AWS account | A sandbox API for agents: full VMs you create, stop, fork and delete from code |
| Where machines run | CircleCI's cloud in the US: AWS us-east-1 and us-east-2, GCP us-east1 and us-central1 | Your own AWS account, in the region you pick | Managed by boat in the EU (Germany, Finland, France), with EU data residency |
| How a machine starts | When a CircleCI pipeline runs a job | When a GitHub Actions job asks for it with a runs-on: label | Any time, by API, CLI or SDK |
| How long a machine runs | 1 hour per job on Free, 3 on Performance, 5 on Scale | One job; GitHub ends self-hosted jobs after 5 days | No session cap on paid plans; you set the time to live |
| Files after a run | The machine is discarded; chosen paths move on through caches, workspaces and artifacts | Root disk is thrown away; one sticky EBS disk per job is snapshotted and restored next job | The whole disk is saved every minute and on stop; a stopped sandbox costs nothing |
| Fork | No fork of a job | No; every job starts from an AMI | Fork a running sandbox with its disk |
| Docker and nested virtualization | Docker and sudo on machine executors; nested virtualization on the Android machine image | Docker and root; KVM on compatible Intel x64 families, not arm64 | Docker, root, systemd and /dev/kvm on every machine |
| Reaching the machine | No exposed ports; rerun a job with SSH, open for 10 minutes after it ends | SSH from a CIDR range you allow, and SSM; no HTTPS URLs | HTTPS URLs for up to 50 ports, token-protected; raw TCP; dedicated IPv4 or IPv6 |
| Desktop and computer use | Headless jobs; browsers in images for tests | Headless runners | 60fps Linux desktop with Chromium, computer use and browser control |
| Coding agents and keys | Chunk starts CircleCI's own agent tasks; secrets reach jobs as environment variables | Bedrock through the instance role; Copilot's coding agent can use a runner | Claude Code, Codex and more preinstalled, with a prompt, steer and events API; a credentials proxy keeps real keys out |
| Operating systems | Linux on x86 and Arm, Windows, and macOS on Apple M4 Pro with Xcode | Linux x64 and arm64, Windows Server 2022 and 2025; no macOS | Linux only (x86) |
| GPUs | Nvidia T4, V100 and other GPU classes on the Scale plan | NVIDIA and AMD GPU instances on any EC2 size | None |
| How you pay | Credits per minute for the machine class, from $15 a month on Performance | AWS bills EC2 at cost; RunsOn charges a licence from $350 a year | Monthly plan from $20, all of it usage credit spent per second |
| Self-hosting | Self-hosted runners, or CircleCI Server on your own Kubernetes | The whole stack runs in your AWS account; code, secrets and logs never go to RunsOn | Managed service only |
Every feature Battleships tracks (43)
| Feature | CircleCI | RunsOn | boat |
|---|---|---|---|
| Full VM (own kernel) | Yes | Yes | Yes |
| Docker inside | Yes | Yes | Yes |
| Nested virtualization | Yes | Yes | Yes |
| Root | Not published | Yes | Yes |
| Systemd | No | Not published | Yes |
| Desktop GUI | No | No | Yes |
| Browser | Yes | No | Yes |
| Computer-use API | No | No | Yes |
| SSH | Yes | Yes | Yes |
| Persistent disk | No | Yes | Yes |
| Pause/resume | No | Yes | Yes |
| Automatic snapshots | No | Yes | Yes |
| Fork/clone | No | No | Yes |
| Live fork (no pause) | No | No | Yes |
| Memory fork | No | No | No |
| Custom image (Docker or snapshot) | Yes | Yes | Yes |
| ≥24 h sessions | No | Yes | Yes |
| Idle auto-stop | Yes | Yes | No |
| Wake on request | No | Yes | No |
| HTTPS preview URLs | No | No | Yes |
| Raw TCP inbound | No | No | Yes |
| Public IPv4 | No | Yes | No |
| Egress allowlist | No | Yes | No |
| Secret proxy | No | No | Yes |
| Secret proxy for any API | No | No | Yes |
| Preinstalled agents | No | No | Yes |
| Agent harness API | Yes | No | Yes |
| MCP server | Yes | No | No |
| GPU | Yes | Yes | No |
| Windows | Yes | Yes | No |
| Arm64 | Yes | Yes | No |
| Self-hosting | Yes | Yes | No |
| BYOC | No | Yes | No |
| Open source | No | No | No |
| Scoped API keys | Not published | Not published | Yes |
| Spend limits | Not published | Not published | Yes |
| Usage per sandbox (API) | Yes | No | Yes |
| Audit logs | Yes | Not published | No |
| SSO | Yes | No | No |
| SOC 2 | Yes | No | No |
| HIPAA | No | No | No |
| EU data residency | No | Yes | Yes |
| Zero data retention | Not published | Not published | Yes |
Performance
We have no comparable measurement for CircleCI or RunsOn: neither was in the StarSling hpc-sandbox-benchmarks run. RunsOn's speed depends on the EC2 family you pick. On that Node.js build-and-test benchmark at 4 vCPU / 8 GB, boat did 26.69 runs per second on dedicated AMD Ryzen 9 9950X (Zen 5) hosts.
Node.js build-and-test loop at 4 vCPU / 8 GB, higher is better. Measured with the open hpc-sandbox-benchmarks suite. CircleCI and RunsOn have no published run in this suite yet, so there is no like-for-like number to compare.
Pricing in five scenarios
Each bill is priced from the providers' public price lists by the open Battleships engine, with the cheapest plan that runs the scenario.
| Scenario | CircleCI, a month | RunsOn, a month | boat, a month |
|---|---|---|---|
| One coding agent, workdays 4 vCPU / 8 GB, 50 GB disk, 8 hours a day, 22 days a month, files kept between days | $96 · Performance · with limits | $61 · Starter · with limits | $20 · $20/mo |
| One agent that never stops 4 vCPU / 8 GB, 50 GB disk, running 24/7 all month | $508 · Performance · with limits | $159 · Starter · with limits | $26 · $20/mo |
| 50 agents in parallel 50 machines of 4 vCPU / 8 GB at once, 8-hour sessions, 1,100 sessions a month, Docker inside | $5,714 · Performance · with limits | $1,597 · Starter · with limits | $317 · $20/mo × 3 seats |
| Sandboxes for your users 2 vCPU / 4 GB, 30,000 sessions of 10 minutes a month, up to 300 at once | $1,617 · Performance · with limits | $515 · Starter · with limits | $200 · $100/mo × 2 seats |
| 1,000 short jobs at once 2 vCPU / 4 GB, 1,000 at once, 3 minutes each, 200,000 jobs a month | $3,562 · Performance · with limits | $1,103 · Starter · with limits | $2,000 · $2000/mo · with limits |
CircleCI: CircleCI bills credits per minute for the machine class a job uses: Performance starts at $15 a month with 30,000 credits, and more credits cost $15 per 25,000. A 4 vCPU / 15 GB Linux machine uses 20 credits a minute, about $0.72 an hour, and Scale is custom and billed annually.
RunsOn: AWS bills the EC2 machines and EBS disks on your own AWS bill at spot or on-demand prices with no markup; S3 cache traffic and data transfer are extra. RunsOn charges one licence a year per company, from $350 for under 50,000 jobs a month.
RunsOn: You pick a monthly plan from $20, and all of it becomes usage credit spent per second on running sandboxes. A stopped sandbox costs nothing and keeps its disk.
Limits in those scenarios
Neither CircleCI nor RunsOn fits any of the five agent scenarios cleanly, because both are CI runners whose machine ends with the job. CircleCI's job stops after 3 hours on Performance, and Performance runs at most 80 jobs at once, so 300 user sandboxes and 1,000 short jobs wait in the queue. RunsOn keeps only the sticky disk between jobs, GitHub ends its jobs after 5 days, and its egress price is not published. On boat the 1,000-job burst is queued and needs more starts per day than the $2,000 plan allows.
| Scenario | CircleCI | RunsOn | boat |
|---|---|---|---|
| One coding agent, workdays |
|
| None that bite |
| One agent that never stops |
|
| None that bite |
| 50 agents in parallel |
|
| None that bite |
| Sandboxes for your users |
|
| None that bite |
| 1,000 short jobs at once |
|
|
|
Which one should you use?
Use CircleCI when
you want a managed CI platform with its own pipelines, macOS on Apple M4 Pro for iOS builds, Windows or Arm machines, and no AWS account to run.
- A whole CI platform. Useful if you want managed CI with its own pipelines, caches, test splitting and orbs, without running any machines yourself.
- macOS on Apple M4 Pro. Useful if you build iOS or Mac apps: CircleCI runs macOS with Xcode, and RunsOn does not support macOS.
- No AWS account to run. Useful if you do not want to manage cloud quotas and stacks: CircleCI hosts the machines and bills one credit pool.
Use RunsOn when
you already use GitHub Actions and want cheaper, bigger runners in your own AWS account and region, including Windows, arm64 or GPU jobs, billed at AWS cost.
- GitHub Actions at AWS cost. Useful if you already use GitHub Actions: change one label and jobs run on EC2 spot or on-demand machines with no markup.
- Your own AWS account and region. Useful when code and logs must stay in your AWS account, or must run in a region CircleCI's US cloud does not cover.
- Any EC2 size and GPUs. Useful for big builds: any EC2 size up to 96 vCPU, arm64, Windows Server and NVIDIA or AMD GPUs.
Where boat fits
CircleCI and RunsOn are both CI services: a machine lives for one job, then it is gone. If what you are really running is agents that work for hours or days, keep their files, and serve something you can open, look at boat instead. boat also runs CI jobs, but it has no pipeline service.
A machine that keeps its state
On CircleCI an agent's job stops after 3 hours on Performance, and on RunsOn after one GitHub job of at most 5 days; both then discard the machine. boat has no session cap on paid plans and saves the whole disk every minute.
Reach it and see it
CircleCI jobs expose no ports and RunsOn runners are reached over SSH or SSM, so a preview needs your own tunnels. boat gives HTTPS URLs on up to 50 ports, raw TCP and a 60fps desktop for computer use.
Coding agents you drive from outside
On both, an agent is a job step you install and read through logs, with keys as secrets on the machine. boat preinstalls Claude Code and Codex with a prompt, steer and events API, and a credentials proxy keeps real keys out.
Use boat when what you are running is agents rather than CI jobs: a full VM you start by API, keep for days with its whole disk, fork, reach over HTTPS and drive with built-in coding agents. Compare it one to one: boat vs CircleCI and boat vs RunsOn.
Questions people ask
Is RunsOn cheaper than CircleCI?
In every scenario we priced, yes: one coding agent on workdays comes to $61 a month on RunsOn and $96 on CircleCI, and 50 agents in parallel to $1,597 against $5,714. For CI runners the use-case table has RunsOn at $432 a month and CircleCI at $4,302. RunsOn's figures use AWS spot rates in us-east-1 plus its yearly licence.
Which is faster, CircleCI or RunsOn?
We have no comparable measurement for either one. RunsOn's speed depends on the EC2 family you choose, and CircleCI's on its machine class. boat did 26.69 runs per second on the StarSling Node.js benchmark at 4 vCPU / 8 GB.
Does RunsOn work with CircleCI pipelines?
No. RunsOn only starts machines for GitHub Actions jobs, through the runs-on: label in your workflows. CircleCI has its own self-hosted runners if you want CircleCI pipelines on your own machines.
Can CircleCI or RunsOn run macOS?
CircleCI can: it runs macOS jobs on Apple M4 Pro with Xcode, even on the Free plan one job at a time. RunsOn does not support macOS, and boat runs Linux only.
How long can a job run on CircleCI and RunsOn?
CircleCI stops a job after 1 hour on Free, 3 on Performance and 5 on Scale. RunsOn sets no limit of its own, but GitHub ends self-hosted jobs after 5 days. boat has no session cap on paid plans.
Can I run AI agents on CircleCI or RunsOn?
Only as a CI job. CircleCI's Chunk starts its own agent tasks, and RunsOn runners can use Bedrock or host Copilot's coding agent, but the machine is gone when the job ends. boat gives agents a VM they keep, with Claude Code and Codex preinstalled.
Try boat
Install the CLI, then run boat new.
7-day free trial