Codemagic vs RunsOn
Codemagic is a hosted CI/CD service for mobile apps, with real Macs, Xcode and store publishing built in. RunsOn puts your GitHub Actions jobs on fresh EC2 machines inside your own AWS account, with Linux, Windows, arm64 and GPUs but no macOS.
The companies and products
Codemagic
Codemagic is a cloud CI/CD service built for mobile developers: it builds, tests and publishes iOS, Android, Flutter and React Native apps.
You describe a workflow in codemagic.yaml, and each build runs on a new virtual machine: an Apple silicon Mac mini with Xcode, an Ubuntu Linux machine, or a Windows machine. The machine is destroyed when the build finishes, and the result can go straight to App Store Connect, Google Play and other stores. It is built for app teams that want builds and code signing handled for them; it is a different kind of product from an agent sandbox, and the two overlap only where an agent needs a machine for a short, scripted job.
RunsOn
RunsOn is self-hosted GitHub Actions runners on AWS: one CloudFormation or Terraform stack in your account, a yearly licence, and compute billed by AWS at cost.
You install RunsOn in your own AWS account, change the runs-on: label in your workflows, and every job then starts on a new EC2 instance that is terminated when the job ends. It covers Linux on x64 and arm64, Windows Server and GPU instances, with caches in S3 and sticky EBS disks. It is a CI runner, a different kind of product from boat, so this page compares the part where they overlap: a machine where an agent or a job does its work.
Which use cases each one fits
For each use case, Battleships prices every sandbox provider on the same workload and sorts them from the cheapest monthly bill up. “362nd cheapest of 367” means 361 of the other 366 providers run that workload, with every feature it needs, for less. “With limits” means the provider runs it but misses something, so it is listed after every full match. Open a use case for its whole ranking and what each provider misses.
| Use case and its workload | Codemagic: price rank | RunsOn: price rank | boat: price rank |
|---|---|---|---|
| Software factory 100 at once · 4 vCPU / 8 GB · 2,200 sessions of 8 h a month A software factory needs machines that keep their work between steps; Codemagic keeps only dependency caches and RunsOn only the sticky disk, while boat keeps the whole disk. | 362nd of 367, with limits $32,718 a month | 122nd of 367, with limits $1,114 a month | 3rd cheapest of 367 $634 a month |
| Always-on AI employees 50 machines 24/7 · 2 vCPU / 4 GB An assistant that never stops does not suit either: Codemagic builds end within 120 minutes and GitHub ends RunsOn jobs after 5 days, while boat has no session cap on paid plans. | 364th of 366, with limits $98,550 a month | 188th of 366, with limits $1,724 a month | 5th cheapest of 366 $657 a month |
| Code interpreter bursts 100 at once · 1 vCPU / 2 GB · 200,000 sessions of 1 min a month RunsOn costs less than Codemagic for short bursts because AWS bills spot minutes at cost, though each burst still has to arrive as a GitHub Actions job. | 337th of 345, with limits $9,147 a month | 229th of 345, with limits $147 a month | 186th of 345, with limits $2,000 a month |
| Sandboxes for your users 300 at once · 2 vCPU / 4 GB · 30,000 sessions of 23 min a month Both start machines for your own builds, not for your users, so giving each user a sandbox means building that layer yourself; boat's plans start at 100 running sandboxes. | 349th of 361, with limits $28,497 a month | 216th of 361, with limits $585 a month | Cheapest of 361 $209 a month |
| RL rollouts 1,000 at once · 2 vCPU / 4 GB · 1,000,000 sessions of 3 min a month | 336th of 360, with limits $135 a month | 245th of 360, with limits $2,714 a month | 34th of 360, with limits $2,000 a month |
| Agent evals 250 at once · 4 vCPU / 8 GB · 15,000 sessions of 20 min a month | 363rd of 364, with limits $10,947 a month | 251st of 364, with limits $389 a month | 10th cheapest of 364 $200 a month |
| CI runners 40 at once · 8 vCPU / 16 GB · 6,000 sessions of 12 min a month RunsOn is the closer fit for general CI, since it runs any GitHub Actions job on EC2 at AWS cost, while Codemagic's smallest Linux machine is 8 vCPU / 32 GB; boat runs CI jobs on full VMs that keep a warm disk. | 207th of 377, with limits $3,387 a month | 58th cheapest of 377 $432 a month | 4th cheapest of 377 $86 a month |
| 24/7 dev boxes 1 machine 24/7 · 4 vCPU / 8 GB A dev box has to survive the night; Codemagic deletes the machine after each build and a RunsOn runner is gone when its job ends, while a boat machine keeps its disk when stopped. | 191st of 368, with limits $333 a month | 158th of 368, with limits $86 a month | 18th cheapest of 368 $26 a month |
| Computer-use agents 50 at once · 4 vCPU / 8 GB · 20,000 sessions of 15 min a month Codemagic lets a person open VNC or RDP on a build machine and RunsOn runners are headless, so neither has a computer-use API; boat streams a 60fps desktop an agent can drive. | 71st of 369, with limits $10,047 a month | 142nd of 369, with limits $353 a month | 6th cheapest of 369 $200 a month |
| Browser-use agents 50 at once · 2 vCPU / 4 GB · 30,000 sessions of 10 min a month | 151st of 367, with limits $10,947 a month | 206th of 367, with limits $224 a month | 7th cheapest of 367 $200 a month |
| Scraping & bots (startup scale) 50 at once · 1 vCPU / 2 GB · 100,000 sessions of 3 min a month | 148th of 346, with limits $13,310 a month | 228th of 346, with limits $205 a month | 6th cheapest of 346 $90 a month |
| Mass scraping & bots (undetectable) 1,000 at once · 1 vCPU / 2 GB · 3,000,000 sessions of 2 min a month | 337th of 338, with limits $263 a month | 122nd of 338, with limits $3,539 a month | 71st of 338, with limits $1,800 a month |
| Parallel red teaming 200 at once · 2 vCPU / 4 GB · 5,000 sessions of 30 min a month | 95th of 360, with limits $6,672 a month | 178th of 360, with limits $157 a month | 2nd cheapest of 360 $200 a month |
| GPU eval jobs 10 at once · 8 vCPU / 32 GB · 500 sessions of 30 min a month | Can't run it | Can't run it | Can't run it |
| macOS agents 5 at once · 4 vCPU / 8 GB · 1,000 sessions of 20 min a month Only Codemagic fits here: it runs macOS on Apple silicon Mac minis with Xcode, while RunsOn has no macOS and boat runs Linux only. | 32nd of 36, with limits $1,952 a month | Can't run it | Can't run it |
Key features
| Feature | Codemagic | RunsOn | boat |
|---|---|---|---|
| What it is | Mobile CI/CD service: builds start from codemagic.yaml or the API | Self-hosted GitHub Actions runners: one EC2 machine per CI job | A sandbox API for agents: full VMs you create, stop, fork and delete from code |
| Where it runs | Codemagic's US data centers; dedicated macOS hosts in the EU on request | In your own AWS account, in the region you pick | Managed by boat in the EU (Germany, Finland, France) |
| How a machine starts | When a build is started from codemagic.yaml or the API | When a GitHub Actions job asks for it with a runs-on: label | Any time, by API, CLI or SDK |
| How long it can run | 120 minutes per build on Free, Pay as you go and Fixed price; custom on Enterprise | One job; GitHub ends self-hosted jobs after 5 days | No session cap on paid plans; you set the time to live |
| Files after it stops | Machine and disk are deleted; only per-workflow dependency caches carry over | Root disk is thrown away; one sticky EBS disk per job is snapshotted and restored next job | Whole disk saved every minute and on stop; a stopped sandbox costs nothing |
| Fork | No; every build starts from a stock image | No; every job starts from an AMI | Fork a running sandbox with its disk |
| Operating systems | macOS on Apple silicon, Linux (x64, arm64 on request) and Windows | Linux x64 and arm64, Windows Server 2022 and 2025; no macOS | Linux only (x86-64) |
| Xcode and app store publishing | Xcode and iOS simulators preinstalled; publishes to App Store Connect, TestFlight and Google Play | No macOS machines, so no Xcode; publishing is your own workflow steps | None; you script publishing yourself |
| Machine sizes and GPUs | Fixed sizes: Mac mini M2 (8-core / 12 GB), M4 (10-core / 16 GB), Linux X2 (8 vCPU / 32 GB) | Any EC2 size up to 96 vCPU, plus NVIDIA or AMD GPU instances | Linux x64 up to 16 vCPU / 32 GB on the $100 plan and above; CPU only |
| Docker and nested virtualization | Docker and Android emulators on Linux; no nested virtualization on macOS | Docker and root; KVM on compatible Intel x64 families, not arm64 | Docker, root, systemd and /dev/kvm on every machine |
| Reaching the machine | Temporary SSH, VNC or RDP through a gateway, turned on per build | SSH from a CIDR range you allow, and SSM; no HTTPS URLs | HTTPS URLs on up to 50 ports, raw TCP, SSH |
| Desktop and computer use | VNC or RDP for a person to debug a build; no computer-use API | Headless runners | 60fps Linux desktop and Chromium, with computer use and browser control |
| Coding agents and keys | codemagic-init helps agents on your own computer; keys reach builds as encrypted environment variables | Bedrock access through the instance role; Copilot's coding agent can use a runner | Claude Code, Codex and more preinstalled with a prompt, steer and events API; credentials proxy keeps real keys out |
| How you pay | Per build minute by machine type, plus a monthly fee for each extra build running at once | AWS bills EC2 and EBS at cost on your bill, plus a yearly RunsOn licence | Monthly plan from $20, all of it usage credit spent per second |
Every feature Battleships tracks (43)
| Feature | Codemagic | RunsOn | boat |
|---|---|---|---|
| Full VM (own kernel) | Yes | Yes | Yes |
| Docker inside | Yes | Yes | Yes |
| Nested virtualization | Yes | Yes | Yes |
| Root | Not published | Yes | Yes |
| Systemd | No | Not published | Yes |
| Desktop GUI | Yes | No | Yes |
| Browser | Yes | No | Yes |
| Computer-use API | No | No | Yes |
| SSH | Yes | Yes | Yes |
| Persistent disk | No | Yes | Yes |
| Pause/resume | No | Yes | Yes |
| Automatic snapshots | No | Yes | Yes |
| Fork/clone | No | No | Yes |
| Live fork (no pause) | No | No | Yes |
| Memory fork | No | No | No |
| Custom image (Docker or snapshot) | No | Yes | Yes |
| ≥24 h sessions | No | Yes | Yes |
| Idle auto-stop | Yes | Yes | No |
| Wake on request | No | Yes | No |
| HTTPS preview URLs | No | No | Yes |
| Raw TCP inbound | No | No | Yes |
| Public IPv4 | No | Yes | No |
| Egress allowlist | No | Yes | No |
| Secret proxy | No | No | Yes |
| Secret proxy for any API | No | No | Yes |
| Preinstalled agents | No | No | Yes |
| Agent harness API | No | No | Yes |
| MCP server | No | No | No |
| GPU | No | Yes | No |
| Windows | Yes | Yes | No |
| Arm64 | Yes | Yes | No |
| Self-hosting | No | Yes | No |
| BYOC | No | Yes | No |
| Open source | No | No | No |
| Scoped API keys | Not published | Not published | Yes |
| Spend limits | Not published | Not published | Yes |
| Usage per sandbox (API) | No | No | Yes |
| Audit logs | Yes | Not published | No |
| SSO | Yes | No | No |
| SOC 2 | Yes | No | No |
| HIPAA | No | No | No |
| EU data residency | Yes | Yes | Yes |
| Zero data retention | Not published | Not published | Yes |
Performance
There is no comparable measurement for Codemagic or RunsOn: neither was in the hpc-sandbox-benchmarks run where boat did 26.69 Node.js build-and-test runs per second at 4 vCPU / 8 GB. Codemagic sells fixed machine sizes with no 4 vCPU / 8 GB option, and RunsOn's speed depends on the EC2 family you pick.
Node.js build-and-test loop at 4 vCPU / 8 GB, higher is better. Measured with the open hpc-sandbox-benchmarks suite. Codemagic and RunsOn have no published run in this suite yet, so there is no like-for-like number to compare.
Pricing in five scenarios
Each bill is priced from the providers' public price lists by the open Battleships engine, with the cheapest plan that runs the scenario.
| Scenario | Codemagic, a month | RunsOn, a month | boat, a month |
|---|---|---|---|
| One coding agent, workdays 4 vCPU / 8 GB, 50 GB disk, 8 hours a day, 22 days a month, files kept between days | $333 · Free (individual) · with limits | $61 · Starter · with limits | $20 · $20/mo |
| One agent that never stops 4 vCPU / 8 GB, 50 GB disk, running 24/7 all month | $333 · Free (individual) · with limits | $159 · Starter · with limits | $26 · $20/mo |
| 50 agents in parallel 50 machines of 4 vCPU / 8 GB at once, 8-hour sessions, 1,100 sessions a month, Docker inside | $21,531 · Pay as you go (team, 4 concurrent) · with limits | $1,597 · Starter · with limits | $317 · $20/mo × 3 seats |
| Sandboxes for your users 2 vCPU / 4 GB, 30,000 sessions of 10 minutes a month, up to 300 at once | $12,297 · Pay as you go (team, 4 concurrent) · with limits | $515 · Starter · with limits | $200 · $100/mo × 2 seats |
| 1,000 short jobs at once 2 vCPU / 4 GB, 1,000 at once, 3 minutes each, 200,000 jobs a month | $27,147 · Pay as you go (team, 4 concurrent) · with limits | $1,103 · Starter · with limits | $2,000 · $2000/mo · with limits |
Codemagic: Codemagic bills build minutes: $0.095 a minute on a Mac mini M2, $0.114 on an M4, and $0.045 on Linux X2 or Windows, with 500 free M2 minutes a month for individuals. Each extra build running at once costs $49 a month, and fixed annual plans start at $3,990 a year.
RunsOn: AWS bills the EC2 instances and EBS disks on your own AWS bill, to the second, at spot or on-demand prices with no markup; S3 cache traffic and data transfer are extra. RunsOn adds one licence a year per company, from EUR 300 or $350 for under 50,000 jobs a month.
RunsOn: You pick a monthly plan from $20, and all of it becomes usage credit spent per second on running sandboxes. A stopped sandbox costs nothing and keeps its disk.
Limits in those scenarios
Neither Codemagic nor RunsOn fits any of the five agent scenarios, because both are CI runners whose machines end with the build or job. Codemagic stops every build within 120 minutes, deletes its files, and starts with one build at a time, so 50 agents or 300 user sandboxes mean buying dozens of 8 vCPU / 32 GB build slots. RunsOn keeps only the sticky disk between jobs, GitHub ends its jobs after 5 days, and AWS egress is not priced here. On boat the one scenario that does not fit is the 1,000-job burst, which needs more starts per minute and per day than the $2,000 plan allows, so it gets queued.
| Scenario | Codemagic | RunsOn | boat |
|---|---|---|---|
| One coding agent, workdays |
|
| None that bite |
| One agent that never stops |
|
| None that bite |
| 50 agents in parallel |
|
| None that bite |
| Sandboxes for your users |
|
| None that bite |
| 1,000 short jobs at once |
|
|
|
Which one should you use?
Use Codemagic when
you build, sign and publish iOS, Android or Flutter apps and need hosted Macs with Xcode for builds that finish within two hours.
- Real Macs with Xcode. Useful if you build iOS or macOS apps: Codemagic runs Apple silicon Mac minis with Xcode and iOS simulators ready.
- A full mobile release pipeline. Useful if you ship mobile apps: Codemagic signs builds and publishes to App Store Connect, TestFlight and Google Play.
- Nothing to run yourself. Useful if you have no AWS account to manage: Codemagic hosts the build machines, so there is no stack to install.
Use RunsOn when
you want cheaper, bigger GitHub Actions runners in your own AWS account, including Windows, arm64 or GPU jobs, billed at AWS cost.
- GitHub Actions at AWS cost. Useful if you live in GitHub Actions: change one runs-on: label and jobs run on EC2 billed at cost, with no markup.
- Windows, arm64, GPUs and big machines. Useful for jobs that need Windows Server, native arm64, NVIDIA GPUs or any EC2 size up to 96 vCPU.
- Your own AWS account, any region. Useful when code, secrets and logs must stay in your AWS account, behind your own network rules.
Where boat fits
Codemagic and RunsOn are both CI services: a machine appears for one build or job and is gone when it ends. If what you are really running is agents, not builds, boat gives each one a full Linux VM you start from code, keep as long as the work needs, fork, and reach over HTTPS. boat is not a CI service, so for iOS builds or GitHub runners on your own AWS account, the two of them are built for that job.
A machine that stays
An agent that installs tools and builds for hours loses its machine on Codemagic after 120 minutes, and on RunsOn when the job ends, keeping only the sticky disk. A boat sandbox has no session cap on paid plans and saves its whole disk every minute.
Ports and a desktop you can reach
Codemagic machines sit on a private network with only temporary SSH, VNC or RDP, and RunsOn runners are reached over SSM or SSH, so a preview needs your own tunnels or load balancer. boat gives HTTPS URLs on 50 ports, raw TCP and a 60fps desktop.
Agents driven from outside, keys kept out
On both, an agent runs as a build step with its keys on the machine, and you wait for the build to end. On boat coding agents are preinstalled, you prompt and steer them by API, and the credentials proxy keeps real keys out of the sandbox.
Use boat when you need machines for agents rather than CI builds: a full Linux VM you start by API, keep with its whole disk, fork, reach over HTTPS and drive with built-in coding agents. Compare it one to one: boat vs Codemagic and boat vs RunsOn.
Questions people ask
Is Codemagic cheaper than RunsOn?
For general CI in our model, no: CI runners come to $3,387 a month on Codemagic against $432 on RunsOn, because RunsOn bills AWS spot minutes at cost. Codemagic's per-minute prices cover macOS builds, which RunsOn cannot run at all. boat comes to $86 a month for the same CI model.
Which is faster, Codemagic or RunsOn?
There is no comparable measurement for either. Neither was in the benchmark run where boat did 26.69 runs per second at 4 vCPU / 8 GB, and RunsOn's speed depends on the EC2 family you choose.
Can RunsOn build iOS apps?
No. RunsOn supports Linux x64 and arm64, Windows Server and GPU instances, but not macOS, so Xcode builds need a Mac service such as Codemagic. boat runs Linux only, so it cannot build iOS apps either.
Can Codemagic and RunsOn run Docker?
Yes on Linux for both. Codemagic runs Docker and Android emulators on Linux machines, but its macOS machines have no nested virtualization. RunsOn gives Docker and root, with KVM on compatible Intel x64 families but not arm64. boat has Docker, systemd and /dev/kvm on every machine.
Can I run AI coding agents on Codemagic or RunsOn?
Only as a step inside a build or job. Codemagic stops a build within 120 minutes and deletes the machine, and GitHub ends a RunsOn job after 5 days. On boat the agent is preinstalled in a VM that keeps its disk, and you prompt and steer it by API.
Where does my code run?
Codemagic runs builds in its US data centers, with dedicated macOS hosts in the EU on request. RunsOn runs in your own AWS account, in the region you pick. boat runs only in the EU, in Germany, Finland and France.
Try boat
Install the CLI, then run boat new.
7-day free trial