GitLab.com hosted runners vs Codemagic
GitLab.com hosted runners run the CI jobs of any GitLab project on fresh Linux, Windows or macOS VMs, for up to 3 hours each. Codemagic is a CI service built for mobile apps, with Mac minis, code signing and store publishing, for builds of up to 120 minutes.
The companies and products
GitLab.com hosted runners
GitLab.com hosted runners are the machines that run GitLab CI/CD jobs for projects on GitLab.com, paid for in compute minutes.
This is a CI service, not a sandbox API. Each job gets a new VM on Google Cloud (Linux and Windows) or on AWS Mac hardware (macOS), with passwordless sudo, and the VM is deleted right after the job. It is built for building, testing and deploying the code in a GitLab project, and GitLab's Duo Agent Platform also runs Claude Code and Codex agents as CI/CD jobs on these runners.
Codemagic
Codemagic is a cloud CI/CD service built for mobile developers: it builds, tests and publishes iOS, Android, Flutter and React Native apps.
You describe a workflow in codemagic.yaml, and each build runs on a new virtual machine: an Apple silicon Mac mini with Xcode, an Ubuntu Linux machine, or a Windows machine. The machine is destroyed when the build finishes, and the result can go straight to App Store Connect, Google Play and other stores. It is built for app teams that want builds and code signing handled for them; it is a different kind of product from an agent sandbox, and the two overlap only where an agent needs a machine for a short, scripted job.
Which use cases each one fits
For each use case, Battleships prices every sandbox provider on the same workload and sorts them from the cheapest monthly bill up. “128th cheapest of 377” means 127 of the other 376 providers run that workload, with every feature it needs, for less. “With limits” means the provider runs it but misses something, so it is listed after every full match. Open a use case for its whole ranking and what each provider misses.
| Use case and its workload | GitLab.com hosted runners: price rank | Codemagic: price rank | boat: price rank |
|---|---|---|---|
| Software factory 100 at once · 4 vCPU / 8 GB · 2,200 sessions of 8 h a month A software factory needs agents that keep their checkout and services between tasks, which neither a 3-hour GitLab job nor a 2-hour Codemagic build gives you. | 347th of 367, with limits $14,454 a month | 362nd of 367, with limits $32,718 a month | 3rd cheapest of 367 $634 a month |
| Always-on AI employees 50 machines 24/7 · 2 vCPU / 4 GB An always-on assistant cannot live in jobs that end within 2 or 3 hours and start from a clean machine each time. | 353rd of 366, with limits $21,858 a month | 364th of 366, with limits $98,550 a month | 5th cheapest of 366 $657 a month |
| Code interpreter bursts 100 at once · 1 vCPU / 2 GB · 200,000 sessions of 1 min a month | 320th of 345, with limits $1,896 a month | 337th of 345, with limits $9,147 a month | 186th of 345, with limits $2,000 a month |
| Sandboxes for your users 300 at once · 2 vCPU / 4 GB · 30,000 sessions of 23 min a month Neither is built to hand machines to your own users: GitLab blocks inbound traffic and Codemagic sells build slots one at a time, while boat's plans start at 100 running sandboxes. | 331st of 361, with limits $6,133 a month | 349th of 361, with limits $28,497 a month | Cheapest of 361 $209 a month |
| RL rollouts 1,000 at once · 2 vCPU / 4 GB · 1,000,000 sessions of 3 min a month | 318th of 360, with limits $29,762 a month | 336th of 360, with limits $135 a month | 34th of 360, with limits $2,000 a month |
| Agent evals 250 at once · 4 vCPU / 8 GB · 15,000 sessions of 20 min a month | 330th of 364, with limits $4,758 a month | 363rd of 364, with limits $10,947 a month | 10th cheapest of 364 $200 a month |
| CI runners 40 at once · 8 vCPU / 16 GB · 6,000 sessions of 12 min a month GitLab's runners are a full fit for CI on a GitLab project, while Codemagic is a compromise outside mobile builds because its smallest Linux machine is 8 vCPU / 32 GB; boat suits CI that wants a warm disk between runs. | 128th cheapest of 377 $2,089 a month | 207th of 377, with limits $3,387 a month | 4th cheapest of 377 $86 a month |
| 24/7 dev boxes 1 machine 24/7 · 4 vCPU / 8 GB Neither keeps a machine overnight: GitLab deletes the VM after each job and Codemagic destroys it after each build, while a boat sandbox keeps its disk when stopped. | 358th of 368, with limits $805 a month | 191st of 368, with limits $333 a month | 18th cheapest of 368 $26 a month |
| Computer-use agents 50 at once · 4 vCPU / 8 GB · 20,000 sessions of 15 min a month GitLab's runners are headless and Codemagic only offers VNC or RDP for a person to debug a build; boat streams a 60fps desktop an agent can drive. | 342nd of 369, with limits $4,278 a month | 71st of 369, with limits $10,047 a month | 6th cheapest of 369 $200 a month |
| Browser-use agents 50 at once · 2 vCPU / 4 GB · 30,000 sessions of 10 min a month | 144th of 367, with limits $2,118 a month | 151st of 367, with limits $10,947 a month | 7th cheapest of 367 $200 a month |
| Scraping & bots (startup scale) 50 at once · 1 vCPU / 2 GB · 100,000 sessions of 3 min a month | 143rd of 346, with limits $2,854 a month | 148th of 346, with limits $13,310 a month | 6th cheapest of 346 $90 a month |
| Mass scraping & bots (undetectable) 1,000 at once · 1 vCPU / 2 GB · 3,000,000 sessions of 2 min a month | 319th of 338, with limits $58,429 a month | 337th of 338, with limits $263 a month | 71st of 338, with limits $1,800 a month |
| Parallel red teaming 200 at once · 2 vCPU / 4 GB · 5,000 sessions of 30 min a month | 87th of 360, with limits $1,383 a month | 95th of 360, with limits $6,672 a month | 2nd cheapest of 360 $200 a month |
| GPU eval jobs 10 at once · 8 vCPU / 32 GB · 500 sessions of 30 min a month GitLab offers a Linux runner with an NVIDIA T4 on Premium and Ultimate; Codemagic and boat have no GPUs. | Can't run it | Can't run it | Can't run it |
| macOS agents 5 at once · 4 vCPU / 8 GB · 1,000 sessions of 20 min a month Both run real macOS on Apple silicon: GitLab's macOS runners are in beta and need Premium or Ultimate, Codemagic is built around Mac minis with Xcode; boat runs Linux only. | 24th of 36, with limits $1,069 a month | 32nd of 36, with limits $1,952 a month | Can't run it |
Key features
| Feature | GitLab.com hosted runners | Codemagic | boat |
|---|---|---|---|
| Kind of product | CI runners for jobs in a GitLab project's .gitlab-ci.yml | Mobile CI/CD: builds start from codemagic.yaml or the API | Sandbox API: create, stop, fork and reach machines from your code |
| How long a machine can run | Up to 3 hours per job | 120 minutes per build on Free, Pay as you go and Fixed price; custom on Enterprise | No session cap on paid plans; you set the time to live |
| Files between runs | VM deleted after each job; shared cache (5 GB per archive) or artifacts carry files over | Machine and disk deleted; per-workflow dependency caches carry over | Disk persists, saved every minute and on stop; a stopped sandbox costs nothing |
| Fork | No; every job starts from a fresh VM | No; every build starts from a stock image | Fork a running sandbox with its disk |
| Docker and nested virtualization | Docker-in-Docker in privileged mode; nested virtualization not documented | Docker and Android emulators on Linux; no nested virtualization on macOS | Docker, root, systemd and /dev/kvm on every machine |
| Operating systems | Linux x64 and Arm64, Windows (beta), macOS on Apple silicon (beta) | macOS on Apple silicon, Linux (x64, arm64 on request) and Windows | Linux x64 only |
| GPUs | A Linux runner with an NVIDIA T4 on Premium and Ultimate | None | None |
| Mobile release tools | Not built in; you script signing and uploads in your pipeline | Code signing and publishing to App Store Connect, TestFlight and Google Play built in | Not built in; you script it yourself |
| Reaching the machine | None; inbound traffic is blocked and instance runners have no web terminal | Temporary SSH, VNC or RDP through a gateway, turned on per build | SSH, HTTPS URLs on up to 50 ports, raw TCP |
| Desktop and computer use | Headless | VNC or RDP for a person to debug a build; no computer-use API | Linux desktop streamed at 60fps, Chromium, computer use API |
| Coding agents | Duo Agent Platform runs Claude Code, Codex and others from issues and merge requests, on Premium and Ultimate | codemagic-init helps agents on your own computer; nothing runs agents on build machines | Seven agents preinstalled, driven from outside with a prompt, steer and events API |
| Keeping API keys out of the machine | GitLab-managed credentials for its Claude and Codex agents only; other secrets are CI/CD variables | Encrypted environment variables delivered into the build machine | Credentials proxy for model providers and any HTTPS API |
| Self-hosting | Open-source GitLab Runner on your own machines, using no compute minutes | No | No |
| Regions | US (Google Cloud us-east1 by default for Linux) | US data centers; dedicated macOS hosts in the EU on request | EU only: Germany, Finland, France |
Every feature Battleships tracks (43)
| Feature | GitLab.com hosted runners | Codemagic | boat |
|---|---|---|---|
| Full VM (own kernel) | Yes | Yes | Yes |
| Docker inside | Yes | Yes | Yes |
| Nested virtualization | No | Yes | Yes |
| Root | Yes | Not published | Yes |
| Systemd | No | No | Yes |
| Desktop GUI | No | Yes | Yes |
| Browser | Yes | Yes | Yes |
| Computer-use API | No | No | Yes |
| SSH | No | Yes | Yes |
| Persistent disk | No | No | Yes |
| Pause/resume | No | No | Yes |
| Automatic snapshots | No | No | Yes |
| Fork/clone | No | No | Yes |
| Live fork (no pause) | No | No | Yes |
| Memory fork | No | No | No |
| Custom image (Docker or snapshot) | Yes | No | Yes |
| ≥24 h sessions | No | No | Yes |
| Idle auto-stop | Yes | Yes | No |
| Wake on request | No | No | No |
| HTTPS preview URLs | No | No | Yes |
| Raw TCP inbound | No | No | Yes |
| Public IPv4 | No | No | No |
| Egress allowlist | No | No | No |
| Secret proxy | Yes | No | Yes |
| Secret proxy for any API | No | No | Yes |
| Preinstalled agents | Yes | No | Yes |
| Agent harness API | Yes | No | Yes |
| MCP server | No | No | No |
| GPU | Yes | No | No |
| Windows | Yes | Yes | No |
| Arm64 | Yes | Yes | No |
| Self-hosting | Yes | No | No |
| BYOC | No | No | No |
| Open source | Yes | No | No |
| Scoped API keys | Not published | Not published | Yes |
| Spend limits | Not published | Not published | Yes |
| Usage per sandbox (API) | No | No | Yes |
| Audit logs | Yes | Yes | No |
| SSO | Yes | Yes | No |
| SOC 2 | Yes | Yes | No |
| HIPAA | No | No | No |
| EU data residency | No | Yes | Yes |
| Zero data retention | Not published | Not published | Yes |
Performance
There is no comparable CPU measurement for GitLab.com hosted runners or for Codemagic in the hpc-sandbox-benchmarks test. boat did 26.69 runs per second on it at 4 vCPU / 8 GB, on dedicated AMD Ryzen 9 9950X hosts. Codemagic also sells only fixed sizes, such as Linux X2 at 8 vCPU / 32 GB, so there is no 4 vCPU / 8 GB machine to put next to boat's.
Node.js build-and-test loop at 4 vCPU / 8 GB, higher is better. Measured with the open hpc-sandbox-benchmarks suite. GitLab.com hosted runners and Codemagic have no published run in this suite yet, so there is no like-for-like number to compare.
Pricing in five scenarios
Each bill is priced from the providers' public price lists by the open Battleships engine, with the cheapest plan that runs the scenario.
| Scenario | GitLab.com hosted runners, a month | Codemagic, a month | boat, a month |
|---|---|---|---|
| One coding agent, workdays 4 vCPU / 8 GB, 50 GB disk, 8 hours a day, 22 days a month, files kept between days | $119 · Premium · with limits | $333 · Free (individual) · with limits | $20 · $20/mo |
| One agent that never stops 4 vCPU / 8 GB, 50 GB disk, running 24/7 all month | $805 · Premium · with limits | $333 · Free (individual) · with limits | $26 · $20/mo |
| 50 agents in parallel 50 machines of 4 vCPU / 8 GB at once, 8-hour sessions, 1,100 sessions a month, Docker inside | $9,491 · Premium · with limits | $21,531 · Pay as you go (team, 4 concurrent) · with limits | $317 · $20/mo × 3 seats |
| Sandboxes for your users 2 vCPU / 4 GB, 30,000 sessions of 10 minutes a month, up to 300 at once | $2,658 · Premium · with limits | $12,297 · Pay as you go (team, 4 concurrent) · with limits | $200 · $100/mo × 2 seats |
| 1,000 short jobs at once 2 vCPU / 4 GB, 1,000 at once, 3 minutes each, 200,000 jobs a month | $5,896 · Premium · with limits | $27,147 · Pay as you go (team, 4 concurrent) · with limits | $2,000 · $2000/mo · with limits |
GitLab.com hosted runners: Each job uses compute minutes equal to its run time times a cost factor, from 1 for small Linux up to 12 for the biggest Linux and macOS machines. Plans include monthly minutes per group (400 on Free, 10,000 on Premium at $29 per user a month, billed yearly), and extra minutes cost $10 per 1,000, bought ahead of time.
Codemagic: Codemagic bills build minutes: $0.095 a minute on a Mac mini M2, $0.114 on an M4 and $0.045 on Linux X2 or Windows, with 500 free M2 minutes a month for individuals. Each extra build at once costs $49 a month, and fixed annual plans start at $3,990 a year.
Codemagic: Plans start at $20 a month, and the whole fee is usage credit spent per second while a sandbox runs. A stopped sandbox costs nothing and keeps its disk.
Limits in those scenarios
Neither GitLab.com hosted runners nor Codemagic fits any of the five agent scenarios, because both are CI services: GitLab jobs stop at 3 hours, Codemagic builds at 120 minutes, and nothing on the disk survives. Codemagic's Pay as you go plan adds one build at a time for $49 a month each, on machines no smaller than 8 vCPU / 32 GB, and GitLab publishes no limit on how many runner VMs run at once, so the 300 and 1,000 machine scenarios are not confirmed for either. On boat only the 1,000-job burst does not fit, because starts above 90 a minute are queued on the $2,000 plan.
| Scenario | GitLab.com hosted runners | Codemagic | boat |
|---|---|---|---|
| One coding agent, workdays |
|
| None that bite |
| One agent that never stops |
|
| None that bite |
| 50 agents in parallel |
|
| None that bite |
| Sandboxes for your users |
|
| None that bite |
| 1,000 short jobs at once |
|
|
|
Which one should you use?
Use GitLab.com hosted runners when
your code lives on GitLab and you want CI for it, especially for Linux Arm64, Windows, big Linux machines, a T4 GPU, or runners you host yourself.
- Native CI for GitLab projects. If your code lives on GitLab, these runners work from the first pipeline, with caches and artifacts built in.
- More kinds of machines. Teams that need Linux Arm64, big Linux machines up to 32 vCPU, Windows or a T4 GPU get a hosted runner for each.
- Your own runners, free of minutes. You can run the open-source GitLab Runner on your own machines, and those jobs use no compute minutes.
Use Codemagic when
you build, sign and publish iOS, Android or Flutter apps and want Mac minis with Xcode for builds that finish within two hours.
- Built for mobile releases. If you ship iOS, Android or Flutter apps, Codemagic signs them and publishes to App Store Connect, TestFlight and Google Play.
- Mac minis with Xcode ready. iOS teams get Mac mini M2 and M4 machines with current Xcode and iOS simulator runtimes preinstalled.
- A way into the build machine. When a build breaks, you can open SSH, VNC or RDP on the machine to look around, which GitLab's hosted runners do not allow.
Where boat fits
Both are CI services: they give a job a clean machine, run it for a few hours at most and throw the machine away. If what you are really running is agents that work for hours, keep their files and serve what they build, look at boat. For GitLab pipelines and for signing and shipping mobile apps, these two remain the right tools.
A machine that stays
On GitLab a job ends after 3 hours and on Codemagic a build ends within 120 minutes, and both wipe the machine, so an agent reinstalls and reclones every time. A boat sandbox has no session cap on paid plans and saves its disk every minute.
Reach it from outside
GitLab blocks inbound traffic to hosted runners, and Codemagic only opens temporary SSH, VNC or RDP by hand per build. boat gives each sandbox SSH, HTTPS URLs on up to 50 ports, raw TCP and a 60fps desktop.
Agents you drive from outside
GitLab starts its agents from issues and merge requests, and Codemagic runs nothing on its machines, so keys arrive as variables the agent can read. boat preinstalls seven agents you drive by API, with real keys kept in a credentials proxy.
Use boat when an agent needs its own Linux machine that keeps its disk, runs past a few hours, takes inbound traffic, shows a desktop and keeps real API keys out of reach. Compare it one to one: boat vs GitLab.com hosted runners and boat vs Codemagic.
Questions people ask
Is GitLab.com hosted runners cheaper than Codemagic?
For general Linux CI, GitLab's runners are cheaper in our tables: $2,089 a month against $3,387 for the same CI runner workload, while boat is $86 a month. For iOS builds the answer depends on your Mac minutes, and on GitLab macOS needs Premium or Ultimate.
Which is better for iOS builds, GitLab or Codemagic?
Codemagic is built for it: Mac minis with Xcode and iOS simulators, plus code signing and App Store Connect publishing built in. GitLab has hosted macOS runners on Apple silicon too, but they are in beta and need Premium or Ultimate.
Which is faster?
Neither GitLab.com hosted runners nor Codemagic has a comparable measurement in the hpc-sandbox-benchmarks CPU test, so we cannot say. boat did 26.69 runs per second there at 4 vCPU / 8 GB.
Can GitLab runners and Codemagic run Docker?
Yes on Linux for both: GitLab runs Docker-in-Docker in privileged mode, and Codemagic Linux machines run Docker and Android emulators. Codemagic's macOS machines cannot run Android emulators, because they have no nested virtualization. boat has Docker and /dev/kvm on every machine.
Can I SSH into a GitLab runner or a Codemagic machine?
Not into a GitLab.com hosted runner: inbound traffic is blocked. Codemagic lets you turn on temporary SSH, VNC or RDP for a build, and access ends 10 minutes after the build. You can SSH into a boat sandbox at any time and open HTTPS URLs on up to 50 ports.
Can I run AI coding agents on either one?
On GitLab Premium and Ultimate, the Duo Agent Platform runs Claude Code and Codex as CI jobs from issues and merge requests, within the 3-hour limit. On Codemagic an agent can only be a build step. On boat agents are preinstalled in a VM that keeps its disk, and you drive them by API.
Try boat
Install the CLI, then run boat new.
7-day free trial
More comparisons
- boat vs GitLab.com hosted runners
- boat vs Codemagic
- GitHub Actions vs GitLab.com hosted runners
- CircleCI vs GitLab.com hosted runners
- Blacksmith vs GitLab.com hosted runners
- Depot vs GitLab.com hosted runners
- WarpBuild vs GitLab.com hosted runners
- Buildkite vs GitLab.com hosted runners
- GitLab.com hosted runners vs Bitrise
- GitLab.com hosted runners vs RunsOn
- GitLab.com hosted runners vs Ubicloud
- GitHub Actions vs Codemagic