GitLab.com hosted runners vs RunsOn
GitLab.com hosted runners run GitLab CI jobs on fresh VMs that GitLab manages, for up to 3 hours each, paid in compute minutes. RunsOn runs GitHub Actions jobs on fresh EC2 machines inside your own AWS account, billed by AWS at cost plus a yearly licence.
The companies and products
GitLab.com hosted runners
GitLab.com hosted runners are the machines that run GitLab CI/CD jobs for projects on GitLab.com, paid for in compute minutes.
This is a CI service, not a sandbox API. Each job gets a new VM on Google Cloud (Linux and Windows) or on AWS Mac hardware (macOS), with passwordless sudo, and the VM is deleted right after the job. It is built for building, testing and deploying the code in a GitLab project, and GitLab's Duo Agent Platform also runs Claude Code and Codex agents as CI/CD jobs on these runners.
RunsOn
RunsOn is self-hosted GitHub Actions runners on AWS: one CloudFormation or Terraform stack in your account, a yearly licence, and compute billed by AWS at cost.
You install RunsOn in your own AWS account, change the runs-on: label in your workflows, and every job then starts on a new EC2 instance that is terminated when the job ends. It covers Linux on x64 and arm64, Windows Server and GPU instances, with caches in S3 and sticky EBS disks. It is a CI runner, a different kind of product from boat, so this page compares the part where they overlap: a machine where an agent or a job does its work.
Which use cases each one fits
For each use case, Battleships prices every sandbox provider on the same workload and sorts them from the cheapest monthly bill up. “128th cheapest of 377” means 127 of the other 376 providers run that workload, with every feature it needs, for less. “With limits” means the provider runs it but misses something, so it is listed after every full match. Open a use case for its whole ranking and what each provider misses.
| Use case and its workload | GitLab.com hosted runners: price rank | RunsOn: price rank | boat: price rank |
|---|---|---|---|
| Software factory 100 at once · 4 vCPU / 8 GB · 2,200 sessions of 8 h a month A software factory needs machines that keep their checkout and caches between tasks; GitLab starts every job on a clean VM, RunsOn keeps only the sticky disk, and boat keeps the whole disk. | 347th of 367, with limits $14,454 a month | 122nd of 367, with limits $1,114 a month | 3rd cheapest of 367 $634 a month |
| Always-on AI employees 50 machines 24/7 · 2 vCPU / 4 GB An assistant that lives on its machine all month does not fit in CI jobs that end after 3 hours on GitLab or 5 days on RunsOn. | 353rd of 366, with limits $21,858 a month | 188th of 366, with limits $1,724 a month | 5th cheapest of 366 $657 a month |
| Code interpreter bursts 100 at once · 1 vCPU / 2 GB · 200,000 sessions of 1 min a month For short bursts RunsOn comes to $147 a month against $1,896 for GitLab.com hosted runners, because AWS bills spot minutes at cost; all three are compromises here. | 320th of 345, with limits $1,896 a month | 229th of 345, with limits $147 a month | 186th of 345, with limits $2,000 a month |
| Sandboxes for your users 300 at once · 2 vCPU / 4 GB · 30,000 sessions of 23 min a month Both start machines only for your own pipelines, not for your users, so handing each user a sandbox means building that layer yourself. | 331st of 361, with limits $6,133 a month | 216th of 361, with limits $585 a month | Cheapest of 361 $209 a month |
| RL rollouts 1,000 at once · 2 vCPU / 4 GB · 1,000,000 sessions of 3 min a month | 318th of 360, with limits $29,762 a month | 245th of 360, with limits $2,714 a month | 34th of 360, with limits $2,000 a month |
| Agent evals 250 at once · 4 vCPU / 8 GB · 15,000 sessions of 20 min a month | 330th of 364, with limits $4,758 a month | 251st of 364, with limits $389 a month | 10th cheapest of 364 $200 a month |
| CI runners 40 at once · 8 vCPU / 16 GB · 6,000 sessions of 12 min a month Both fit as CI runners: GitLab.com hosted runners for GitLab pipelines and RunsOn for GitHub Actions in your AWS account, and boat ranks 4th here with full VMs that keep a warm disk. | 128th cheapest of 377 $2,089 a month | 58th cheapest of 377 $432 a month | 4th cheapest of 377 $86 a month |
| 24/7 dev boxes 1 machine 24/7 · 4 vCPU / 8 GB A dev box needs SSH, open ports and files that stay; GitLab's hosted runners take no inbound traffic, and a RunsOn runner is gone when its job ends. | 358th of 368, with limits $805 a month | 158th of 368, with limits $86 a month | 18th cheapest of 368 $26 a month |
| Computer-use agents 50 at once · 4 vCPU / 8 GB · 20,000 sessions of 15 min a month Both are headless CI machines, while boat streams a 60fps Linux desktop with Chromium and computer use built in. | 342nd of 369, with limits $4,278 a month | 142nd of 369, with limits $353 a month | 6th cheapest of 369 $200 a month |
| Browser-use agents 50 at once · 2 vCPU / 4 GB · 30,000 sessions of 10 min a month | 144th of 367, with limits $2,118 a month | 206th of 367, with limits $224 a month | 7th cheapest of 367 $200 a month |
| Scraping & bots (startup scale) 50 at once · 1 vCPU / 2 GB · 100,000 sessions of 3 min a month | 143rd of 346, with limits $2,854 a month | 228th of 346, with limits $205 a month | 6th cheapest of 346 $90 a month |
| Mass scraping & bots (undetectable) 1,000 at once · 1 vCPU / 2 GB · 3,000,000 sessions of 2 min a month | 319th of 338, with limits $58,429 a month | 122nd of 338, with limits $3,539 a month | 71st of 338, with limits $1,800 a month |
| Parallel red teaming 200 at once · 2 vCPU / 4 GB · 5,000 sessions of 30 min a month | 87th of 360, with limits $1,383 a month | 178th of 360, with limits $157 a month | 2nd cheapest of 360 $200 a month |
| GPU eval jobs 10 at once · 8 vCPU / 32 GB · 500 sessions of 30 min a month | Can't run it | Can't run it | Can't run it |
| macOS agents 5 at once · 4 vCPU / 8 GB · 1,000 sessions of 20 min a month GitLab has hosted macOS runners on Apple silicon in beta for Premium and Ultimate; RunsOn does not support macOS and boat runs Linux only. | 24th of 36, with limits $1,069 a month | Can't run it | Can't run it |
Key features
| Feature | GitLab.com hosted runners | RunsOn | boat |
|---|---|---|---|
| Kind of product | Hosted runners for GitLab CI/CD jobs, paid in compute minutes | Self-hosted GitHub Actions runners, deployed as one stack in your own AWS account | Sandbox API: create, stop, fork and reach full VMs from your code |
| How a machine starts | When a job in a GitLab project's .gitlab-ci.yml pipeline runs | When a GitHub Actions job asks for it with a runs-on: label | Any time, by API, CLI or SDK |
| Where it runs | Managed by GitLab; Linux on Google Cloud us-east1 by default, macOS on AWS Mac hardware | In your own AWS account, in the region you pick; code, secrets and logs stay there | Managed by boat in the EU (Germany, Finland, France) |
| Isolation | New VM per job, deleted right after; Linux jobs run in a privileged Docker container | Fresh EC2 instance per job, terminated when the job ends | Full VM with its own kernel on dedicated AMD Ryzen 9 9950X hosts, with root and systemd |
| How long it can run | Up to 3 hours per job, whatever the project setting | One job; GitHub ends self-hosted jobs after 5 days | No session cap on paid plans; you set the time to live |
| Files between runs | VM deleted after each job; files carry over only through a shared cache (5 GB per archive) or artifacts | Root disk thrown away; one sticky EBS disk per job is snapshotted and restored next job | Whole disk saved every minute and on stop; a stopped sandbox costs nothing |
| Fork | No; every job starts from a fresh VM and a container image | No; every job starts from an AMI | Fork a running sandbox with its disk |
| Docker and nested virtualization | Docker-in-Docker through privileged mode; /dev/kvm not documented | Docker and root; KVM on compatible Intel x64 families, not arm64 | Docker, root, systemd and /dev/kvm on every machine |
| Inbound access | None; inbound traffic is blocked and instance runners have no web terminal | SSH from a CIDR range you allow, and SSM; no HTTPS URLs | SSH, HTTPS URLs for up to 50 ports, raw TCP, dedicated IPv4 or IPv6 |
| Desktop and computer use | Headless; macOS runners do not support jobs that need UI interaction | Headless runners | Linux desktop streamed at 60fps, Chromium, computer use API |
| Coding agents and keys | Duo Agent Platform runs Claude Code, Codex, Amazon Q and Gemini from issues and merge requests on Premium and Ultimate; managed keys for Claude and Codex only | Bedrock access through the instance role; Copilot's coding agent can use a runner | Seven agents preinstalled, driven from outside with a prompt, steer and events API; credentials proxy for any HTTPS API |
| Operating systems and GPUs | Linux x64 and Arm64, Windows (beta), macOS on Apple silicon (beta), Linux with an NVIDIA T4 | Linux x64 and arm64, Windows Server 2022 and 2025, NVIDIA and AMD GPUs up to H100 and H200; no macOS | Linux x64, no GPUs |
| Biggest machine | Linux up to 32 vCPU / 128 GB on Premium and Ultimate | Any EC2 size up to 96 vCPU | Up to 16 vCPU / 32 GB on the $100 plan and above |
| Outbound rules | No egress allowlist or static egress IP | Your own AWS Network Firewall policy or security groups; static egress IPs through NAT | Open internet, no provider-side allowlist |
Every feature Battleships tracks (43)
| Feature | GitLab.com hosted runners | RunsOn | boat |
|---|---|---|---|
| Full VM (own kernel) | Yes | Yes | Yes |
| Docker inside | Yes | Yes | Yes |
| Nested virtualization | No | Yes | Yes |
| Root | Yes | Yes | Yes |
| Systemd | No | Not published | Yes |
| Desktop GUI | No | No | Yes |
| Browser | Yes | No | Yes |
| Computer-use API | No | No | Yes |
| SSH | No | Yes | Yes |
| Persistent disk | No | Yes | Yes |
| Pause/resume | No | Yes | Yes |
| Automatic snapshots | No | Yes | Yes |
| Fork/clone | No | No | Yes |
| Live fork (no pause) | No | No | Yes |
| Memory fork | No | No | No |
| Custom image (Docker or snapshot) | Yes | Yes | Yes |
| ≥24 h sessions | No | Yes | Yes |
| Idle auto-stop | Yes | Yes | No |
| Wake on request | No | Yes | No |
| HTTPS preview URLs | No | No | Yes |
| Raw TCP inbound | No | No | Yes |
| Public IPv4 | No | Yes | No |
| Egress allowlist | No | Yes | No |
| Secret proxy | Yes | No | Yes |
| Secret proxy for any API | No | No | Yes |
| Preinstalled agents | Yes | No | Yes |
| Agent harness API | Yes | No | Yes |
| MCP server | No | No | No |
| GPU | Yes | Yes | No |
| Windows | Yes | Yes | No |
| Arm64 | Yes | Yes | No |
| Self-hosting | Yes | Yes | No |
| BYOC | No | Yes | No |
| Open source | Yes | No | No |
| Scoped API keys | Not published | Not published | Yes |
| Spend limits | Not published | Not published | Yes |
| Usage per sandbox (API) | No | No | Yes |
| Audit logs | Yes | Not published | No |
| SSO | Yes | No | No |
| SOC 2 | Yes | No | No |
| HIPAA | No | No | No |
| EU data residency | No | Yes | Yes |
| Zero data retention | Not published | Not published | Yes |
Performance
There is no comparable measurement for either: neither GitLab.com hosted runners nor RunsOn is in the hpc-sandbox-benchmarks CPU test. boat did 26.69 runs per second on it at 4 vCPU / 8 GB, on dedicated AMD Ryzen 9 9950X hosts. RunsOn's speed depends on the EC2 family you pick.
Node.js build-and-test loop at 4 vCPU / 8 GB, higher is better. Measured with the open hpc-sandbox-benchmarks suite. GitLab.com hosted runners and RunsOn have no published run in this suite yet, so there is no like-for-like number to compare.
Pricing in five scenarios
Each bill is priced from the providers' public price lists by the open Battleships engine, with the cheapest plan that runs the scenario.
| Scenario | GitLab.com hosted runners, a month | RunsOn, a month | boat, a month |
|---|---|---|---|
| One coding agent, workdays 4 vCPU / 8 GB, 50 GB disk, 8 hours a day, 22 days a month, files kept between days | $119 · Premium · with limits | $61 · Starter · with limits | $20 · $20/mo |
| One agent that never stops 4 vCPU / 8 GB, 50 GB disk, running 24/7 all month | $805 · Premium · with limits | $159 · Starter · with limits | $26 · $20/mo |
| 50 agents in parallel 50 machines of 4 vCPU / 8 GB at once, 8-hour sessions, 1,100 sessions a month, Docker inside | $9,491 · Premium · with limits | $1,597 · Starter · with limits | $317 · $20/mo × 3 seats |
| Sandboxes for your users 2 vCPU / 4 GB, 30,000 sessions of 10 minutes a month, up to 300 at once | $2,658 · Premium · with limits | $515 · Starter · with limits | $200 · $100/mo × 2 seats |
| 1,000 short jobs at once 2 vCPU / 4 GB, 1,000 at once, 3 minutes each, 200,000 jobs a month | $5,896 · Premium · with limits | $1,103 · Starter · with limits | $2,000 · $2000/mo · with limits |
GitLab.com hosted runners: Each job uses compute minutes equal to its run time times a cost factor: 1 for small Linux, 2 for medium, up to 12 for the biggest Linux and macOS machines. Plans include monthly minutes per group (400 on Free, 10,000 on Premium at $29 per user a month billed yearly), and extra minutes cost $10 per 1,000, bought ahead of time.
RunsOn: AWS bills the EC2 instances and EBS disks on your own AWS bill, to the second, at spot or on-demand prices with no markup; S3 cache traffic and data transfer are extra. RunsOn charges one licence a year per company, from EUR 300 or $350 for under 50,000 jobs a month, and the prices here use spot rates in us-east-1.
RunsOn: Plans start at $20 a month, and the whole fee is usage credit spent per second on running sandboxes. A stopped sandbox costs nothing and keeps its disk.
Limits in those scenarios
Neither fits any of the five agent scenarios cleanly, because both are CI runners that throw the machine away when the job ends. GitLab cuts every job at 3 hours, keeps nothing on the disk, stops new jobs when a group's minutes run out, and publishes no limit on how many VMs run at once, so the 300 and 1,000 machine scenarios are not confirmed. RunsOn jobs end after GitHub's 5 days, only the sticky disk survives, and its scale limits are your AWS account's quotas, with launches paced at about 2 EC2 calls per second by default. On boat the limit for 1,000 short jobs is starts: the $2,000 plan queues bursts above 90 starts a minute.
| Scenario | GitLab.com hosted runners | RunsOn | boat |
|---|---|---|---|
| One coding agent, workdays |
|
| None that bite |
| One agent that never stops |
|
| None that bite |
| 50 agents in parallel |
|
| None that bite |
| Sandboxes for your users |
|
| None that bite |
| 1,000 short jobs at once |
|
|
|
Which one should you use?
Use GitLab.com hosted runners when
your code lives on GitLab.com and you want CI with nothing to run yourself, especially for macOS builds, or Claude Code and Codex working on your issues and merge requests.
- Native CI for GitLab projects. If your code lives on GitLab.com, these runners work from the first pipeline with no AWS account or extra service to set up.
- macOS builds in the same place. Teams that build for Apple get hosted macOS runners on Apple silicon, in beta for Premium and Ultimate, which RunsOn does not offer.
- Your own runners, free of minutes. You can run the open-source GitLab Runner on your own machines, and those jobs use no compute minutes.
Use RunsOn when
your code lives on GitHub and you want cheaper, bigger GitHub Actions runners in your own AWS account, including arm64, Windows or GPU jobs, billed at AWS cost.
- GitHub Actions at AWS cost. If your code lives on GitHub, change one runs-on: label and jobs run on EC2 billed at cost, with S3 caches and sticky disks.
- Any EC2 size, arm64 and big GPUs. Useful for jobs that need up to 96 vCPU, native arm64, Windows Server or NVIDIA GPUs up to H100 and H200.
- Everything stays in your AWS account. Useful when code, secrets and logs must stay in your own account and region, behind your own network rules.
Where boat fits
Both are good CI runners, and the choice between them mostly follows whether your code lives on GitLab or GitHub. If what you are really running is agents, both make you wrap the work in a CI job on a machine that is thrown away when the job ends. boat gives each agent a full VM you start from your code, keep for as long as the work needs, fork and reach from outside.
A machine that stays
On GitLab every job starts on a clean VM and ends at 3 hours; on RunsOn only the sticky disk comes back and GitHub ends jobs after 5 days. Either way the agent reinstalls packages and reclones on each run.
Reach it from outside
GitLab blocks all inbound traffic to hosted runners, and RunsOn offers only SSH or SSM. A preview, a webhook or a live look at the agent's desktop needs your own tunnel, load balancer or desktop setup.
Agents driven from your code, keys kept out
On both, an agent starts only as a pipeline or workflow job. GitLab keeps keys out only for its managed Claude and Codex agents, and on RunsOn other agents' keys sit as secrets on the machine.
Use boat when you need machines for agents rather than CI jobs: a full VM you start by API, keep with its whole disk, fork, reach over HTTPS and drive with built-in coding agents. Compare it one to one: boat vs GitLab.com hosted runners and boat vs RunsOn.
Questions people ask
Is RunsOn cheaper than GitLab.com hosted runners?
In our CI runners model, RunsOn comes to $432 a month against $2,089 for GitLab.com hosted runners, and boat to $86. A GitLab plan includes monthly minutes, and RunsOn adds a yearly licence on top of the AWS bill.
Which is faster, GitLab.com hosted runners or RunsOn?
Neither has a comparable measurement in the hpc-sandbox-benchmarks CPU test, so we cannot say. RunsOn's speed depends on the EC2 family you choose. boat did 26.69 runs per second at 4 vCPU / 8 GB.
Can I use RunsOn with GitLab?
No. RunsOn starts machines only for GitHub Actions jobs. For a GitLab project, the other way to get your own machines is the open-source GitLab Runner, and jobs on it use no compute minutes.
Can GitLab.com hosted runners and RunsOn run Docker?
Yes. GitLab runs Linux jobs in a privileged Docker container, so Docker-in-Docker works, and RunsOn gives Docker and root, with KVM on compatible Intel x64 machines. boat gives Docker, systemd and /dev/kvm on every machine.
How long can a job run on each?
Up to 3 hours on GitLab.com hosted runners, whatever the project sets. On RunsOn a job can run up to GitHub's 5-day limit for self-hosted runners. A boat sandbox has no session cap on paid plans.
Can I run AI coding agents on them?
Inside CI jobs, yes. GitLab's Duo Agent Platform runs Claude Code, Codex and other agents from issues and merge requests, and RunsOn runners can use Bedrock and host Copilot's coding agent. Both machines are thrown away when the job ends.
Try boat
Install the CLI, then run boat new.
7-day free trial
More comparisons
- boat vs GitLab.com hosted runners
- boat vs RunsOn
- GitHub Actions vs GitLab.com hosted runners
- CircleCI vs GitLab.com hosted runners
- Blacksmith vs GitLab.com hosted runners
- Depot vs GitLab.com hosted runners
- WarpBuild vs GitLab.com hosted runners
- Buildkite vs GitLab.com hosted runners
- GitLab.com hosted runners vs Bitrise
- GitLab.com hosted runners vs Codemagic
- GitLab.com hosted runners vs Ubicloud
- GitHub Actions vs RunsOn