Blacksmith vs GitLab.com hosted runners
Blacksmith runs GitHub Actions jobs on its own machines, and you switch by changing one runner label. GitLab.com hosted runners run the pipelines of GitLab projects, each job on a fresh VM that ends within 3 hours.
The companies and products
Blacksmith
Blacksmith is a CI service: it runs your GitHub Actions jobs on its own faster, cheaper runners instead of GitHub's.
You change the runner label in your workflow file, and each job runs in a fresh Firecracker microVM that boots in under 3 seconds and is destroyed when the job ends. It adds Docker layer caching and sticky disks that carry chosen folders from one run to the next, plus a Testbox CLI in early beta that lets a local coding agent run commands inside a CI job. It is built for teams that want quicker, cheaper GitHub Actions, not as a general sandbox you drive by API; a separate Sandboxes product is marked coming soon.
GitLab.com hosted runners
GitLab.com hosted runners are the machines that run GitLab CI/CD jobs for projects on GitLab.com, paid for in compute minutes.
This is a CI service, not a sandbox API. Each job gets a new VM on Google Cloud (Linux and Windows) or on AWS Mac hardware (macOS), with passwordless sudo, and the VM is deleted right after the job. It is built for building, testing and deploying the code in a GitLab project, and GitLab's Duo Agent Platform also runs Claude Code and Codex agents as CI/CD jobs on these runners.
Which use cases each one fits
For each use case, Battleships prices every sandbox provider on the same workload and sorts them from the cheapest monthly bill up. “78th cheapest of 377” means 77 of the other 376 providers run that workload, with every feature it needs, for less. “With limits” means the provider runs it but misses something, so it is listed after every full match. Open a use case for its whole ranking and what each provider misses.
| Use case and its workload | Blacksmith: price rank | GitLab.com hosted runners: price rank | boat: price rank |
|---|---|---|---|
| Software factory 100 at once · 4 vCPU / 8 GB · 2,200 sessions of 8 h a month Agents that build and run whole apps need a kept disk, hosting and forks, which boat has; Blacksmith and GitLab only run the CI part of that loop. | 248th of 367, with limits $3,607 a month | 347th of 367, with limits $14,454 a month | 3rd cheapest of 367 $634 a month |
| Always-on AI employees 50 machines 24/7 · 2 vCPU / 4 GB An assistant that lives on its machine all month does not fit either one, because each job starts clean and GitLab cuts jobs off at 3 hours. | 78th of 366, with limits $5,463 a month | 353rd of 366, with limits $21,858 a month | 5th cheapest of 366 $657 a month |
| Code interpreter bursts 100 at once · 1 vCPU / 2 GB · 200,000 sessions of 1 min a month | 281st of 345, with limits $488 a month | 320th of 345, with limits $1,896 a month | 186th of 345, with limits $2,000 a month |
| Sandboxes for your users 300 at once · 2 vCPU / 4 GB · 30,000 sessions of 23 min a month Handing sandboxes to your own users means starting machines from your backend, which boat does by API and both CI services do only through pipelines. | 268th of 361, with limits $1,563 a month | 331st of 361, with limits $6,133 a month | Cheapest of 361 $209 a month |
| RL rollouts 1,000 at once · 2 vCPU / 4 GB · 1,000,000 sessions of 3 min a month | 276th of 360, with limits $7,488 a month | 318th of 360, with limits $29,762 a month | 34th of 360, with limits $2,000 a month |
| Agent evals 250 at once · 4 vCPU / 8 GB · 15,000 sessions of 20 min a month | 354th of 364, with limits $1,188 a month | 330th of 364, with limits $4,758 a month | 10th cheapest of 364 $200 a month |
| CI runners 40 at once · 8 vCPU / 16 GB · 6,000 sessions of 12 min a month Both fit; pick by where your code lives, since Blacksmith runs GitHub Actions jobs and GitLab's runners run GitLab pipelines, while boat suits CI you drive by API. | 78th cheapest of 377 $708 a month | 128th cheapest of 377 $2,089 a month | 4th cheapest of 377 $86 a month |
| 24/7 dev boxes 1 machine 24/7 · 4 vCPU / 8 GB A dev box has to keep its files and stay reachable, but both services delete the machine when the job ends. | 180th of 368, with limits $207 a month | 358th of 368, with limits $805 a month | 18th cheapest of 368 $26 a month |
| Computer-use agents 50 at once · 4 vCPU / 8 GB · 20,000 sessions of 15 min a month Computer-use agents need a screen; both CI services run headless jobs, while boat streams a 60fps Linux desktop. | 243rd of 369, with limits $1,088 a month | 342nd of 369, with limits $4,278 a month | 6th cheapest of 369 $200 a month |
| Browser-use agents 50 at once · 2 vCPU / 4 GB · 30,000 sessions of 10 min a month | 131st of 367, with limits $588 a month | 144th of 367, with limits $2,118 a month | 7th cheapest of 367 $200 a month |
| Scraping & bots (startup scale) 50 at once · 1 vCPU / 2 GB · 100,000 sessions of 3 min a month | 130th of 346, with limits $719 a month | 143rd of 346, with limits $2,854 a month | 6th cheapest of 346 $90 a month |
| Mass scraping & bots (undetectable) 1,000 at once · 1 vCPU / 2 GB · 3,000,000 sessions of 2 min a month | 262nd of 338, with limits $14,613 a month | 319th of 338, with limits $58,429 a month | 71st of 338, with limits $1,800 a month |
| Parallel red teaming 200 at once · 2 vCPU / 4 GB · 5,000 sessions of 30 min a month | 64th of 360, with limits $350 a month | 87th of 360, with limits $1,383 a month | 2nd cheapest of 360 $200 a month |
| GPU eval jobs 10 at once · 8 vCPU / 32 GB · 500 sessions of 30 min a month | Can't run it | Can't run it | Can't run it |
| macOS agents 5 at once · 4 vCPU / 8 GB · 1,000 sessions of 20 min a month Both have macOS runners on Apple silicon, GitLab's in beta and only on Premium or Ultimate; boat runs Linux only. | 28th of 36, with limits $1,508 a month | 24th of 36, with limits $1,069 a month | Can't run it |
Key features
| Feature | Blacksmith | GitLab.com hosted runners | boat |
|---|---|---|---|
| What it is | GitHub Actions runners; each machine lives for one workflow job | GitLab CI/CD runners for GitLab.com projects, paid in compute minutes | A sandbox API: create, stop, fork and reach full VMs from your code |
| Isolation | Firecracker microVM per job with its own kernel | New VM per job on Google Compute Engine; Linux jobs run in a privileged Docker container | Full VM with its own kernel on dedicated AMD Ryzen 9 9950X hosts, with root and systemd |
| How a machine starts | A GitHub Actions job, after you change the runner label in your workflow | A job in a GitLab project's .gitlab-ci.yml pipeline | One API call, SDK or CLI, from anywhere |
| How long it lives | Until the job ends, then the microVM is destroyed | Up to 3 hours per job, then the VM is deleted | Until you stop it; no session cap on paid plans |
| What carries over between runs | Sticky disk folders, up to 5 per job, evicted after 7 days unused | A shared cache (5 GB per archive, removed after 14 days unused) or artifacts | The whole disk, saved every minute and on stop; you can also fork a running sandbox |
| Nested virtualization | Yes on x64 Linux runners, not on ARM | Not documented; privileged mode is described for Docker-in-Docker only | Yes, /dev/kvm for Android emulators and VMs |
| Running many at once | Docs say Blacksmith imposes no concurrency limits | No published limit; when a group's minutes run out, new jobs stop | Starts above 90 a minute are queued, up to 600 an hour on the $2,000 plan |
| Inbound access | Opt-in debug SSH, only for the user who triggered the job; no ports or URLs | None; inbound traffic is blocked and instance runners have no web terminal | SSH, HTTPS URLs for up to 50 ports, raw TCP, dedicated IP |
| Desktop and computer use | Headless runners | Headless; macOS runners do not support jobs that need UI interaction | 60fps Linux desktop with Chromium and computer use |
| Coding agents | Its own [code]smith agent from Slack, GitHub or a web UI; Testbox CLI in early beta for local agents | Duo Agent Platform runs Claude Code, Codex, Amazon Q and Gemini from issues and merge requests, on Premium and Ultimate | Seven agents preinstalled, driven from outside with a prompt, steer and events API |
| Keeping API keys out of the machine | GitHub Actions secrets and OIDC tokens available inside the job | GitLab-managed credentials for its Claude and Codex agents only; other secrets enter as CI/CD variables | Credentials proxy for model providers and any HTTPS API; switch a credential to another account when credits run out |
| Operating systems and GPUs | Ubuntu x64 and ARM, Windows Server 2025 (beta), macOS on Apple Silicon; no GPUs | Linux x64 and Arm64, Windows (beta), macOS on Apple silicon (beta), and a Linux runner with an NVIDIA T4 | Linux x64, no GPUs |
| Regions | US east, US west, Amsterdam and Frankfurt | US (Google Cloud us-east1 by default for Linux) | EU only (Germany, Finland, France) |
| Self-hosting | No | Open-source GitLab Runner on your own machines, with no compute minutes used | No |
Every feature Battleships tracks (43)
| Feature | Blacksmith | GitLab.com hosted runners | boat |
|---|---|---|---|
| Full VM (own kernel) | Yes | Yes | Yes |
| Docker inside | Yes | Yes | Yes |
| Nested virtualization | Yes | No | Yes |
| Root | Yes | Yes | Yes |
| Systemd | No | No | Yes |
| Desktop GUI | No | No | Yes |
| Browser | Yes | Yes | Yes |
| Computer-use API | No | No | Yes |
| SSH | Yes | No | Yes |
| Persistent disk | Yes | No | Yes |
| Pause/resume | No | No | Yes |
| Automatic snapshots | No | No | Yes |
| Fork/clone | No | No | Yes |
| Live fork (no pause) | No | No | Yes |
| Memory fork | No | No | No |
| Custom image (Docker or snapshot) | No | Yes | Yes |
| ≥24 h sessions | Yes | No | Yes |
| Idle auto-stop | Yes | Yes | No |
| Wake on request | No | No | No |
| HTTPS preview URLs | No | No | Yes |
| Raw TCP inbound | No | No | Yes |
| Public IPv4 | No | No | No |
| Egress allowlist | No | No | No |
| Secret proxy | No | Yes | Yes |
| Secret proxy for any API | No | No | Yes |
| Preinstalled agents | No | Yes | Yes |
| Agent harness API | No | Yes | Yes |
| MCP server | No | No | No |
| GPU | No | Yes | No |
| Windows | Yes | Yes | No |
| Arm64 | Yes | Yes | No |
| Self-hosting | No | Yes | No |
| BYOC | No | No | No |
| Open source | No | Yes | No |
| Scoped API keys | Not published | Not published | Yes |
| Spend limits | Not published | Not published | Yes |
| Usage per sandbox (API) | Yes | No | Yes |
| Audit logs | Not published | Yes | No |
| SSO | No | Yes | No |
| SOC 2 | Yes | Yes | No |
| HIPAA | No | No | No |
| EU data residency | Yes | No | Yes |
| Zero data retention | Not published | Not published | Yes |
Performance
There is no comparable CPU measurement for Blacksmith or for GitLab.com hosted runners: neither was part of the same benchmark run. boat did 26.69 runs per second on the StarSling hpc-sandbox-benchmarks Node.js test at 4 vCPU / 8 GB, on dedicated AMD Ryzen 9 9950X hosts.
Node.js build-and-test loop at 4 vCPU / 8 GB, higher is better. Measured with the open hpc-sandbox-benchmarks suite. Blacksmith and GitLab.com hosted runners have no published run in this suite yet, so there is no like-for-like number to compare.
Pricing in five scenarios
Each bill is priced from the providers' public price lists by the open Battleships engine, with the cheapest plan that runs the scenario.
| Scenario | Blacksmith, a month | GitLab.com hosted runners, a month | boat, a month |
|---|---|---|---|
| One coding agent, workdays 4 vCPU / 8 GB, 50 GB disk, 8 hours a day, 22 days a month, files kept between days | $36 · Pay-as-you-go · with limits | $119 · Premium · with limits | $20 · $20/mo |
| One agent that never stops 4 vCPU / 8 GB, 50 GB disk, running 24/7 all month | $207 · Pay-as-you-go · with limits | $805 · Premium · with limits | $26 · $20/mo |
| 50 agents in parallel 50 machines of 4 vCPU / 8 GB at once, 8-hour sessions, 1,100 sessions a month, Docker inside | $2,364 · Pay-as-you-go · with limits | $9,491 · Premium · with limits | $317 · $20/mo × 3 seats |
| Sandboxes for your users 2 vCPU / 4 GB, 30,000 sessions of 10 minutes a month, up to 300 at once | $663 · Pay-as-you-go · with limits | $2,658 · Premium · with limits | $200 · $100/mo × 2 seats |
| 1,000 short jobs at once 2 vCPU / 4 GB, 1,000 at once, 3 minutes each, 200,000 jobs a month | $1,488 · Pay-as-you-go · with limits | $5,896 · Premium · with limits | $2,000 · $2000/mo · with limits |
Blacksmith: Blacksmith bills per minute of job time, scaled by vCPU and OS: Ubuntu x64 is $0.004 a minute for 2 vCPU, ARM $0.0025, Windows $0.008 and macOS $0.08 for 6 vCPU, with 3,000 free minutes a month. Sticky disks cost $0.50 per GB a month.
GitLab.com hosted runners: Each job uses compute minutes equal to its run time times a cost factor, from 1 for small Linux up to 12 for the biggest machines. Plans include minutes per group (10,000 on Premium, which is $29 per user a month billed yearly), and extra minutes cost $10 per 1,000, bought ahead.
GitLab.com hosted runners: Plans start at $20 a month, and the whole fee is usage credit. You pay per second while a sandbox runs, and a stopped sandbox costs nothing and keeps its disk.
Limits in those scenarios
Neither service fits the agent scenarios on this page, because both are CI runners. On Blacksmith only sticky disk folders survive a run, so packages installed elsewhere reset; on GitLab every job ends after 3 hours, nothing on the disk survives, and new jobs stop when a group's minutes run out. Blacksmith's docs say it has no concurrency limit and GitLab publishes none, so the 300 and 1,000 machine scenarios are not confirmed for either, and boat's $2,000 plan queues starts above 90 a minute for the 1,000-job burst.
| Scenario | Blacksmith | GitLab.com hosted runners | boat |
|---|---|---|---|
| One coding agent, workdays |
|
| None that bite |
| One agent that never stops |
|
| None that bite |
| 50 agents in parallel |
|
| None that bite |
| Sandboxes for your users |
|
| None that bite |
| 1,000 short jobs at once |
|
|
|
Which one should you use?
Use Blacksmith when
your code is on GitHub and you want Actions jobs to run faster and cheaper by changing one label, with no concurrency cap.
- A drop-in for GitHub Actions. If your code and workflows live on GitHub, you switch by changing one runner label and keep everything else.
- Lower price per minute. Ubuntu x64 costs $0.004 a minute for 2 vCPU, against $0.01 for GitLab's small Linux runner once included minutes run out.
- No concurrency limit, more regions. Big test matrices start at once with no published cap, on runners in the US east, US west, Amsterdam and Frankfurt.
Use GitLab.com hosted runners when
your code is on GitLab.com, or you want a hosted GPU runner, your own runners free of minutes, or agents working on GitLab issues.
- Native CI for GitLab projects. For a project on GitLab.com, hosted runners are on by default and run pipelines with caches and artifacts built in.
- Your own runners, free of minutes. You can run the open-source GitLab Runner on your own machines, and those jobs use no compute minutes.
- A GPU and agents on issues. Premium and Ultimate add a Linux runner with an NVIDIA T4 and let Claude Code or Codex work on issues and merge requests.
Where boat fits
Blacksmith and GitLab.com hosted runners are both CI services: a machine exists only while a pipeline job runs. If what you are really running is agents, boat gives each one its own full VM that you start by API, keep as long as you like and reach from outside.
A machine that stays
Both services wipe the machine after each job, and GitLab stops jobs at 3 hours. Workarounds are sticky disks (5 folders) or a 5 GB cache, so agents keep reinstalling. A boat disk is saved every minute and can be forked.
Reach in: hosting and a desktop
Blacksmith offers only debug SSH for whoever started the job, and GitLab blocks all inbound traffic, so previewing an agent's app needs a tunnel inside the job. boat gives HTTPS on 50 ports, raw TCP and a 60fps desktop.
Agents driven from outside, keys kept out
Blacksmith machines start from workflows and GitLab's agents start from issue mentions, with other tokens sitting inside the job. boat lets you prompt and steer seven preinstalled agents by API, with real keys in a credentials proxy.
Use boat when your agents need their own machine outside CI: one you start by API, that keeps its disk, runs past 3 hours, hosts what it builds and keeps real API keys out of reach. Compare it one to one: boat vs Blacksmith and boat vs GitLab.com hosted runners.
Questions people ask
Is Blacksmith cheaper than GitLab.com hosted runners?
Per minute, yes: Ubuntu x64 on Blacksmith is $0.004 a minute for 2 vCPU, and an extra GitLab minute on a small Linux runner is $0.01. For the CI runners use case the estimate is $708 against $2,089 a month. GitLab plans include minutes, so a small team may stay within them.
Can I use Blacksmith with GitLab?
No. Blacksmith runs GitHub Actions jobs only, started by workflows. For a GitLab project you use GitLab's hosted runners or run the open-source GitLab Runner on your own machines.
Which is faster, Blacksmith or GitLab.com hosted runners?
We have no comparable CPU measurement for either, so we cannot say. boat did 26.69 runs per second on the same benchmark at 4 vCPU / 8 GB.
Can Blacksmith and GitLab runners run Docker?
Both run Docker. Blacksmith adds a Docker layer cache and supports nested virtualization on x64 Linux runners; GitLab Linux jobs run in a privileged container for Docker-in-Docker, and nested virtualization is not documented.
Do files persist between jobs?
Not on the machine. Blacksmith keeps up to 5 sticky disk folders per job, and GitLab carries files over through a shared cache or artifacts. A boat sandbox keeps its whole disk, saved every minute and on stop.
Can I run AI agents on Blacksmith or GitLab runners?
Partly. Blacksmith's Testbox CLI, in early beta, lets a local agent run commands in a CI job, and its Sandboxes product is marked coming soon. GitLab's Duo Agent Platform runs Claude Code and Codex as CI jobs that end within 3 hours.
Try boat
Install the CLI, then run boat new.
7-day free trial