Blacksmith vs RunsOn
Both make GitHub Actions jobs faster by changing one runner label. Blacksmith runs the jobs for you on its own machines and bills per minute, while RunsOn puts a fresh EC2 machine in your own AWS account for each job and charges a yearly licence on top of the AWS bill.
The companies and products
Blacksmith
Blacksmith is a CI service: it runs your GitHub Actions jobs on its own faster, cheaper runners instead of GitHub's.
You change the runner label in your workflow file, and each job runs in a fresh Firecracker microVM that boots in under 3 seconds and is destroyed when the job ends. It adds Docker layer caching and sticky disks that carry chosen folders from one run to the next, plus a Testbox CLI in early beta that lets a local coding agent run commands inside a CI job. It is built for teams that want quicker, cheaper GitHub Actions, not as a general sandbox you drive by API; a separate Sandboxes product is marked coming soon.
RunsOn
RunsOn is self-hosted GitHub Actions runners on AWS: one CloudFormation or Terraform stack in your account, a yearly licence, and compute billed by AWS at cost.
You install RunsOn in your own AWS account, change the runs-on: label in your workflows, and every job then starts on a new EC2 instance that is terminated when the job ends. It covers Linux on x64 and arm64, Windows Server and GPU instances, with caches in S3 and sticky EBS disks. It is a CI runner, a different kind of product from boat, so this page compares the part where they overlap: a machine where an agent or a job does its work.
Which use cases each one fits
For each use case, Battleships prices every sandbox provider on the same workload and sorts them from the cheapest monthly bill up. “78th cheapest of 377” means 77 of the other 376 providers run that workload, with every feature it needs, for less. “With limits” means the provider runs it but misses something, so it is listed after every full match. Open a use case for its whole ranking and what each provider misses.
| Use case and its workload | Blacksmith: price rank | RunsOn: price rank | boat: price rank |
|---|---|---|---|
| Software factory 100 at once · 4 vCPU / 8 GB · 2,200 sessions of 8 h a month Agents that build and run whole apps need hosting, a kept disk and forks, which boat has; Blacksmith and RunsOn only run the CI part of that loop. | 248th of 367, with limits $3,607 a month | 122nd of 367, with limits $1,114 a month | 3rd cheapest of 367 $634 a month |
| Always-on AI employees 50 machines 24/7 · 2 vCPU / 4 GB An agent that runs all month needs a machine that never stops; both CI runners live for one job, RunsOn jobs end after GitHub's 5-day limit, and boat has no session cap on paid plans. | 78th of 366, with limits $5,463 a month | 188th of 366, with limits $1,724 a month | 5th cheapest of 366 $657 a month |
| Code interpreter bursts 100 at once · 1 vCPU / 2 GB · 200,000 sessions of 1 min a month | 281st of 345, with limits $488 a month | 229th of 345, with limits $147 a month | 186th of 345, with limits $2,000 a month |
| Sandboxes for your users 300 at once · 2 vCPU / 4 GB · 30,000 sessions of 23 min a month Giving each of your users a sandbox means starting machines from your own backend, which boat does by API and neither Blacksmith nor RunsOn does outside GitHub workflows. | 268th of 361, with limits $1,563 a month | 216th of 361, with limits $585 a month | Cheapest of 361 $209 a month |
| RL rollouts 1,000 at once · 2 vCPU / 4 GB · 1,000,000 sessions of 3 min a month | 276th of 360, with limits $7,488 a month | 245th of 360, with limits $2,714 a month | 34th of 360, with limits $2,000 a month |
| Agent evals 250 at once · 4 vCPU / 8 GB · 15,000 sessions of 20 min a month | 354th of 364, with limits $1,188 a month | 251st of 364, with limits $389 a month | 10th cheapest of 364 $200 a month |
| CI runners 40 at once · 8 vCPU / 16 GB · 6,000 sessions of 12 min a month This is what both are built for: Blacksmith is the hands-off pick, RunsOn suits teams that want jobs in their own AWS account, and boat runs CI on full VMs you start by API. | 78th cheapest of 377 $708 a month | 58th cheapest of 377 $432 a month | 4th cheapest of 377 $86 a month |
| 24/7 dev boxes 1 machine 24/7 · 4 vCPU / 8 GB A dev box has to stay up and keep its files, but on both Blacksmith and RunsOn the machine is gone when its job ends and only sticky disk data comes back. | 180th of 368, with limits $207 a month | 158th of 368, with limits $86 a month | 18th cheapest of 368 $26 a month |
| Computer-use agents 50 at once · 4 vCPU / 8 GB · 20,000 sessions of 15 min a month Computer-use agents need a screen; Blacksmith and RunsOn runners are headless, and boat streams a 60fps Linux desktop with Chromium. | 243rd of 369, with limits $1,088 a month | 142nd of 369, with limits $353 a month | 6th cheapest of 369 $200 a month |
| Browser-use agents 50 at once · 2 vCPU / 4 GB · 30,000 sessions of 10 min a month | 131st of 367, with limits $588 a month | 206th of 367, with limits $224 a month | 7th cheapest of 367 $200 a month |
| Scraping & bots (startup scale) 50 at once · 1 vCPU / 2 GB · 100,000 sessions of 3 min a month | 130th of 346, with limits $719 a month | 228th of 346, with limits $205 a month | 6th cheapest of 346 $90 a month |
| Mass scraping & bots (undetectable) 1,000 at once · 1 vCPU / 2 GB · 3,000,000 sessions of 2 min a month | 262nd of 338, with limits $14,613 a month | 122nd of 338, with limits $3,539 a month | 71st of 338, with limits $1,800 a month |
| Parallel red teaming 200 at once · 2 vCPU / 4 GB · 5,000 sessions of 30 min a month | 64th of 360, with limits $350 a month | 178th of 360, with limits $157 a month | 2nd cheapest of 360 $200 a month |
| GPU eval jobs 10 at once · 8 vCPU / 32 GB · 500 sessions of 30 min a month Neither fits in our model, though RunsOn can launch NVIDIA and AMD GPU instances for CI jobs, which Blacksmith and boat do not offer. | Can't run it | Can't run it | Can't run it |
| macOS agents 5 at once · 4 vCPU / 8 GB · 1,000 sessions of 20 min a month Blacksmith has macOS runners on Apple Silicon; RunsOn does not support macOS and boat runs Linux only. | 28th of 36, with limits $1,508 a month | Can't run it | Can't run it |
Key features
| Feature | Blacksmith | RunsOn | boat |
|---|---|---|---|
| What it is | Managed GitHub Actions runners on Blacksmith's own hardware | Self-hosted GitHub Actions runners, deployed as one stack in your AWS account | A full VM per agent that you create, stop, fork and delete by API |
| Isolation | Firecracker microVM per job, destroyed with its state when the job ends | Fresh EC2 instance per job, terminated when the job ends | Full VM with its own kernel on dedicated AMD Ryzen 9 9950X hosts |
| How a machine starts | A GitHub Actions job with a Blacksmith runner label | A GitHub Actions job with a runs-on: label | One API call, SDK or CLI, any time |
| How long it lives | One job; Testbox stops after 30 idle minutes by default | One job; GitHub ends self-hosted jobs after 5 days | Until you stop it; no session cap on paid plans |
| What survives between runs | Up to 5 sticky disk folders per job, evicted after 7 days unused | One sticky EBS disk per job, snapshotted and restored next job | The whole disk, saved every minute and on stop; a stopped sandbox costs nothing |
| Fork | No; every machine starts fresh from a job | No; every job starts from an AMI | Fork a running sandbox with its disk |
| Nested virtualization | Yes on x64 Linux runners, not on ARM | KVM on compatible Intel x64 families, not arm64 | /dev/kvm on every machine, with Docker, root and systemd |
| Operating systems | Ubuntu x64 and ARM, Windows Server 2025 (beta), macOS on Apple Silicon | Linux x64 and arm64, Windows Server 2022 and 2025; no macOS | Linux (x64) |
| Biggest machine and GPUs | Up to 32 vCPU / 128 GB on Ubuntu x64; no GPUs | Any EC2 size up to 96 vCPU, plus NVIDIA and AMD GPUs | Up to 16 vCPU / 32 GB on the $100 plan and above; CPU only |
| Where it runs | Blacksmith's hardware in Ashburn, Phoenix, Amsterdam and Frankfurt | Your own AWS account, in the region you pick | Managed by boat in the EU (Germany, Finland, France) |
| Running many at once | Docs say Blacksmith imposes no concurrency limits | Set by your AWS account's quotas; launches paced at about 2 EC2 calls per second by default | Starts per minute, hour and day depend on the plan |
| Inbound access | Opt-in debug SSH for the user who triggered the job; no ports or URLs | SSH from a CIDR range you allow, and SSM; no HTTPS URLs | HTTPS URLs for up to 50 ports, token-protected by default; raw TCP; dedicated IP |
| Desktop and computer use | Headless runners | Headless runners | 60fps Linux desktop with Chromium and computer use |
| Coding agents and keys | Its own [code]smith agent; Testbox CLI (early beta) for local agents; secrets inside the job | Bedrock through the instance role; Copilot's coding agent can use a runner; secrets inside the job | Seven agents preinstalled with a prompt, steer and events API; credentials proxy keeps real keys out for any HTTPS API |
Every feature Battleships tracks (43)
| Feature | Blacksmith | RunsOn | boat |
|---|---|---|---|
| Full VM (own kernel) | Yes | Yes | Yes |
| Docker inside | Yes | Yes | Yes |
| Nested virtualization | Yes | Yes | Yes |
| Root | Yes | Yes | Yes |
| Systemd | No | Not published | Yes |
| Desktop GUI | No | No | Yes |
| Browser | Yes | No | Yes |
| Computer-use API | No | No | Yes |
| SSH | Yes | Yes | Yes |
| Persistent disk | Yes | Yes | Yes |
| Pause/resume | No | Yes | Yes |
| Automatic snapshots | No | Yes | Yes |
| Fork/clone | No | No | Yes |
| Live fork (no pause) | No | No | Yes |
| Memory fork | No | No | No |
| Custom image (Docker or snapshot) | No | Yes | Yes |
| ≥24 h sessions | Yes | Yes | Yes |
| Idle auto-stop | Yes | Yes | No |
| Wake on request | No | Yes | No |
| HTTPS preview URLs | No | No | Yes |
| Raw TCP inbound | No | No | Yes |
| Public IPv4 | No | Yes | No |
| Egress allowlist | No | Yes | No |
| Secret proxy | No | No | Yes |
| Secret proxy for any API | No | No | Yes |
| Preinstalled agents | No | No | Yes |
| Agent harness API | No | No | Yes |
| MCP server | No | No | No |
| GPU | No | Yes | No |
| Windows | Yes | Yes | No |
| Arm64 | Yes | Yes | No |
| Self-hosting | No | Yes | No |
| BYOC | No | Yes | No |
| Open source | No | No | No |
| Scoped API keys | Not published | Not published | Yes |
| Spend limits | Not published | Not published | Yes |
| Usage per sandbox (API) | Yes | No | Yes |
| Audit logs | Not published | Not published | No |
| SSO | No | No | No |
| SOC 2 | Yes | No | No |
| HIPAA | No | No | No |
| EU data residency | Yes | Yes | Yes |
| Zero data retention | Not published | Not published | Yes |
Performance
There is no comparable measurement for Blacksmith or RunsOn: neither was part of the hpc-sandbox-benchmarks run where boat did 26.69 runs per second at 4 vCPU / 8 GB on dedicated AMD Ryzen 9 9950X hosts. Blacksmith says its runners use fast gaming CPUs, and RunsOn's speed depends on the EC2 family you pick, but we have no number measured the same way to set side by side.
Node.js build-and-test loop at 4 vCPU / 8 GB, higher is better. Measured with the open hpc-sandbox-benchmarks suite. Blacksmith and RunsOn have no published run in this suite yet, so there is no like-for-like number to compare.
Pricing in five scenarios
Each bill is priced from the providers' public price lists by the open Battleships engine, with the cheapest plan that runs the scenario.
| Scenario | Blacksmith, a month | RunsOn, a month | boat, a month |
|---|---|---|---|
| One coding agent, workdays 4 vCPU / 8 GB, 50 GB disk, 8 hours a day, 22 days a month, files kept between days | $36 · Pay-as-you-go · with limits | $61 · Starter · with limits | $20 · $20/mo |
| One agent that never stops 4 vCPU / 8 GB, 50 GB disk, running 24/7 all month | $207 · Pay-as-you-go · with limits | $159 · Starter · with limits | $26 · $20/mo |
| 50 agents in parallel 50 machines of 4 vCPU / 8 GB at once, 8-hour sessions, 1,100 sessions a month, Docker inside | $2,364 · Pay-as-you-go · with limits | $1,597 · Starter · with limits | $317 · $20/mo × 3 seats |
| Sandboxes for your users 2 vCPU / 4 GB, 30,000 sessions of 10 minutes a month, up to 300 at once | $663 · Pay-as-you-go · with limits | $515 · Starter · with limits | $200 · $100/mo × 2 seats |
| 1,000 short jobs at once 2 vCPU / 4 GB, 1,000 at once, 3 minutes each, 200,000 jobs a month | $1,488 · Pay-as-you-go · with limits | $1,103 · Starter · with limits | $2,000 · $2000/mo · with limits |
Blacksmith: Blacksmith bills per minute of job time, scaled by vCPU and OS: Ubuntu x64 is $0.004 a minute for 2 vCPU, ARM $0.0025, Windows $0.008 and macOS $0.08 for 6 vCPU, with 3,000 free minutes a month. Sticky disks cost $0.50 per GB a month and a static IP $100 a month.
RunsOn: AWS bills the EC2 instances and EBS disks on your own AWS bill, to the second, at spot or on-demand prices with no markup; S3 cache traffic and data transfer are extra. RunsOn charges one licence a year per company, from EUR 300 or $350 for under 50,000 jobs a month, and the prices here use spot rates in us-east-1.
RunsOn: Plans start at $20 a month, and the whole fee is usage credit spent per second on running sandboxes. A stopped sandbox costs nothing and keeps its disk.
Limits in those scenarios
None of the five scenarios fits Blacksmith or RunsOn cleanly, because each is a CI runner: a machine exists only for one GitHub Actions job, and only sticky disk data survives, so packages installed elsewhere reset. RunsOn jobs also end after GitHub's 5-day self-hosted limit, so an agent that never stops needs restarts. Blacksmith's docs say it has no concurrency limit, while RunsOn's ceiling is your AWS account's quotas. On boat the limit for 1,000 short jobs is starts: the $2,000 plan queues bursts above 90 starts per minute.
| Scenario | Blacksmith | RunsOn | boat |
|---|---|---|---|
| One coding agent, workdays |
|
| None that bite |
| One agent that never stops |
|
| None that bite |
| 50 agents in parallel |
|
| None that bite |
| Sandboxes for your users |
|
| None that bite |
| 1,000 short jobs at once |
|
|
|
Which one should you use?
Use Blacksmith when
you want faster GitHub Actions with nothing to run yourself, need macOS runners, or want a big test matrix to start all at once.
- Nothing to run yourself. Teams that want faster GitHub Actions without owning any cloud setup change one label, and Blacksmith runs the machines.
- macOS runners. Teams that build or test iOS and macOS apps can run those jobs on Apple Silicon, which RunsOn does not support.
- No concurrency limit. Big test matrices can start all at once, since Blacksmith's docs say it imposes no concurrency limits.
Use RunsOn when
your CI must run in your own AWS account and region, at AWS cost, or needs GPUs or EC2 machines up to 96 vCPU.
- Your own AWS account. Teams whose code, secrets and logs must stay in their AWS account keep them there, in any region they pick.
- AWS prices with no markup. AWS bills the EC2 time at spot or on-demand cost, and RunsOn adds one flat yearly licence from $350.
- GPUs and very large machines. Jobs that need NVIDIA or AMD GPUs, or EC2 machines up to 96 vCPU, can get them on RunsOn.
Where boat fits
Blacksmith and RunsOn are CI runners: a machine exists only while a GitHub Actions job runs. If what you are really running is agents, you want a machine your own code starts, that keeps its whole disk, stays up as long as the work needs and can be reached from outside. That is what boat is built for, and it can still run CI jobs on the same VMs.
A machine that stays and keeps its disk
On both, the machine is destroyed when the job ends, and only sticky disk data comes back, so an agent reinstalls its tools every run. boat keeps the whole disk, saved every minute, with no session cap on paid plans and forks of running sandboxes.
Started by your code, not a workflow
Blacksmith and RunsOn start machines only for GitHub Actions jobs, so serving users or agents means wrapping each one in a workflow run. boat starts a sandbox with one API call, SDK or CLI from your own backend.
Reach in: hosting, a desktop, agents
Their runners are headless and reached only over SSH or SSM, so previews and computer use need your own tunnels and desktop setup. boat gives HTTPS URLs on 50 ports, raw TCP, a 60fps desktop and coding agents you drive from outside.
Use boat when you need machines for agents rather than CI jobs: a full VM you start by API, keep with its whole disk, fork, reach over HTTPS and drive with built-in coding agents. Compare it one to one: boat vs Blacksmith and boat vs RunsOn.
Questions people ask
Is Blacksmith cheaper than RunsOn?
In our CI runner use case RunsOn comes to $432 a month and Blacksmith to $708, because AWS bills spot minutes at cost and RunsOn adds only a yearly licence. Blacksmith includes the machines, caching and upkeep in its per-minute price, so there is no AWS setup to run. boat comes to $86 a month for the same use case.
Which is faster, Blacksmith or RunsOn?
We have no comparable CPU measurement for either. Blacksmith runs on its own gaming CPUs, and RunsOn's speed depends on the EC2 family you choose. boat did 26.69 runs per second on the hpc-sandbox-benchmarks Node.js test at 4 vCPU / 8 GB.
Do Blacksmith and RunsOn keep files between jobs?
Only on sticky disks. Blacksmith keeps up to 5 folders per job and evicts them after 7 days unused; RunsOn snapshots one EBS disk per job and restores it next job. The rest of the machine is new every time, while boat keeps a sandbox's whole disk, saved every minute and on stop.
Can Blacksmith or RunsOn run Docker and nested virtualization?
Both run Docker. Blacksmith supports nested virtualization on x64 Linux runners but not ARM, and RunsOn offers KVM on compatible Intel x64 families but not arm64. boat has Docker, root, systemd and /dev/kvm on every machine.
Can I run an AI agent on Blacksmith or RunsOn?
Only inside GitHub Actions jobs. Blacksmith has a Testbox CLI in early beta and its own [code]smith agent, with a Sandboxes product marked coming soon; RunsOn gives runners Bedrock access and can host Copilot's coding agent. boat starts agent sandboxes by API with seven coding agents preinstalled.
Where does my code run?
Blacksmith runs jobs on its own hardware in Ashburn, Phoenix, Amsterdam and Frankfurt. RunsOn runs in your own AWS account and region, and your code, secrets and logs are never sent to RunsOn. boat runs only in the EU (Germany, Finland and France).
Try boat
Install the CLI, then run boat new.
7-day free trial