Blacksmith vs RunsOn
Both make GitHub Actions jobs faster by changing one runner label. Blacksmith runs the jobs for you on its own machines and bills per minute, and its Sandboxes product for agents is in a waitlist beta. RunsOn puts a new EC2 machine in your own AWS account for each job and charges a yearly licence on top of the AWS bill.
The companies and products
Blacksmith
Blacksmith is a CI service: it runs your GitHub Actions jobs on its own runners instead of GitHub's, at a lower per-minute price.
You change the runner label in your workflow file, and each job runs in a fresh Firecracker microVM that boots in under 3 seconds and is destroyed when the job ends. It adds Docker layer caching and sticky disks that carry chosen folders from one run to the next, plus a Testbox CLI in early beta that lets a local coding agent run commands inside a CI job. Its Sandboxes product for coding agents is in a waitlist beta: the page shows a Sandbox.create() API and environment snapshots to boot from, and a preview SDK covers exec, files, exposed ports and pause/resume. Sandboxes have no published price yet.
- Kind of product
- Drop-in GitHub Actions runners; jobs start from workflows
- Compliance
- SOC 2 Type 2, GDPR compliant
RunsOn
RunsOn is self-hosted GitHub Actions runners on AWS: one stack in your account, a yearly licence, and compute that AWS bills at cost.
You install RunsOn in your own AWS account with CloudFormation or Terraform (Flex), or with Terraform (Fleet). You change the runs-on: label in your workflows. Each job then starts on a new EC2 instance, and the instance is terminated when the job ends. RunsOn covers Linux on x64 and arm64, Windows Server and GPU instances, with caches in S3, sticky disks and warm pools. It is a CI runner, a different kind of product from boat. This page compares the part where they overlap: a machine where an agent or a job does its work.
Which use cases each one fits
For each use case, Battleships prices every sandbox provider on the same workload and sorts them from the cheapest monthly bill up. “84th cheapest of 374” means 83 of the other 373 providers run that workload, with every feature it needs, for less. “With limits” means the provider runs it but misses something, so it is listed after every full match. Open a use case for its whole ranking and what each provider misses.
| Use case and its workload | Blacksmith: price rank | RunsOn: price rank | boat: price rank |
|---|---|---|---|
| Software factory 100 at once · 4 vCPU / 8 GB · 2,200 sessions of 8 h a month Agents that build and run whole apps need hosting, a kept disk and forks, which boat has. Blacksmith and RunsOn run the CI part of that loop; Blacksmith's Sandboxes beta lists exposed ports and environment snapshots. | 262nd of 365, with limits $3,607 a month | 143rd of 365, with limits $970 a month | 5th cheapest of 365 $634 a month |
| Always-on AI employees 50 machines 24/7 · 2 vCPU / 4 GB An agent that runs all month needs a machine that keeps running. Both CI runners live for one job, and GitHub ends RunsOn jobs after 5 days. boat has no session cap once auto-stop is off (default 1 h). | 140th of 361, with limits $5,463 a month | 116th of 361, with limits $1,724 a month | 6th cheapest of 361 $657 a month |
| Code interpreter bursts 100 at once · 1 vCPU / 2 GB · 200,000 sessions of 1 min a month | 278th of 336, with limits $488 a month | 226th of 336, with limits $147 a month | 189th of 336, with limits $2,000 a month |
| Sandboxes for your users 300 at once · 2 vCPU / 4 GB · 30,000 sessions of 23 min a month Giving each of your users a sandbox means starting machines from your own backend. boat does this by API today; RunsOn starts machines only for GitHub workflows, and Blacksmith's Sandboxes API is a waitlist beta in preview. | 263rd of 349, with limits $1,563 a month | 211th of 349, with limits $585 a month | 3rd cheapest of 349 $209 a month |
| RL rollouts 1,000 at once · 2 vCPU / 4 GB · 1,000,000 sessions of 3 min a month | 270th of 349, with limits $7,488 a month | 240th of 349, with limits $2,714 a month | 39th of 349, with limits $2,000 a month |
| Agent evals 250 at once · 4 vCPU / 8 GB · 15,000 sessions of 20 min a month | 350th of 359, with limits $1,188 a month | 250th of 359, with limits $341 a month | 10th cheapest of 359 $200 a month |
| CI runners 40 at once · 8 vCPU / 16 GB · 6,000 sessions of 12 min a month This is what both are built for: Blacksmith is the hands-off pick, and RunsOn suits teams that want jobs in their own AWS account. boat ranks 2nd here and runs CI on full VMs you start by API, within its per-plan start limits. | 84th cheapest of 374 $708 a month | 56th cheapest of 374 $432 a month | 2nd cheapest of 374 $86 a month |
| 24/7 dev boxes 1 machine 24/7 · 4 vCPU / 8 GB A dev box has to stay up and keep its files. On both, the runner ends with its job; you carry state over with sticky disks, and on RunsOn also with stopped warm pools. A boat sandbox keeps its disk for at least 14 days after a stop. | 197th of 371, with limits $207 a month | 175th of 371, with limits $78 a month | 19th cheapest of 371 $26 a month |
| Computer-use agents 50 at once · 4 vCPU / 8 GB · 20,000 sessions of 15 min a month Computer-use agents need a screen. Neither CI runner ships one; you set it up: run Xvfb in a Blacksmith job and view it over debug SSH, or install a desktop in your RunsOn image and reach it over SSM or SSH. boat streams a 60 fps Linux desktop with Chrome. | 273rd of 367, with limits $1,088 a month | 173rd of 367, with limits $310 a month | 7th cheapest of 367 $200 a month |
| Browser-use agents 50 at once · 2 vCPU / 4 GB · 30,000 sessions of 10 min a month | 188th of 356, with limits $588 a month | 177th of 356, with limits $224 a month | 7th cheapest of 356 $200 a month |
| Scraping & bots (startup scale) 50 at once · 1 vCPU / 2 GB · 100,000 sessions of 3 min a month | 197th of 343, with limits $719 a month | 185th of 343, with limits $205 a month | 6th cheapest of 343 $90 a month |
| Mass scraping & bots (undetectable) 1,000 at once · 1 vCPU / 2 GB · 3,000,000 sessions of 2 min a month | 262nd of 327, with limits $14,613 a month | 124th of 327, with limits $3,539 a month | 73rd of 327, with limits $1,800 a month |
| Parallel red teaming 200 at once · 2 vCPU / 4 GB · 5,000 sessions of 30 min a month | 115th of 351, with limits $350 a month | 98th of 351, with limits $157 a month | 3rd cheapest of 351 $200 a month |
| GPU eval jobs 10 at once · 8 vCPU / 32 GB · 500 sessions of 30 min a month Neither fits in our model, though RunsOn can launch NVIDIA and AMD GPU instances for CI jobs, which Blacksmith and boat do not offer. | Not ranked: misses a requirement | Not ranked: misses a requirement | Not ranked: misses a requirement |
| macOS agents 5 at once · 4 vCPU / 8 GB · 1,000 sessions of 20 min a month Blacksmith has macOS runners on Apple Silicon; RunsOn does not support macOS, and boat runs Linux only. | 26th of 35, with limits $1,508 a month | Not ranked: misses a requirement | Not ranked: misses a requirement |
Key features
| Feature | Blacksmith | RunsOn | boat |
|---|---|---|---|
| What it is | Managed GitHub Actions runners on Blacksmith's own hardware; Sandboxes for agents in waitlist beta | Self-hosted GitHub Actions runners, deployed as one stack in your AWS account | A full VM per agent that you create, stop, fork and delete by API |
| Isolation | Firecracker microVM per job, destroyed with its state when the job ends | New EC2 instance per job, terminated when the job ends | Full VM with its own kernel on hosts dedicated to boat (AMD Ryzen 7 9700X and Ryzen 9 9950X); vCPUs are shared |
| How a machine starts | A GitHub Actions job with a Blacksmith runner label; Sandboxes beta (preview SDK) adds Sandbox.create() from your own code | A GitHub Actions job with a runs-on: label; launches paced by your AWS quotas | One API call, SDK or CLI, any time; each create, fork or resume counts against the plan's included starts |
| How long it lives | One job; Testbox stops after 30 idle minutes by default | One job; GitHub ends self-hosted jobs after 5 days | Until you stop it or auto-stop ends it (default 1 h, up to 30 days, or off on a paid plan) |
| What survives between runs | Up to 5 sticky disk folders per job, evicted after 7 days unused; Sandboxes beta adds pause/resume | Sticky EBS disks, snapshotted after a job and restored before the next; stopped warm pools keep dependencies installed | The disk, saved every minute and on stop; memory and running programs are not saved. A stopped sandbox has no compute charge and keeps its disk for at least 14 days |
| Fork | Runners: no fork. Sandboxes beta boots many sandboxes from one environment snapshot, not from a running one | Not documented; every job starts from an AMI | Fork a running sandbox with its disk, from the last 1-minute snapshot |
| Nested virtualization | Yes on x64 Linux runners, not on ARM | KVM on compatible Intel x64 families, not arm64 | /dev/kvm on every sandbox, with Docker, root and systemd |
| Operating systems | Ubuntu x64 and ARM, Windows Server 2025 (beta), macOS on Apple Silicon | Linux x64 and arm64, Windows Server 2022 and 2025; no macOS | Linux (x64); Windows programs through Wine |
| Biggest machine and GPUs | Up to 32 vCPU / 128 GB on Ubuntu x64; no GPUs | Any EC2 size up to 96 vCPU, plus NVIDIA and AMD GPUs | Up to 16 vCPU / 32 GB on the $100 plan and above; CPU only |
| Where it runs | Blacksmith's hardware in Ashburn, Phoenix, Amsterdam and Frankfurt | Your own AWS account, in the region you pick | Managed by boat on machines in Germany, Finland and France; no US or Asia region, and you cannot pick one |
| Running many at once | Docs say Blacksmith imposes no concurrency limits | Set by your AWS account's quotas; launches paced at about 2 EC2 calls per second by default | 100 to 2,000 sandboxes by plan; included starts per minute, hour and day, extra starts $0.03 to $0.07; boat never queues a start |
| Inbound access | Runners: opt-in debug SSH for the user who triggered the job; no ports or URLs. Sandboxes beta (preview SDK) adds exposed ports | SSH from a CIDR range you allow, and SSM; HTTPS previews you set up with your own load balancer or tunnel in AWS | HTTPS URLs for up to 50 ports, token-gated by default; SSH; public IPv6; no static IP, and the IP changes at each resume |
| Desktop and computer use | No desktop stream; you set it up: run Xvfb in the job and view it over debug SSH | Full runner images include browsers for headless use; a desktop you set up in your image, reached over SSM or optional SSH | 1920x1080, 60 fps Linux desktop with Chrome and computer tools |
| Coding agents and keys | Its own [code]smith agent; Testbox CLI (early beta) for local agents; secrets inside the job | Bedrock through the instance role; Copilot's coding agent can use a runner; secrets inside the job | Seven agents preinstalled with a prompt, steer and events API; optional paid credentials proxy keeps the real keys of supported providers out |
Every feature Battleships tracks (43)
| Feature | Blacksmith | RunsOn | boat |
|---|---|---|---|
| Full VM (own kernel) | Yes | Yes | Yes |
| Docker inside | Yes | Yes | Yes |
| Nested virtualization | Yes | Yes | Yes |
| Root | Yes | Yes | Yes |
| Systemd | No documented way | Not published | Yes |
| Desktop GUI | No documented way | No documented way | Yes |
| Browser | Yes | Yes | Yes |
| Computer-use API | No documented way | No documented way | Yes |
| SSH | Yes | Yes | Yes |
| Persistent disk | Yes | Yes | Yes |
| Pause/resume | No documented way | Yes | Yes |
| Automatic snapshots | No documented way | Yes | Yes |
| Fork/clone | No documented way | No documented way | Yes |
| Live fork (no pause) | No documented way | No documented way | Yes |
| Memory fork | No documented way | No documented way | No documented way |
| Custom image (Docker or snapshot) | No documented way | Yes | Yes |
| ≥24 h sessions | Yes | Yes | Yes |
| Idle auto-stop | Yes | Yes | No documented way |
| Wake on request | No documented way | Yes | No documented way |
| HTTPS preview URLs | No documented way | No documented way | Yes |
| Raw TCP inbound | No documented way | No documented way | Yes |
| Public IPv4 | No documented way | Yes | No documented way |
| Egress allowlist | No documented way | Yes | No documented way |
| Secret proxy | No documented way | No documented way | Yes |
| Secret proxy for any API | No documented way | No documented way | Yes |
| Preinstalled agents | No documented way | No documented way | Yes |
| Agent harness API | No documented way | No documented way | Yes |
| MCP server | No documented way | No documented way | No documented way |
| GPU | No documented way | Yes | No documented way |
| Windows | Yes | Yes | No documented way |
| Arm64 | Yes | Yes | No documented way |
| Self-hosting | No documented way | Yes | No documented way |
| BYOC | No documented way | Yes | No documented way |
| Open source | No documented way | No documented way | No documented way |
| Scoped API keys | Not published | Not published | Yes |
| Spend limits | Not published | Not published | Yes |
| Usage per sandbox (API) | Yes | No documented way | Yes |
| Audit logs | Not published | Not published | No documented way |
| SSO | No documented way | No documented way | No documented way |
| SOC 2 | Yes | No documented way | No documented way |
| HIPAA | No documented way | No documented way | No documented way |
| EU data residency | Yes | Yes | No documented way |
| Zero data retention | Not published | Not published | Yes |
Performance
Blacksmith and RunsOn are not in the public benchmarks we use, so we cannot say which of the three is faster. RunsOn's speed depends on the EC2 family you pick. For reference, boat did 28.06 runs per second on the V8 Web Tooling Benchmark at 4 vCPU / 8 GB, second of 15 in StarSling's run of 3 October 2026 (run 37073680030). Every boat sandbox in that run ran on an AMD Ryzen 9 9950X host; boat also places sandboxes on Ryzen 7 9700X hosts and does not promise a CPU model.
V8 Web Tooling Benchmark at a 4 vCPU / 8 GB target, runs per second, higher is better. Source: StarSling's run of 3 October 2026 (run 37073680030), median of 5 sandboxes. Providers ran on different hardware in different regions. StarSling gives a shared rank when the run cannot separate two providers. Every boat sandbox in this run ran on an AMD Ryzen 9 9950X host. boat also places some sandboxes on AMD Ryzen 7 9700X hosts, which this run did not measure. Blacksmith and RunsOn are not in the public benchmarks we use, so we cannot say which is faster.
Pricing in five scenarios
Each bill is priced from the providers' public price lists by the open Battleships engine, with the cheapest plan that runs the scenario. Start limits, per-start fees and plan minimums are in the price, boat's included.
| Scenario | Blacksmith, a month | RunsOn, a month | boat, a month |
|---|---|---|---|
| One coding agent, workdays 4 vCPU / 8 GB, 50 GB disk, 8 hours a day, 22 days a month, files kept between days | $36 · Pay-as-you-go · with limits | $61 · Starter · with limits | $20 · $20/mo |
| One agent that never stops 4 vCPU / 8 GB, 50 GB disk, running 24/7 all month | $207 · Pay-as-you-go · with limits | $159 · Starter · with limits | $26 · $20/mo |
| 50 agents in parallel 50 machines of 4 vCPU / 8 GB at once, 8-hour sessions, 1,100 sessions a month, Docker inside | $2,364 · Pay-as-you-go · with limits | $1,597 · Starter · with limits | $317 · $20/mo × 3 seats |
| Sandboxes for your users 2 vCPU / 4 GB, 30,000 sessions of 10 minutes a month, up to 300 at once | $663 · Pay-as-you-go · with limits | $515 · Starter · with limits | $200 · $100/mo × 2 seats |
| 1,000 short jobs at once 2 vCPU / 4 GB, 1,000 at once, 3 minutes each, 200,000 jobs a month | $1,488 · Pay-as-you-go · with limits | $1,103 · Starter · with limits | $8,800 · $2000/mo · with limits |
Blacksmith: Blacksmith bills per minute of job time, scaled by vCPU and OS: Ubuntu x64 is $0.004 a minute for 2 vCPU, ARM $0.0025, Windows $0.008 and macOS $0.08 for 6 vCPU, with 3,000 free minutes a month and no plan fee. Sticky disks cost $0.50 per GB a month and a static IP $100 a month. Sandboxes have no published price.
RunsOn: AWS bills the EC2 instances and EBS disks on your own AWS bill, to the second, at spot or on-demand prices with no markup; S3 cache traffic, data transfer and a public IPv4 are extra. RunsOn charges one licence a year per company, from EUR 300 for under 50,000 jobs a month. The prices here use spot rates in us-east-1; spot machines can be interrupted.
RunsOn: Plans cost $20, $100, $500 or $2,000 a month, and the whole plan price comes back as VM time for that month; unused time does not carry over. A card is required, also for the 7-day trial. A 4 vCPU / 8 GB sandbox costs $0.036 an hour, billed per second while it runs. Each plan includes a set number of starts (the $20 plan: 12 a minute, 200 a day, 100 running at once), and extra starts cost $0.03 to $0.07. A stopped sandbox has no compute charge and keeps its disk for at least 14 days; saved snapshots above the included count and the credentials proxy still charge.
Limits in those scenarios
None of the five scenarios fits Blacksmith or RunsOn cleanly, because each is a CI runner: a machine exists only for one GitHub Actions job, and only sticky disk data survives, so packages installed elsewhere reset. RunsOn jobs also end after GitHub's 5-day self-hosted limit, so an agent that never stops needs restarts. Blacksmith's docs say it has no concurrency limit, while RunsOn's ceiling is your AWS account's quotas. On boat the limit for 1,000 short jobs is starts: the $2,000 plan charges for starts above 90 a minute and refuses them above 270, and boat never queues a start.
| Scenario | Blacksmith | RunsOn | boat |
|---|---|---|---|
| One coding agent, workdays |
|
| None for this scenario |
| One agent that never stops |
|
| None for this scenario |
| 50 agents in parallel |
|
| None for this scenario |
| Sandboxes for your users |
|
| None for this scenario |
| 1,000 short jobs at once |
|
|
|
Which one should you use?
Use Blacksmith when
you want faster GitHub Actions with nothing to run yourself, need macOS runners, or want a big test matrix to start all at once.
- Nothing to run yourself. Teams that want faster GitHub Actions without owning any cloud setup change one label, and Blacksmith runs the machines.
- macOS runners. Teams that build or test iOS and macOS apps can run those jobs on Apple Silicon, which RunsOn does not support.
- No concurrency limit. Big test matrices can start all at once, since Blacksmith's docs say it imposes no concurrency limits.
Use RunsOn when
your CI must run in your own AWS account and region, at AWS cost, or needs GPUs or EC2 machines up to 96 vCPU.
- Your own AWS account. Teams whose code, secrets and logs must stay in their AWS account keep them there, in any region they pick.
- AWS prices with no markup. AWS bills the EC2 time at spot or on-demand cost, and RunsOn adds one flat yearly licence from EUR 300.
- GPUs and very large machines. Jobs that need NVIDIA or AMD GPUs, or EC2 machines up to 96 vCPU, can get them on RunsOn.
Where boat fits
Blacksmith and RunsOn are CI runners: a machine exists only while a GitHub Actions job runs. If what you are really running is agents, you want a machine your own code starts, that keeps its disk, stays up as long as the work needs and can be reached from outside. That is what boat is built for, and it can still run CI jobs on the same VMs.
A machine that stays and keeps its disk
On both, the runner is destroyed when the job ends, and only sticky disk data comes back, so an agent sets up the rest again every run. boat saves the disk every minute, has no session cap once auto-stop is off (default 1 h), and forks running sandboxes; a resume is like a reboot.
Started by your code, not a workflow
Blacksmith runners and RunsOn start machines for GitHub Actions jobs, so serving users or agents means wrapping each one in a workflow run; Blacksmith's Sandboxes SDK is still in preview. boat starts a sandbox with one API call, SDK or CLI from your own backend, within its per-plan start limits.
Reach in: hosting, a desktop, agents
Their runners are reached over SSH or SSM, so previews and computer use need your own tunnels and desktop setup. boat gives token-gated HTTPS URLs on up to 50 ports, a 60 fps desktop and coding agents you drive from outside.
Use boat when you need machines for agents rather than CI jobs: a full VM you start by API, keep with its disk, fork, reach over HTTPS and drive with built-in coding agents. Compare it one to one: boat vs Blacksmith and boat vs RunsOn.
boat's own limits
- Plans start at $20 a month. The plan price comes back as VM time for that month. Unused time does not carry over. Source
- Starts (create, fork, resume): the $20 plan includes 12 a minute, 60 an hour and 200 a day. The $2,000 plan includes 90 a minute, 600 an hour and 2,400 a day. Extra starts cost $0.03 to $0.07. A start fails with 429 above 3 times the included starts per minute, or 6 times per hour or per day. Source
- A stopped sandbox has no compute charge. boat keeps its disk for at least 14 days. A saved snapshot keeps it longer; extra saved snapshots and the AI credentials proxy are paid. Source
- Auto-stop defaults to 1 hour. You can turn it off on a paid plan. The trial allows 2 hours at most. Source
- boat saves the disk every minute and at each stop. Memory, running programs and /tmp are not saved: a resume is like a reboot. A machine failure loses writes after the last good snapshot. Source
- Machines run in Germany, Finland and France only. Snapshot copies can be stored outside the EU. Source
- Public IPv6 and outbound IPv4. No dedicated IPv4. The IP changes at each resume. Source
- Hosts are AMD Ryzen 7 9700X and Ryzen 9 9950X. vCPUs are shared. No GPU. You cannot boot your own Docker image. Source
Questions people ask
Is Blacksmith cheaper than RunsOn?
In our CI runner use case RunsOn comes to $432 a month and Blacksmith to $708, because AWS bills spot minutes at cost and RunsOn adds only a yearly licence. Blacksmith includes the machines, caching and upkeep in its per-minute price, so there is no AWS setup to run. boat comes to $86 a month for the same use case.
Which is faster, Blacksmith or RunsOn?
Neither is in the public benchmarks we use, so we cannot say. RunsOn's speed depends on the EC2 family you choose. For reference, boat did 28.06 runs per second on the V8 Web Tooling Benchmark at 4 vCPU / 8 GB in StarSling's run of 3 October 2026.
Do Blacksmith and RunsOn keep files between jobs?
On sticky disks, yes. Blacksmith keeps up to 5 folders per job and evicts them after 7 days unused. RunsOn snapshots sticky EBS disks after a job and restores them before the next, and stopped warm pools keep dependencies installed. The rest of the machine is new every time. boat saves a sandbox's disk every minute and on stop; memory is not saved.
Can Blacksmith or RunsOn run Docker and nested virtualization?
Both run Docker. Blacksmith supports nested virtualization on x64 Linux runners but not ARM, and RunsOn offers KVM on compatible Intel x64 families but not arm64. boat has Docker, root, systemd and /dev/kvm on every sandbox.
Can I run an AI agent on Blacksmith or RunsOn?
Inside GitHub Actions jobs, yes. Blacksmith has a Testbox CLI in early beta and its own [code]smith agent, and its Sandboxes product is in waitlist beta with a preview SDK. RunsOn gives runners Bedrock access and can host Copilot's coding agent. boat starts agent sandboxes by API with seven coding agents preinstalled.
Where does my code run?
Blacksmith runs jobs on its own hardware in Ashburn, Phoenix, Amsterdam and Frankfurt. RunsOn runs in your own AWS account and region, and your code, secrets and logs are never sent to RunsOn. boat runs sandboxes on machines in Germany, Finland and France, and you cannot pick a region; snapshot copies can be stored outside the EU.
Try boat
Install the CLI, then run boat new.
7-day free trial