Blacksmith vs RunsOn

Both make GitHub Actions jobs faster by changing one runner label. Blacksmith runs the jobs for you on its own machines and bills per minute, and its Sandboxes product for agents is in a waitlist beta. RunsOn puts a new EC2 machine in your own AWS account for each job and charges a yearly licence on top of the AWS bill.

The companies and products

Blacksmith

Blacksmith is a CI service: it runs your GitHub Actions jobs on its own runners instead of GitHub's, at a lower per-minute price.

You change the runner label in your workflow file, and each job runs in a fresh Firecracker microVM that boots in under 3 seconds and is destroyed when the job ends. It adds Docker layer caching and sticky disks that carry chosen folders from one run to the next, plus a Testbox CLI in early beta that lets a local coding agent run commands inside a CI job. Its Sandboxes product for coding agents is in a waitlist beta: the page shows a Sandbox.create() API and environment snapshots to boot from, and a preview SDK covers exec, files, exposed ports and pause/resume. Sandboxes have no published price yet.

RunsOn

RunsOn is self-hosted GitHub Actions runners on AWS: one stack in your account, a yearly licence, and compute that AWS bills at cost.

You install RunsOn in your own AWS account with CloudFormation or Terraform (Flex), or with Terraform (Fleet). You change the runs-on: label in your workflows. Each job then starts on a new EC2 instance, and the instance is terminated when the job ends. RunsOn covers Linux on x64 and arm64, Windows Server and GPU instances, with caches in S3, sticky disks and warm pools. It is a CI runner, a different kind of product from boat. This page compares the part where they overlap: a machine where an agent or a job does its work.

Which use cases each one fits

For each use case, Battleships prices every sandbox provider on the same workload and sorts them from the cheapest monthly bill up. “84th cheapest of 374” means 83 of the other 373 providers run that workload, with every feature it needs, for less. “With limits” means the provider runs it but misses something, so it is listed after every full match. Open a use case for its whole ranking and what each provider misses.

Use case and its workloadBlacksmith: price rankRunsOn: price rankboat: price rank
Software factory
100 at once · 4 vCPU / 8 GB · 2,200 sessions of 8 h a month
Agents that build and run whole apps need hosting, a kept disk and forks, which boat has. Blacksmith and RunsOn run the CI part of that loop; Blacksmith's Sandboxes beta lists exposed ports and environment snapshots.
262nd of 365, with limits
$3,607 a month
143rd of 365, with limits
$970 a month
5th cheapest of 365
$634 a month
Always-on AI employees
50 machines 24/7 · 2 vCPU / 4 GB
An agent that runs all month needs a machine that keeps running. Both CI runners live for one job, and GitHub ends RunsOn jobs after 5 days. boat has no session cap once auto-stop is off (default 1 h).
140th of 361, with limits
$5,463 a month
116th of 361, with limits
$1,724 a month
6th cheapest of 361
$657 a month
Code interpreter bursts
100 at once · 1 vCPU / 2 GB · 200,000 sessions of 1 min a month
278th of 336, with limits
$488 a month
226th of 336, with limits
$147 a month
189th of 336, with limits
$2,000 a month
Sandboxes for your users
300 at once · 2 vCPU / 4 GB · 30,000 sessions of 23 min a month
Giving each of your users a sandbox means starting machines from your own backend. boat does this by API today; RunsOn starts machines only for GitHub workflows, and Blacksmith's Sandboxes API is a waitlist beta in preview.
263rd of 349, with limits
$1,563 a month
211th of 349, with limits
$585 a month
3rd cheapest of 349
$209 a month
RL rollouts
1,000 at once · 2 vCPU / 4 GB · 1,000,000 sessions of 3 min a month
270th of 349, with limits
$7,488 a month
240th of 349, with limits
$2,714 a month
39th of 349, with limits
$2,000 a month
Agent evals
250 at once · 4 vCPU / 8 GB · 15,000 sessions of 20 min a month
350th of 359, with limits
$1,188 a month
250th of 359, with limits
$341 a month
10th cheapest of 359
$200 a month
CI runners
40 at once · 8 vCPU / 16 GB · 6,000 sessions of 12 min a month
This is what both are built for: Blacksmith is the hands-off pick, and RunsOn suits teams that want jobs in their own AWS account. boat ranks 2nd here and runs CI on full VMs you start by API, within its per-plan start limits.
84th cheapest of 374
$708 a month
56th cheapest of 374
$432 a month
2nd cheapest of 374
$86 a month
24/7 dev boxes
1 machine 24/7 · 4 vCPU / 8 GB
A dev box has to stay up and keep its files. On both, the runner ends with its job; you carry state over with sticky disks, and on RunsOn also with stopped warm pools. A boat sandbox keeps its disk for at least 14 days after a stop.
197th of 371, with limits
$207 a month
175th of 371, with limits
$78 a month
19th cheapest of 371
$26 a month
Computer-use agents
50 at once · 4 vCPU / 8 GB · 20,000 sessions of 15 min a month
Computer-use agents need a screen. Neither CI runner ships one; you set it up: run Xvfb in a Blacksmith job and view it over debug SSH, or install a desktop in your RunsOn image and reach it over SSM or SSH. boat streams a 60 fps Linux desktop with Chrome.
273rd of 367, with limits
$1,088 a month
173rd of 367, with limits
$310 a month
7th cheapest of 367
$200 a month
Browser-use agents
50 at once · 2 vCPU / 4 GB · 30,000 sessions of 10 min a month
188th of 356, with limits
$588 a month
177th of 356, with limits
$224 a month
7th cheapest of 356
$200 a month
Scraping & bots (startup scale)
50 at once · 1 vCPU / 2 GB · 100,000 sessions of 3 min a month
197th of 343, with limits
$719 a month
185th of 343, with limits
$205 a month
6th cheapest of 343
$90 a month
Mass scraping & bots (undetectable)
1,000 at once · 1 vCPU / 2 GB · 3,000,000 sessions of 2 min a month
262nd of 327, with limits
$14,613 a month
124th of 327, with limits
$3,539 a month
73rd of 327, with limits
$1,800 a month
Parallel red teaming
200 at once · 2 vCPU / 4 GB · 5,000 sessions of 30 min a month
115th of 351, with limits
$350 a month
98th of 351, with limits
$157 a month
3rd cheapest of 351
$200 a month
GPU eval jobs
10 at once · 8 vCPU / 32 GB · 500 sessions of 30 min a month
Neither fits in our model, though RunsOn can launch NVIDIA and AMD GPU instances for CI jobs, which Blacksmith and boat do not offer.
Not ranked: misses a requirementNot ranked: misses a requirementNot ranked: misses a requirement
macOS agents
5 at once · 4 vCPU / 8 GB · 1,000 sessions of 20 min a month
Blacksmith has macOS runners on Apple Silicon; RunsOn does not support macOS, and boat runs Linux only.
26th of 35, with limits
$1,508 a month
Not ranked: misses a requirementNot ranked: misses a requirement

Key features

FeatureBlacksmithRunsOnboat
What it isManaged GitHub Actions runners on Blacksmith's own hardware; Sandboxes for agents in waitlist betaSelf-hosted GitHub Actions runners, deployed as one stack in your AWS accountA full VM per agent that you create, stop, fork and delete by API
IsolationFirecracker microVM per job, destroyed with its state when the job endsNew EC2 instance per job, terminated when the job endsFull VM with its own kernel on hosts dedicated to boat (AMD Ryzen 7 9700X and Ryzen 9 9950X); vCPUs are shared
How a machine startsA GitHub Actions job with a Blacksmith runner label; Sandboxes beta (preview SDK) adds Sandbox.create() from your own codeA GitHub Actions job with a runs-on: label; launches paced by your AWS quotasOne API call, SDK or CLI, any time; each create, fork or resume counts against the plan's included starts
How long it livesOne job; Testbox stops after 30 idle minutes by defaultOne job; GitHub ends self-hosted jobs after 5 daysUntil you stop it or auto-stop ends it (default 1 h, up to 30 days, or off on a paid plan)
What survives between runsUp to 5 sticky disk folders per job, evicted after 7 days unused; Sandboxes beta adds pause/resumeSticky EBS disks, snapshotted after a job and restored before the next; stopped warm pools keep dependencies installedThe disk, saved every minute and on stop; memory and running programs are not saved. A stopped sandbox has no compute charge and keeps its disk for at least 14 days
ForkRunners: no fork. Sandboxes beta boots many sandboxes from one environment snapshot, not from a running oneNot documented; every job starts from an AMIFork a running sandbox with its disk, from the last 1-minute snapshot
Nested virtualizationYes on x64 Linux runners, not on ARMKVM on compatible Intel x64 families, not arm64/dev/kvm on every sandbox, with Docker, root and systemd
Operating systemsUbuntu x64 and ARM, Windows Server 2025 (beta), macOS on Apple SiliconLinux x64 and arm64, Windows Server 2022 and 2025; no macOSLinux (x64); Windows programs through Wine
Biggest machine and GPUsUp to 32 vCPU / 128 GB on Ubuntu x64; no GPUsAny EC2 size up to 96 vCPU, plus NVIDIA and AMD GPUsUp to 16 vCPU / 32 GB on the $100 plan and above; CPU only
Where it runsBlacksmith's hardware in Ashburn, Phoenix, Amsterdam and FrankfurtYour own AWS account, in the region you pickManaged by boat on machines in Germany, Finland and France; no US or Asia region, and you cannot pick one
Running many at onceDocs say Blacksmith imposes no concurrency limitsSet by your AWS account's quotas; launches paced at about 2 EC2 calls per second by default100 to 2,000 sandboxes by plan; included starts per minute, hour and day, extra starts $0.03 to $0.07; boat never queues a start
Inbound accessRunners: opt-in debug SSH for the user who triggered the job; no ports or URLs. Sandboxes beta (preview SDK) adds exposed portsSSH from a CIDR range you allow, and SSM; HTTPS previews you set up with your own load balancer or tunnel in AWSHTTPS URLs for up to 50 ports, token-gated by default; SSH; public IPv6; no static IP, and the IP changes at each resume
Desktop and computer useNo desktop stream; you set it up: run Xvfb in the job and view it over debug SSHFull runner images include browsers for headless use; a desktop you set up in your image, reached over SSM or optional SSH1920x1080, 60 fps Linux desktop with Chrome and computer tools
Coding agents and keysIts own [code]smith agent; Testbox CLI (early beta) for local agents; secrets inside the jobBedrock through the instance role; Copilot's coding agent can use a runner; secrets inside the jobSeven agents preinstalled with a prompt, steer and events API; optional paid credentials proxy keeps the real keys of supported providers out
Every feature Battleships tracks (43)
FeatureBlacksmithRunsOnboat
Full VM (own kernel)YesYesYes
Docker insideYesYesYes
Nested virtualizationYesYesYes
RootYesYesYes
SystemdNo documented wayNot publishedYes
Desktop GUINo documented wayNo documented wayYes
BrowserYesYesYes
Computer-use APINo documented wayNo documented wayYes
SSHYesYesYes
Persistent diskYesYesYes
Pause/resumeNo documented wayYesYes
Automatic snapshotsNo documented wayYesYes
Fork/cloneNo documented wayNo documented wayYes
Live fork (no pause)No documented wayNo documented wayYes
Memory forkNo documented wayNo documented wayNo documented way
Custom image (Docker or snapshot)No documented wayYesYes
≥24 h sessionsYesYesYes
Idle auto-stopYesYesNo documented way
Wake on requestNo documented wayYesNo documented way
HTTPS preview URLsNo documented wayNo documented wayYes
Raw TCP inboundNo documented wayNo documented wayYes
Public IPv4No documented wayYesNo documented way
Egress allowlistNo documented wayYesNo documented way
Secret proxyNo documented wayNo documented wayYes
Secret proxy for any APINo documented wayNo documented wayYes
Preinstalled agentsNo documented wayNo documented wayYes
Agent harness APINo documented wayNo documented wayYes
MCP serverNo documented wayNo documented wayNo documented way
GPUNo documented wayYesNo documented way
WindowsYesYesNo documented way
Arm64YesYesNo documented way
Self-hostingNo documented wayYesNo documented way
BYOCNo documented wayYesNo documented way
Open sourceNo documented wayNo documented wayNo documented way
Scoped API keysNot publishedNot publishedYes
Spend limitsNot publishedNot publishedYes
Usage per sandbox (API)YesNo documented wayYes
Audit logsNot publishedNot publishedNo documented way
SSONo documented wayNo documented wayNo documented way
SOC 2YesNo documented wayNo documented way
HIPAANo documented wayNo documented wayNo documented way
EU data residencyYesYesNo documented way
Zero data retentionNot publishedNot publishedYes

Performance

BlacksmithNot measured
RunsOnNot measured
boat28.06 runs/s

Blacksmith and RunsOn are not in the public benchmarks we use, so we cannot say which of the three is faster. RunsOn's speed depends on the EC2 family you pick. For reference, boat did 28.06 runs per second on the V8 Web Tooling Benchmark at 4 vCPU / 8 GB, second of 15 in StarSling's run of 3 October 2026 (run 37073680030). Every boat sandbox in that run ran on an AMD Ryzen 9 9950X host; boat also places sandboxes on Ryzen 7 9700X hosts and does not promise a CPU model.

V8 Web Tooling Benchmark at a 4 vCPU / 8 GB target, runs per second, higher is better. Source: StarSling's run of 3 October 2026 (run 37073680030), median of 5 sandboxes. Providers ran on different hardware in different regions. StarSling gives a shared rank when the run cannot separate two providers. Every boat sandbox in this run ran on an AMD Ryzen 9 9950X host. boat also places some sandboxes on AMD Ryzen 7 9700X hosts, which this run did not measure. Blacksmith and RunsOn are not in the public benchmarks we use, so we cannot say which is faster.

Pricing in five scenarios

Each bill is priced from the providers' public price lists by the open Battleships engine, with the cheapest plan that runs the scenario. Start limits, per-start fees and plan minimums are in the price, boat's included.

ScenarioBlacksmith, a monthRunsOn, a monthboat, a month
One coding agent, workdays
4 vCPU / 8 GB, 50 GB disk, 8 hours a day, 22 days a month, files kept between days
$36 · Pay-as-you-go · with limits$61 · Starter · with limits$20 · $20/mo
One agent that never stops
4 vCPU / 8 GB, 50 GB disk, running 24/7 all month
$207 · Pay-as-you-go · with limits$159 · Starter · with limits$26 · $20/mo
50 agents in parallel
50 machines of 4 vCPU / 8 GB at once, 8-hour sessions, 1,100 sessions a month, Docker inside
$2,364 · Pay-as-you-go · with limits$1,597 · Starter · with limits$317 · $20/mo × 3 seats
Sandboxes for your users
2 vCPU / 4 GB, 30,000 sessions of 10 minutes a month, up to 300 at once
$663 · Pay-as-you-go · with limits$515 · Starter · with limits$200 · $100/mo × 2 seats
1,000 short jobs at once
2 vCPU / 4 GB, 1,000 at once, 3 minutes each, 200,000 jobs a month
$1,488 · Pay-as-you-go · with limits$1,103 · Starter · with limits$8,800 · $2000/mo · with limits

Blacksmith: Blacksmith bills per minute of job time, scaled by vCPU and OS: Ubuntu x64 is $0.004 a minute for 2 vCPU, ARM $0.0025, Windows $0.008 and macOS $0.08 for 6 vCPU, with 3,000 free minutes a month and no plan fee. Sticky disks cost $0.50 per GB a month and a static IP $100 a month. Sandboxes have no published price.

RunsOn: AWS bills the EC2 instances and EBS disks on your own AWS bill, to the second, at spot or on-demand prices with no markup; S3 cache traffic, data transfer and a public IPv4 are extra. RunsOn charges one licence a year per company, from EUR 300 for under 50,000 jobs a month. The prices here use spot rates in us-east-1; spot machines can be interrupted.

RunsOn: Plans cost $20, $100, $500 or $2,000 a month, and the whole plan price comes back as VM time for that month; unused time does not carry over. A card is required, also for the 7-day trial. A 4 vCPU / 8 GB sandbox costs $0.036 an hour, billed per second while it runs. Each plan includes a set number of starts (the $20 plan: 12 a minute, 200 a day, 100 running at once), and extra starts cost $0.03 to $0.07. A stopped sandbox has no compute charge and keeps its disk for at least 14 days; saved snapshots above the included count and the credentials proxy still charge.

Limits in those scenarios

None of the five scenarios fits Blacksmith or RunsOn cleanly, because each is a CI runner: a machine exists only for one GitHub Actions job, and only sticky disk data survives, so packages installed elsewhere reset. RunsOn jobs also end after GitHub's 5-day self-hosted limit, so an agent that never stops needs restarts. Blacksmith's docs say it has no concurrency limit, while RunsOn's ceiling is your AWS account's quotas. On boat the limit for 1,000 short jobs is starts: the $2,000 plan charges for starts above 90 a minute and refuses them above 270, and boat never queues a start.

ScenarioBlacksmithRunsOnboat
One coding agent, workdays
  • different kind of product: CI runner
  • files survive a stop only in sticky disk paths: packages installed elsewhere reset
  • egress price not published
  • different kind of product: CI runner
  • files survive a stop only in the sticky disk: packages installed elsewhere reset
  • disk beyond 30 GiB: price unknown
  • egress price not published
None for this scenario
One agent that never stops
  • different kind of product: CI runner
  • files survive a stop only in sticky disk paths: packages installed elsewhere reset
  • egress price not published
  • different kind of product: CI runner
  • sessions end after 120 h: needs restarts to run 24/7
  • files survive a stop only in the sticky disk: packages installed elsewhere reset
  • disk beyond 30 GiB: price unknown
None for this scenario
50 agents in parallel
  • different kind of product: CI runner
  • files survive a stop only in sticky disk paths: packages installed elsewhere reset
  • egress price not published
  • how many machines can run at once: limit not published, probably fine at 50 (the typical published limit is 50)
  • different kind of product: CI runner
  • files survive a stop only in the sticky disk: packages installed elsewhere reset
  • egress price not published
None for this scenario
Sandboxes for your users
  • different kind of product: CI runner
  • not confirmed: how many machines can run at once (you need 300; the typical published limit is 50)
  • egress price not published
  • different kind of product: CI runner
  • egress price not published
None for this scenario
1,000 short jobs at once
  • different kind of product: CI runner
  • not confirmed: how many machines can run at once (you need 1,000; the typical published limit is 50)
  • egress price not published
  • different kind of product: CI runner
  • egress price not published
  • 2,400 starts/day included, paid above that (needs ~6,667)
  • 600 starts/hour included, paid above that (needs ~1,000)
  • starts above 90/min are paid, and starts above 270/min fail with 429

Which one should you use?

Use Blacksmith when

you want faster GitHub Actions with nothing to run yourself, need macOS runners, or want a big test matrix to start all at once.

  • Nothing to run yourself. Teams that want faster GitHub Actions without owning any cloud setup change one label, and Blacksmith runs the machines.
  • macOS runners. Teams that build or test iOS and macOS apps can run those jobs on Apple Silicon, which RunsOn does not support.
  • No concurrency limit. Big test matrices can start all at once, since Blacksmith's docs say it imposes no concurrency limits.

Use RunsOn when

your CI must run in your own AWS account and region, at AWS cost, or needs GPUs or EC2 machines up to 96 vCPU.

  • Your own AWS account. Teams whose code, secrets and logs must stay in their AWS account keep them there, in any region they pick.
  • AWS prices with no markup. AWS bills the EC2 time at spot or on-demand cost, and RunsOn adds one flat yearly licence from EUR 300.
  • GPUs and very large machines. Jobs that need NVIDIA or AMD GPUs, or EC2 machines up to 96 vCPU, can get them on RunsOn.

Where boat fits

Blacksmith and RunsOn are CI runners: a machine exists only while a GitHub Actions job runs. If what you are really running is agents, you want a machine your own code starts, that keeps its disk, stays up as long as the work needs and can be reached from outside. That is what boat is built for, and it can still run CI jobs on the same VMs.

A machine that stays and keeps its disk

On both, the runner is destroyed when the job ends, and only sticky disk data comes back, so an agent sets up the rest again every run. boat saves the disk every minute, has no session cap once auto-stop is off (default 1 h), and forks running sandboxes; a resume is like a reboot.

Started by your code, not a workflow

Blacksmith runners and RunsOn start machines for GitHub Actions jobs, so serving users or agents means wrapping each one in a workflow run; Blacksmith's Sandboxes SDK is still in preview. boat starts a sandbox with one API call, SDK or CLI from your own backend, within its per-plan start limits.

Reach in: hosting, a desktop, agents

Their runners are reached over SSH or SSM, so previews and computer use need your own tunnels and desktop setup. boat gives token-gated HTTPS URLs on up to 50 ports, a 60 fps desktop and coding agents you drive from outside.

Use boat when you need machines for agents rather than CI jobs: a full VM you start by API, keep with its disk, fork, reach over HTTPS and drive with built-in coding agents. Compare it one to one: boat vs Blacksmith and boat vs RunsOn.

boat's own limits
  • Plans start at $20 a month. The plan price comes back as VM time for that month. Unused time does not carry over. Source
  • Starts (create, fork, resume): the $20 plan includes 12 a minute, 60 an hour and 200 a day. The $2,000 plan includes 90 a minute, 600 an hour and 2,400 a day. Extra starts cost $0.03 to $0.07. A start fails with 429 above 3 times the included starts per minute, or 6 times per hour or per day. Source
  • A stopped sandbox has no compute charge. boat keeps its disk for at least 14 days. A saved snapshot keeps it longer; extra saved snapshots and the AI credentials proxy are paid. Source
  • Auto-stop defaults to 1 hour. You can turn it off on a paid plan. The trial allows 2 hours at most. Source
  • boat saves the disk every minute and at each stop. Memory, running programs and /tmp are not saved: a resume is like a reboot. A machine failure loses writes after the last good snapshot. Source
  • Machines run in Germany, Finland and France only. Snapshot copies can be stored outside the EU. Source
  • Public IPv6 and outbound IPv4. No dedicated IPv4. The IP changes at each resume. Source
  • Hosts are AMD Ryzen 7 9700X and Ryzen 9 9950X. vCPUs are shared. No GPU. You cannot boot your own Docker image. Source

Questions people ask

Is Blacksmith cheaper than RunsOn?

In our CI runner use case RunsOn comes to $432 a month and Blacksmith to $708, because AWS bills spot minutes at cost and RunsOn adds only a yearly licence. Blacksmith includes the machines, caching and upkeep in its per-minute price, so there is no AWS setup to run. boat comes to $86 a month for the same use case.

Which is faster, Blacksmith or RunsOn?

Neither is in the public benchmarks we use, so we cannot say. RunsOn's speed depends on the EC2 family you choose. For reference, boat did 28.06 runs per second on the V8 Web Tooling Benchmark at 4 vCPU / 8 GB in StarSling's run of 3 October 2026.

Do Blacksmith and RunsOn keep files between jobs?

On sticky disks, yes. Blacksmith keeps up to 5 folders per job and evicts them after 7 days unused. RunsOn snapshots sticky EBS disks after a job and restores them before the next, and stopped warm pools keep dependencies installed. The rest of the machine is new every time. boat saves a sandbox's disk every minute and on stop; memory is not saved.

Can Blacksmith or RunsOn run Docker and nested virtualization?

Both run Docker. Blacksmith supports nested virtualization on x64 Linux runners but not ARM, and RunsOn offers KVM on compatible Intel x64 families but not arm64. boat has Docker, root, systemd and /dev/kvm on every sandbox.

Can I run an AI agent on Blacksmith or RunsOn?

Inside GitHub Actions jobs, yes. Blacksmith has a Testbox CLI in early beta and its own [code]smith agent, and its Sandboxes product is in waitlist beta with a preview SDK. RunsOn gives runners Bedrock access and can host Copilot's coding agent. boat starts agent sandboxes by API with seven coding agents preinstalled.

Where does my code run?

Blacksmith runs jobs on its own hardware in Ashburn, Phoenix, Amsterdam and Frankfurt. RunsOn runs in your own AWS account and region, and your code, secrets and logs are never sent to RunsOn. boat runs sandboxes on machines in Germany, Finland and France, and you cannot pick a region; snapshot copies can be stored outside the EU.

Try boat

Install the CLI, then run boat new.

7-day free trial

More comparisons