Blacksmith vs Codemagic
Blacksmith is a drop-in replacement for GitHub Actions runners: you change one label and your jobs run on its machines. Codemagic is its own CI service for mobile apps, with Macs that have Xcode ready and built-in publishing to the app stores.
The companies and products
Blacksmith
Blacksmith is a CI service: it runs your GitHub Actions jobs on its own faster, cheaper runners instead of GitHub's.
You change the runner label in your workflow file, and each job runs in a fresh Firecracker microVM that boots in under 3 seconds and is destroyed when the job ends. It adds Docker layer caching and sticky disks that carry chosen folders from one run to the next, plus a Testbox CLI in early beta that lets a local coding agent run commands inside a CI job. It is built for teams that want quicker, cheaper GitHub Actions, not as a general sandbox you drive by API; a separate Sandboxes product is marked coming soon.
Codemagic
Codemagic is a cloud CI/CD service built for mobile developers: it builds, tests and publishes iOS, Android, Flutter and React Native apps.
You describe a workflow in codemagic.yaml, and each build runs on a new virtual machine: an Apple silicon Mac mini with Xcode, an Ubuntu Linux machine, or a Windows machine. The machine is destroyed when the build finishes, and the result can go straight to App Store Connect, Google Play and other stores. It is built for app teams that want builds and code signing handled for them; it is a different kind of product from an agent sandbox, and the two overlap only where an agent needs a machine for a short, scripted job.
Which use cases each one fits
For each use case, Battleships prices every sandbox provider on the same workload and sorts them from the cheapest monthly bill up. “78th cheapest of 377” means 77 of the other 376 providers run that workload, with every feature it needs, for less. “With limits” means the provider runs it but misses something, so it is listed after every full match. Open a use case for its whole ranking and what each provider misses.
| Use case and its workload | Blacksmith: price rank | Codemagic: price rank | boat: price rank |
|---|---|---|---|
| Software factory 100 at once · 4 vCPU / 8 GB · 2,200 sessions of 8 h a month Agents that build and run whole apps need hosting, a kept disk and forks, which boat has; Blacksmith and Codemagic only run the CI part of that loop. | 248th of 367, with limits $3,607 a month | 362nd of 367, with limits $32,718 a month | 3rd cheapest of 367 $634 a month |
| Always-on AI employees 50 machines 24/7 · 2 vCPU / 4 GB An assistant that runs all month needs a machine that never ends; Blacksmith machines end with the job and Codemagic builds stop within 120 minutes, and boat has no session cap on paid plans. | 78th of 366, with limits $5,463 a month | 364th of 366, with limits $98,550 a month | 5th cheapest of 366 $657 a month |
| Code interpreter bursts 100 at once · 1 vCPU / 2 GB · 200,000 sessions of 1 min a month | 281st of 345, with limits $488 a month | 337th of 345, with limits $9,147 a month | 186th of 345, with limits $2,000 a month |
| Sandboxes for your users 300 at once · 2 vCPU / 4 GB · 30,000 sessions of 23 min a month Giving each of your users a sandbox means starting hundreds of machines from your own backend; Blacksmith starts them only from GitHub workflows, Codemagic sells build slots one at a time, and boat does it by API. | 268th of 361, with limits $1,563 a month | 349th of 361, with limits $28,497 a month | Cheapest of 361 $209 a month |
| RL rollouts 1,000 at once · 2 vCPU / 4 GB · 1,000,000 sessions of 3 min a month | 276th of 360, with limits $7,488 a month | 336th of 360, with limits $135 a month | 34th of 360, with limits $2,000 a month |
| Agent evals 250 at once · 4 vCPU / 8 GB · 15,000 sessions of 20 min a month | 354th of 364, with limits $1,188 a month | 363rd of 364, with limits $10,947 a month | 10th cheapest of 364 $200 a month |
| CI runners 40 at once · 8 vCPU / 16 GB · 6,000 sessions of 12 min a month Blacksmith is a full match here at $708 a month against $3,387 a month for Codemagic, whose only Linux size is 8 vCPU / 32 GB; boat also fits at $86 a month, with a disk that stays warm between jobs. | 78th cheapest of 377 $708 a month | 207th of 377, with limits $3,387 a month | 4th cheapest of 377 $86 a month |
| 24/7 dev boxes 1 machine 24/7 · 4 vCPU / 8 GB A dev box has to keep its files overnight; Blacksmith keeps only sticky disk folders and Codemagic keeps only dependency caches, while a boat machine keeps its whole disk when stopped. | 180th of 368, with limits $207 a month | 191st of 368, with limits $333 a month | 18th cheapest of 368 $26 a month |
| Computer-use agents 50 at once · 4 vCPU / 8 GB · 20,000 sessions of 15 min a month Codemagic scores better than Blacksmith because a person can open VNC or RDP on a build machine, but neither has a computer-use API; boat streams a 60fps desktop an agent can drive. | 243rd of 369, with limits $1,088 a month | 71st of 369, with limits $10,047 a month | 6th cheapest of 369 $200 a month |
| Browser-use agents 50 at once · 2 vCPU / 4 GB · 30,000 sessions of 10 min a month | 131st of 367, with limits $588 a month | 151st of 367, with limits $10,947 a month | 7th cheapest of 367 $200 a month |
| Scraping & bots (startup scale) 50 at once · 1 vCPU / 2 GB · 100,000 sessions of 3 min a month | 130th of 346, with limits $719 a month | 148th of 346, with limits $13,310 a month | 6th cheapest of 346 $90 a month |
| Mass scraping & bots (undetectable) 1,000 at once · 1 vCPU / 2 GB · 3,000,000 sessions of 2 min a month | 262nd of 338, with limits $14,613 a month | 337th of 338, with limits $263 a month | 71st of 338, with limits $1,800 a month |
| Parallel red teaming 200 at once · 2 vCPU / 4 GB · 5,000 sessions of 30 min a month | 64th of 360, with limits $350 a month | 95th of 360, with limits $6,672 a month | 2nd cheapest of 360 $200 a month |
| GPU eval jobs 10 at once · 8 vCPU / 32 GB · 500 sessions of 30 min a month | Can't run it | Can't run it | Can't run it |
| macOS agents 5 at once · 4 vCPU / 8 GB · 1,000 sessions of 20 min a month Both run macOS on Apple silicon: Codemagic adds Xcode, iOS simulators and store publishing, Blacksmith runs macOS jobs inside GitHub Actions, and boat runs Linux only. | 28th of 36, with limits $1,508 a month | 32nd of 36, with limits $1,952 a month | Can't run it |
Key features
| Feature | Blacksmith | Codemagic | boat |
|---|---|---|---|
| What it is | Drop-in GitHub Actions runners; each machine lives for one workflow job | Mobile CI/CD service; builds start from codemagic.yaml or its API | A full VM per agent that you create, stop and start by API |
| How long a machine lives | Until the job ends, then the microVM is destroyed | 120 minutes per build at most on Free, Pay as you go and Fixed price; custom on Enterprise | Until you stop it; no session cap on paid plans |
| What survives between runs | Sticky disk folders, up to 5 per job, evicted after 7 days unused | Per-workflow dependency caches only (3 GB on Free, 10 GB on paid plans) | The whole disk, saved every minute and on stop; a stopped sandbox costs nothing |
| Fork | No; every machine starts fresh from a job | No; every build starts from a stock image | Fork a running sandbox with its disk |
| Docker and nested virtualization | Docker; nested virtualization on x64 Linux runners, not on ARM | Docker and Android emulators on Linux; no nested virtualization on macOS | Docker, root, systemd and /dev/kvm on every machine |
| Operating systems | Ubuntu x64 and ARM, Windows Server 2025 (beta), macOS on Apple Silicon | macOS on Apple silicon, Linux (x64, arm64 on request) and Windows | Linux (x64) only |
| Macs for iOS work | macOS runners on Apple Silicon, used from GitHub Actions | Mac mini M2 and M4 with Xcode and iOS simulator runtimes preinstalled | None |
| App store publishing | Not built in; whatever your workflow steps do | Built in: code signing and publishing to App Store Connect, TestFlight and Google Play | Not built in; you script it yourself |
| Builds at once | Docs say there are no concurrency limits | One build at a time on Pay as you go, $49 a month for each extra; fixed plans include 3 | Plans start at 100 running sandboxes |
| Machine sizes | Linux runners up to 32 vCPU / 128 GB on Ubuntu x64 | Fixed sizes: Linux X2 at 8 vCPU / 32 GB, Mac mini M2 and M4 | Up to 16 vCPU / 32 GB on the $100 plan and above |
| Reaching the machine | Opt-in debug SSH, only for the user who triggered the job; no ports or URLs | Temporary SSH, VNC or RDP through a gateway, turned on per build | HTTPS URLs on up to 50 ports, token-protected by default; raw TCP; SSH |
| Desktop and computer use | Headless runners | VNC or RDP for a person to debug a build; no computer-use API | 60fps Linux desktop with Chromium and computer use |
| Coding agents | Its own [code]smith agent from Slack, GitHub or a web UI; Testbox CLI (early beta) for local agents | codemagic-init helps agents on your own computer write workflows; nothing runs agents on build machines | Seven agents preinstalled, with a prompt, steer and events API |
| Regions | US east, US west, Amsterdam and Frankfurt | US data centers; dedicated macOS hosts in the EU on request | EU only (Germany, Finland, France) |
Every feature Battleships tracks (43)
| Feature | Blacksmith | Codemagic | boat |
|---|---|---|---|
| Full VM (own kernel) | Yes | Yes | Yes |
| Docker inside | Yes | Yes | Yes |
| Nested virtualization | Yes | Yes | Yes |
| Root | Yes | Not published | Yes |
| Systemd | No | No | Yes |
| Desktop GUI | No | Yes | Yes |
| Browser | Yes | Yes | Yes |
| Computer-use API | No | No | Yes |
| SSH | Yes | Yes | Yes |
| Persistent disk | Yes | No | Yes |
| Pause/resume | No | No | Yes |
| Automatic snapshots | No | No | Yes |
| Fork/clone | No | No | Yes |
| Live fork (no pause) | No | No | Yes |
| Memory fork | No | No | No |
| Custom image (Docker or snapshot) | No | No | Yes |
| ≥24 h sessions | Yes | No | Yes |
| Idle auto-stop | Yes | Yes | No |
| Wake on request | No | No | No |
| HTTPS preview URLs | No | No | Yes |
| Raw TCP inbound | No | No | Yes |
| Public IPv4 | No | No | No |
| Egress allowlist | No | No | No |
| Secret proxy | No | No | Yes |
| Secret proxy for any API | No | No | Yes |
| Preinstalled agents | No | No | Yes |
| Agent harness API | No | No | Yes |
| MCP server | No | No | No |
| GPU | No | No | No |
| Windows | Yes | Yes | No |
| Arm64 | Yes | Yes | No |
| Self-hosting | No | No | No |
| BYOC | No | No | No |
| Open source | No | No | No |
| Scoped API keys | Not published | Not published | Yes |
| Spend limits | Not published | Not published | Yes |
| Usage per sandbox (API) | Yes | No | Yes |
| Audit logs | Not published | Yes | No |
| SSO | No | Yes | No |
| SOC 2 | Yes | Yes | No |
| HIPAA | No | No | No |
| EU data residency | Yes | Yes | Yes |
| Zero data retention | Not published | Not published | Yes |
Performance
There is no comparable measurement for Blacksmith or Codemagic: neither was part of the StarSling hpc-sandbox-benchmarks run. boat did 26.69 runs per second on its Node.js loop at 4 vCPU / 8 GB, on dedicated AMD Ryzen 9 9950X hosts. Codemagic also sells only fixed sizes, so it has no 4 vCPU / 8 GB machine to set beside it.
Node.js build-and-test loop at 4 vCPU / 8 GB, higher is better. Measured with the open hpc-sandbox-benchmarks suite. Blacksmith and Codemagic have no published run in this suite yet, so there is no like-for-like number to compare.
Pricing in five scenarios
Each bill is priced from the providers' public price lists by the open Battleships engine, with the cheapest plan that runs the scenario.
| Scenario | Blacksmith, a month | Codemagic, a month | boat, a month |
|---|---|---|---|
| One coding agent, workdays 4 vCPU / 8 GB, 50 GB disk, 8 hours a day, 22 days a month, files kept between days | $36 · Pay-as-you-go · with limits | $333 · Free (individual) · with limits | $20 · $20/mo |
| One agent that never stops 4 vCPU / 8 GB, 50 GB disk, running 24/7 all month | $207 · Pay-as-you-go · with limits | $333 · Free (individual) · with limits | $26 · $20/mo |
| 50 agents in parallel 50 machines of 4 vCPU / 8 GB at once, 8-hour sessions, 1,100 sessions a month, Docker inside | $2,364 · Pay-as-you-go · with limits | $21,531 · Pay as you go (team, 4 concurrent) · with limits | $317 · $20/mo × 3 seats |
| Sandboxes for your users 2 vCPU / 4 GB, 30,000 sessions of 10 minutes a month, up to 300 at once | $663 · Pay-as-you-go · with limits | $12,297 · Pay as you go (team, 4 concurrent) · with limits | $200 · $100/mo × 2 seats |
| 1,000 short jobs at once 2 vCPU / 4 GB, 1,000 at once, 3 minutes each, 200,000 jobs a month | $1,488 · Pay-as-you-go · with limits | $27,147 · Pay as you go (team, 4 concurrent) · with limits | $2,000 · $2000/mo · with limits |
Blacksmith: Blacksmith bills per minute of job time, scaled by vCPU and OS: Ubuntu x64 is $0.004 a minute for 2 vCPU, ARM $0.0025, Windows $0.008 and macOS $0.08 for 6 vCPU, with 3,000 free minutes a month. Sticky disks cost $0.50 per GB a month.
Codemagic: Codemagic bills build minutes: $0.095 a minute on a Mac mini M2, $0.114 on an M4, and $0.045 on Linux X2 or Windows, with 500 free M2 minutes a month for individuals. Each extra build at once costs $49 a month, and fixed annual plans start at $3,990 a year for unlimited minutes on 3 builds at once.
Codemagic: Plans start at $20 a month, and the whole fee is usage credit spent per second while a sandbox runs. A stopped sandbox costs nothing and keeps its disk.
Limits in those scenarios
Neither Blacksmith nor Codemagic fits any of the five agent scenarios, because both are CI services. On Blacksmith only sticky disk folders survive between runs, so packages installed anywhere else are gone; its prices here use ARM runners. Codemagic caps each build at 2 hours, keeps no disk, and its smallest Linux machine is 8 vCPU / 32 GB, priced here with 4 builds at once. boat fits four of the five; for 1,000 short jobs at once its $2,000 plan queues starts above 90 per minute.
| Scenario | Blacksmith | Codemagic | boat |
|---|---|---|---|
| One coding agent, workdays |
|
| None that bite |
| One agent that never stops |
|
| None that bite |
| 50 agents in parallel |
|
| None that bite |
| Sandboxes for your users |
|
| None that bite |
| 1,000 short jobs at once |
|
|
|
Which one should you use?
Use Blacksmith when
your CI already runs on GitHub Actions and you want faster jobs, no concurrency limit and Linux, Windows or macOS runners by changing one label.
- A drop-in for GitHub Actions. If your CI already lives in GitHub Actions, you keep your workflows, change one runner label, and get Docker layer caching and sticky disks.
- No limit on jobs at once. Teams with big test matrices can start every job at once, since Blacksmith's docs say it imposes no concurrency limits.
- Bigger Linux machines and ARM. Heavy Linux builds can use runners up to 32 vCPU and 128 GB, and ARM runners for cheaper per-minute jobs.
Use Codemagic when
you build, sign and publish iOS, Android or Flutter apps and want Macs with Xcode and store publishing handled for you.
- Macs with Xcode ready. Teams building iOS or macOS apps get Mac mini M2 and M4 machines with current Xcode and iOS simulators preinstalled.
- A full mobile release pipeline. App teams can build Flutter, React Native and native apps, sign them and publish to App Store Connect, TestFlight and Google Play.
- Not tied to GitHub Actions. Builds start from a codemagic.yaml file or Codemagic's API, so a mobile team does not need GitHub Actions workflows to use it.
Where boat fits
Blacksmith and Codemagic are CI services: a machine exists for one job or build and is then destroyed. If what you are really running is agents, which need a machine that stays up, keeps its files and can be reached from outside, look at boat. It gives each agent a full Linux VM with its own kernel, Docker, systemd and /dev/kvm, started by API.
A machine that outlives the job
Blacksmith destroys the microVM when the job ends and keeps only sticky disk folders; Codemagic stops builds within 120 minutes and deletes the machine. A boat sandbox has no session cap on paid plans, saves its disk every minute, and can be forked.
Reach what the agent built
Blacksmith runners have no ports or URLs, and Codemagic machines sit on a private network, reachable by SSH or VNC for 10 minutes after a build. boat gives HTTPS URLs on 50 ports, raw TCP and a 60fps desktop.
Agents driven from outside, keys kept out
On both, an agent would be a workflow step you cannot steer, with real keys inside the job as secrets or environment variables. boat preinstalls seven agents you drive from outside by API, and its credentials proxy keeps keys for any HTTPS API out of the machine.
Use boat when what you are running is agents: a full Linux VM you start by API, that keeps its disk, hosts what it builds, shows a desktop and runs coding agents you drive from outside. Compare it one to one: boat vs Blacksmith and boat vs Codemagic.
Questions people ask
Is Blacksmith cheaper than Codemagic?
Per minute, usually yes: a 6 vCPU macOS runner on Blacksmith is $0.08 a minute, and a Codemagic Mac mini M2 is $0.095. In our agent scenarios Blacksmith comes to $207 against $333 a month for one always-on machine, and $2,364 against $21,531 a month for 50 agents, though neither product fits those jobs cleanly.
Which is faster, Blacksmith or Codemagic?
We have no comparable measurement for either: neither was part of the CPU benchmark run we use. boat did 26.69 runs per second on that benchmark at 4 vCPU / 8 GB.
Can Blacksmith and Codemagic run Docker?
Yes, both run Docker. Blacksmith supports nested virtualization on x64 Linux runners but not on ARM. Codemagic runs Android emulators on Linux machines, but not on macOS, which has no nested virtualization.
Which is better for iOS builds?
Codemagic is built for it: Mac minis with Xcode and iOS simulators preinstalled, code signing and publishing to App Store Connect and TestFlight. Blacksmith also has macOS runners on Apple Silicon for iOS jobs you already run in GitHub Actions. boat runs Linux only.
Do Blacksmith or Codemagic keep files between builds?
Only in part. Blacksmith keeps up to 5 sticky disk folders per job, evicted after 7 days unused, and Codemagic keeps per-workflow dependency caches. In both, the machine itself is destroyed after each job, while boat saves a sandbox's whole disk every minute and when it stops.
Can I run AI coding agents on Blacksmith or Codemagic?
Only partly. Blacksmith's Testbox CLI, in early beta, lets a local agent run commands inside a CI job, and its Sandboxes product is marked coming soon. On Codemagic an agent can only be a step inside a build that stops within 120 minutes.
Try boat
Install the CLI, then run boat new.
7-day free trial