Blacksmith vs Buildkite
Blacksmith runs your existing GitHub Actions jobs on its own runners, and you switch by changing one label; its Sandboxes product for agents is in a waitlist beta. Buildkite is a whole CI platform with its own pipelines, and its hosted agents are the machines it runs those jobs on.
The companies and products
Blacksmith
Blacksmith is a CI service: it runs your GitHub Actions jobs on its own runners instead of GitHub's, at a lower per-minute price.
You change the runner label in your workflow file, and each job runs in a fresh Firecracker microVM that boots in under 3 seconds and is destroyed when the job ends. It adds Docker layer caching and sticky disks that carry chosen folders from one run to the next, plus a Testbox CLI in early beta that lets a local coding agent run commands inside a CI job. Its Sandboxes product for coding agents is in a waitlist beta: the page shows a Sandbox.create() API and environment snapshots to boot from, and a preview SDK covers exec, files, exposed ports and pause/resume. Sandboxes have no published price yet.
- Kind of product
- Drop-in GitHub Actions runners; jobs start from workflows
- Compliance
- SOC 2 Type 2, GDPR compliant
Buildkite
Buildkite is a CI and software delivery platform, and hosted agents are the build machines it runs for you.
You write Buildkite pipelines, and each job runs on a new virtual machine that Buildkite starts for that job and destroys when it ends. Cache volumes and custom agent images carry files and packages to the next job. Hosted agents run Linux (AMD64 and ARM64) and macOS on Apple M4, and Windows is in public preview. You can also run Buildkite's agent on your own machines. It is a CI service, not an agent sandbox, so this page compares the part where they overlap: a machine where an agent or a job does its work.
Which use cases each one fits
For each use case, Battleships prices every sandbox provider on the same workload and sorts them from the cheapest monthly bill up. “84th cheapest of 374” means 83 of the other 373 providers run that workload, with every feature it needs, for less. “With limits” means the provider runs it but misses something, so it is listed after every full match. Open a use case for its whole ranking and what each provider misses.
| Use case and its workload | Blacksmith: price rank | Buildkite: price rank | boat: price rank |
|---|---|---|---|
| Software factory 100 at once · 4 vCPU / 8 GB · 2,200 sessions of 8 h a month An agent that builds a whole app needs the same machine for hours with its files intact. Both destroy the job machine when the job ends; you keep state in sticky disk folders, or in cache volumes and custom agent images. Blacksmith's Sandboxes beta lists exposed ports and environment snapshots. | 262nd of 365, with limits $3,607 a month | 81st of 365, with limits $11,648 a month | 5th cheapest of 365 $634 a month |
| Always-on AI employees 50 machines 24/7 · 2 vCPU / 4 GB An assistant that lives on its machine all month does not fit a CI job on either, and Buildkite also stops a Linux job after 8 hours (longer on request). boat has no session cap once auto-stop is off (default 1 h). | 140th of 361, with limits $5,463 a month | 349th of 361, with limits $17,564 a month | 6th cheapest of 361 $657 a month |
| Code interpreter bursts 100 at once · 1 vCPU / 2 GB · 200,000 sessions of 1 min a month | 278th of 336, with limits $488 a month | 309th of 336, with limits $1,587 a month | 189th of 336, with limits $2,000 a month |
| Sandboxes for your users 300 at once · 2 vCPU / 4 GB · 30,000 sessions of 23 min a month Giving each of your users a sandbox means starting machines from your own backend with a URL they can reach. boat does this by API today. Buildkite documents no inbound ports, and Blacksmith's Sandboxes API is still a waitlist beta in preview. | 263rd of 349, with limits $1,563 a month | 320th of 349, with limits $4,984 a month | 3rd cheapest of 349 $209 a month |
| RL rollouts 1,000 at once · 2 vCPU / 4 GB · 1,000,000 sessions of 3 min a month | 270th of 349, with limits $7,488 a month | 310th of 349, with limits $23,881 a month | 39th of 349, with limits $2,000 a month |
| Agent evals 250 at once · 4 vCPU / 8 GB · 15,000 sessions of 20 min a month | 350th of 359, with limits $1,188 a month | 322nd of 359, with limits $3,884 a month | 10th cheapest of 359 $200 a month |
| CI runners 40 at once · 8 vCPU / 16 GB · 6,000 sessions of 12 min a month Blacksmith is the easy pick if your CI already lives in GitHub Actions, Buildkite if you want its own pipelines and test tooling. boat fits CI you start by API on full VMs with Docker and /dev/kvm, within its per-plan start limits. | 84th cheapest of 374 $708 a month | 202nd of 374, with limits $2,318 a month | 2nd cheapest of 374 $86 a month |
| 24/7 dev boxes 1 machine 24/7 · 4 vCPU / 8 GB A dev box has to stay up and keep its installed packages. On both services you set it up: sticky disks on Blacksmith, cache volumes or a custom agent image on Buildkite; each job still starts a new machine. boat keeps a stopped sandbox's disk for at least 14 days. | 197th of 371, with limits $207 a month | 357th of 371, with limits $715 a month | 19th cheapest of 371 $26 a month |
| Computer-use agents 50 at once · 4 vCPU / 8 GB · 20,000 sessions of 15 min a month Buildkite's desktop access is for a person who debugs a macOS job, and Blacksmith ships no desktop stream; you run Xvfb in the job and view it over debug SSH. boat streams a 1920x1080, 60 fps Linux desktop with computer tools. | 273rd of 367, with limits $1,088 a month | 119th of 367, with limits $3,500 a month | 7th cheapest of 367 $200 a month |
| Browser-use agents 50 at once · 2 vCPU / 4 GB · 30,000 sessions of 10 min a month | 188th of 356, with limits $588 a month | 210th of 356, with limits $1,772 a month | 7th cheapest of 356 $200 a month |
| Scraping & bots (startup scale) 50 at once · 1 vCPU / 2 GB · 100,000 sessions of 3 min a month | 197th of 343, with limits $719 a month | 216th of 343, with limits $2,354 a month | 6th cheapest of 343 $90 a month |
| Mass scraping & bots (undetectable) 1,000 at once · 1 vCPU / 2 GB · 3,000,000 sessions of 2 min a month | 262nd of 327, with limits $14,613 a month | 313th of 327, with limits $46,814 a month | 73rd of 327, with limits $1,800 a month |
| Parallel red teaming 200 at once · 2 vCPU / 4 GB · 5,000 sessions of 30 min a month | 115th of 351, with limits $350 a month | 143rd of 351, with limits $1,184 a month | 3rd cheapest of 351 $200 a month |
| GPU eval jobs 10 at once · 8 vCPU / 32 GB · 500 sessions of 30 min a month None of the three fits: all hosted shapes on Blacksmith, Buildkite and boat are CPU only. | Not ranked: misses a requirement | Not ranked: misses a requirement | Not ranked: misses a requirement |
| macOS agents 5 at once · 4 vCPU / 8 GB · 1,000 sessions of 20 min a month Both run macOS, Blacksmith on Apple Silicon and Buildkite on Apple M4 with Xcode; boat runs Linux only. Blacksmith ranks 26th here at $1,508 a month and Buildkite 34th at $2,310. | 26th of 35, with limits $1,508 a month | 34th of 35, with limits $2,310 a month | Not ranked: misses a requirement |
Key features
| Feature | Blacksmith | Buildkite | boat |
|---|---|---|---|
| What it is | Drop-in GitHub Actions runners; jobs start from your workflows. Sandboxes for agents in waitlist beta | A CI platform; hosted agents run jobs from Buildkite pipelines | A full VM per agent that you create, stop and start by API, CLI or SDK |
| Isolation | Firecracker microVM per job, destroyed with its state after the job | A virtual machine per job, destroyed after the job; Apple Virtualization framework on macOS | Full VM with its own kernel on hosts dedicated to boat (AMD Ryzen 7 9700X and Ryzen 9 9950X); vCPUs are shared |
| How long a machine lives | Runners: until the job ends; Testbox stops after 30 idle minutes by default | Up to 8 hours per job on Linux and 4 on macOS, longer on request | Until you stop it or auto-stop ends it (default 1 h, up to 30 days, or off on a paid plan) |
| Files after a run | Runners: only sticky disk folders, up to 5 per job, evicted after 7 days unused; Sandboxes beta adds pause/resume | You set up what carries over: cache volume paths, saved after successful jobs, best effort, up to 14 days; or packages baked into a custom agent image | The disk, saved every minute and on stop; memory and /tmp are not saved. A stopped sandbox has no compute charge and keeps its disk for at least 14 days |
| Fork | Runners: no fork. Sandboxes beta boots many sandboxes from one environment snapshot, not from a running one | No VM fork; each job gets its own copy of the last committed cache volume | Fork a running sandbox with its disk, from the last 1-minute snapshot |
| Docker and nested virtualization | Docker; nested virtualization on x64 Linux runners, not on ARM | Docker on Linux; macOS has the Docker CLI, and you connect a daemon you provide; /dev/kvm not documented | Docker, root, systemd and /dev/kvm on every sandbox |
| Reaching the machine | Runners: opt-in debug SSH, only for the user who triggered the job; no ports or URLs. Sandboxes beta (preview SDK) adds exposed ports | No inbound ports documented; a terminal into a running job through Buildkite's UI or bk job ssh; for a URL, you set up an outbound tunnel | HTTPS URLs for up to 50 ports, token-gated by default; SSH; public IPv6; no static IP, and the IP changes at each resume |
| Desktop and computer use | No desktop stream; you set it up: run Xvfb in the job and view it over debug SSH | Browser-based desktop access for a person who debugs a running macOS job; no computer-use API documented | 1920x1080, 60 fps Linux desktop with Chrome and computer tools |
| Coding agents | Its own [code]smith agent from Slack, GitHub or a web UI; Testbox CLI in early beta for local agents | You set it up: install Claude Code in a custom agent image or a step and run it as a pipeline job | Claude Code, Codex and five more preinstalled, with a prompt, steer and events API |
| Keeping API keys out | GitHub Actions secrets and OIDC tokens available inside the job | Anthropic proxy: jobs call it with the job token and Buildkite sets the API key header; other secrets enter the job | Optional paid credentials proxy ($1 per day on, plus $1 per 1,000 requests) for supported model providers and your HTTPS APIs; other secrets still enter the sandbox |
| Operating systems | Ubuntu x64 and ARM, Windows Server 2025 (beta), macOS on Apple Silicon | Linux (AMD64 and ARM64), macOS on Apple M4 with Xcode, Windows Server 2022 in public preview | Linux (x64) only; Windows programs through Wine |
| Biggest machine | Up to 32 vCPU / 128 GB on Ubuntu x64, pay as you go | Linux up to 8 vCPU / 32 GB on Pro and 64 vCPU / 256 GB on Enterprise | Up to 16 vCPU / 32 GB on the $100 plan and above |
| Running at once | Docs say Blacksmith imposes no concurrency limits | Pro shares pools of 256 Linux AMD64 vCPU, 64 ARM64 vCPU and 48 macOS vCPU; jobs above a pool wait in the queue | 100 to 2,000 sandboxes by plan; included starts per minute, hour and day, extra starts $0.03 to $0.07; starts above 3 times the per-minute number fail, and boat never queues a start |
| Regions | US east, US west, Amsterdam and Frankfurt | US East Coast data centers; no region choice | Germany, Finland and France; no US or Asia region, and you cannot pick one |
Every feature Battleships tracks (43)
| Feature | Blacksmith | Buildkite | boat |
|---|---|---|---|
| Full VM (own kernel) | Yes | Yes | Yes |
| Docker inside | Yes | Yes | Yes |
| Nested virtualization | Yes | No documented way | Yes |
| Root | Yes | Not published | Yes |
| Systemd | No documented way | No documented way | Yes |
| Desktop GUI | No documented way | Yes | Yes |
| Browser | Yes | Yes | Yes |
| Computer-use API | No documented way | No documented way | Yes |
| SSH | Yes | Yes | Yes |
| Persistent disk | Yes | Yes | Yes |
| Pause/resume | No documented way | No documented way | Yes |
| Automatic snapshots | No documented way | Yes | Yes |
| Fork/clone | No documented way | Yes | Yes |
| Live fork (no pause) | No documented way | No documented way | Yes |
| Memory fork | No documented way | No documented way | No documented way |
| Custom image (Docker or snapshot) | No documented way | Yes | Yes |
| ≥24 h sessions | Yes | No documented way | Yes |
| Idle auto-stop | Yes | Yes | No documented way |
| Wake on request | No documented way | No documented way | No documented way |
| HTTPS preview URLs | No documented way | No documented way | Yes |
| Raw TCP inbound | No documented way | No documented way | Yes |
| Public IPv4 | No documented way | No documented way | No documented way |
| Egress allowlist | No documented way | No documented way | No documented way |
| Secret proxy | No documented way | Yes | Yes |
| Secret proxy for any API | No documented way | No documented way | Yes |
| Preinstalled agents | No documented way | No documented way | Yes |
| Agent harness API | No documented way | No documented way | Yes |
| MCP server | No documented way | Yes | No documented way |
| GPU | No documented way | No documented way | No documented way |
| Windows | Yes | Yes | No documented way |
| Arm64 | Yes | Yes | No documented way |
| Self-hosting | No documented way | Yes | No documented way |
| BYOC | No documented way | No documented way | No documented way |
| Open source | No documented way | No documented way | No documented way |
| Scoped API keys | Not published | Not published | Yes |
| Spend limits | Not published | Not published | Yes |
| Usage per sandbox (API) | Yes | No documented way | Yes |
| Audit logs | Not published | Not published | No documented way |
| SSO | No documented way | Yes | No documented way |
| SOC 2 | Yes | Yes | No documented way |
| HIPAA | No documented way | No documented way | No documented way |
| EU data residency | Yes | No documented way | No documented way |
| Zero data retention | Not published | Not published | Yes |
Performance
Blacksmith and Buildkite hosted agents are not in the public benchmarks we use, so we cannot say which of the three is faster. For reference, boat did 28.06 runs per second on the V8 Web Tooling Benchmark at 4 vCPU / 8 GB, second of 15 in StarSling's run of 3 October 2026 (run 37073680030). Every boat sandbox in that run ran on an AMD Ryzen 9 9950X host; boat also places sandboxes on Ryzen 7 9700X hosts and does not promise a CPU model.
V8 Web Tooling Benchmark at a 4 vCPU / 8 GB target, runs per second, higher is better. Source: StarSling's run of 3 October 2026 (run 37073680030), median of 5 sandboxes. Providers ran on different hardware in different regions. StarSling gives a shared rank when the run cannot separate two providers. Every boat sandbox in this run ran on an AMD Ryzen 9 9950X host. boat also places some sandboxes on AMD Ryzen 7 9700X hosts, which this run did not measure. Blacksmith and Buildkite are not in the public benchmarks we use, so we cannot say which is faster.
Pricing in five scenarios
Each bill is priced from the providers' public price lists by the open Battleships engine, with the cheapest plan that runs the scenario. Start limits, per-start fees and plan minimums are in the price, boat's included.
| Scenario | Blacksmith, a month | Buildkite, a month | boat, a month |
|---|---|---|---|
| One coding agent, workdays 4 vCPU / 8 GB, 50 GB disk, 8 hours a day, 22 days a month, files kept between days | $36 · Pay-as-you-go · with limits | $166 · Pro · with limits | $20 · $20/mo |
| One agent that never stops 4 vCPU / 8 GB, 50 GB disk, running 24/7 all month | $207 · Pay-as-you-go · with limits | $715 · Pro · with limits | $26 · $20/mo |
| 50 agents in parallel 50 machines of 4 vCPU / 8 GB at once, 8-hour sessions, 1,100 sessions a month, Docker inside | $2,364 · Pay-as-you-go · with limits | $7,677 · Pro · with limits | $317 · $20/mo × 3 seats |
| Sandboxes for your users 2 vCPU / 4 GB, 30,000 sessions of 10 minutes a month, up to 300 at once | $663 · Pay-as-you-go · with limits | $2,204 · Pro · with limits | $200 · $100/mo × 2 seats |
| 1,000 short jobs at once 2 vCPU / 4 GB, 1,000 at once, 3 minutes each, 200,000 jobs a month | $1,488 · Pay-as-you-go · with limits | $4,787 · Pro · with limits | $8,800 · $2000/mo · with limits |
Blacksmith: Blacksmith bills per minute of job time with no seat fee: Ubuntu x64 is $0.004 a minute for 2 vCPU, ARM $0.0025, Windows $0.008 and macOS $0.08 for 6 vCPU, with 3,000 free minutes a month. Sticky disks cost $0.50 per GB a month and a static IP $100 a month. Sandboxes have no published price.
Buildkite: Buildkite Pro costs $30 per active user a month and includes 4,000 Linux AMD64 vCPU minutes. After that, Linux is $0.004 per vCPU minute, Windows $0.008 and macOS $0.02, metered to the second with no minimums. Enterprise pricing is custom.
Buildkite: Plans cost $20, $100, $500 or $2,000 a month, and the whole plan price comes back as VM time for that month; unused time does not carry over. A card is required, also for the 7-day trial. A 4 vCPU / 8 GB sandbox costs $0.036 an hour, billed per second while it runs. Each plan includes a set number of starts (the $20 plan: 12 a minute, 200 a day, 100 running at once), and extra starts cost $0.03 to $0.07. A stopped sandbox has no compute charge and keeps its disk for at least 14 days; saved snapshots above the included count and the credentials proxy still charge.
Limits in those scenarios
None of the agent scenarios fit either service cleanly, because both are CI runners: packages installed outside sticky disks, cache volumes or a custom image reset after each job. Buildkite also stops a Linux job after 8 hours, so an agent that never stops needs restarts, and Pro's pool of 256 Linux vCPU makes 300 user sandboxes or 1,000 short jobs at once wait in a queue. Blacksmith's docs say it has no concurrency limit, but every runner start still has to be a GitHub Actions job. On boat, for the 1,000-job burst, the $2,000 plan charges for starts above 90 a minute and refuses them above 270; boat never queues a start. Auto-stop defaults to 1 hour.
| Scenario | Blacksmith | Buildkite | boat |
|---|---|---|---|
| One coding agent, workdays |
|
| None for this scenario |
| One agent that never stops |
|
| None for this scenario |
| 50 agents in parallel |
|
| None for this scenario |
| Sandboxes for your users |
|
| None for this scenario |
| 1,000 short jobs at once |
|
|
|
Which one should you use?
Use Blacksmith when
your CI runs on GitHub Actions and you want faster jobs by changing one label, with no concurrency cap, US or EU regions, and macOS or Windows runners.
- A drop-in for GitHub Actions. If your CI already lives in GitHub Actions, you keep your workflows and switch by changing one runner label.
- No concurrency cap and no seat fee. For big test matrices, Blacksmith's docs say it sets no concurrency limit, and you pay per job minute with no per-user fee.
- US and EU regions. For teams that want runners near them, it runs in Ashburn, Phoenix, Amsterdam and Frankfurt.
Use Buildkite when
you want a full CI platform with its own pipelines and test tooling, self-hosted agents beside hosted ones, macOS on Apple M4, or Enterprise machines up to 64 vCPU.
- A whole CI platform. For teams that want managed CI outside GitHub: pipelines, test tooling, cache volumes, and self-hosted agents next to hosted ones.
- macOS on Apple M4 with Xcode. For iOS and Mac builds, its Mac hosted agents run current Xcode on Apple M4, up to 12 vCPU and 56 GB.
- Very large build machines. For heavy builds on Enterprise, Linux shapes go up to 64 vCPU and 256 GB, in AMD64 and ARM64.
Where boat fits
Blacksmith and Buildkite are both CI services: a job machine exists only while a job runs, then it is destroyed. If what you are really running is agents that work for hours or days, start from your own backend and need to be reached from outside, boat gives each one a full Linux VM that stays. For plain CI on GitHub Actions or Buildkite pipelines, either of them is the better fit.
A machine that keeps its state
On both services the job machine is destroyed after each job. Only sticky disk folders, cache volume paths or a custom agent image carry over, so an agent sets up the rest again every run. boat saves the disk every minute, lets you fork a running sandbox, and has no session cap once auto-stop is off (default 1 h).
Reach in: URLs, ports and a desktop
Neither service exposes public ports today, only debug SSH or a terminal for a person, so opening what an agent built needs your own tunnel. boat gives token-gated HTTPS URLs on up to 50 ports and a 60 fps desktop for computer use.
Agents you drive from outside, keys kept out
On Blacksmith runners and Buildkite an agent is a job you start from a workflow and read through logs. Buildkite proxies the key for Anthropic. boat ships seven agents with a prompt, steer and events API, and an optional paid credentials proxy for supported model providers and your HTTPS APIs.
Use boat when what you run is agents, not CI jobs: one full Linux VM per agent that you start by API, that keeps its disk, hosts what it builds, shows a desktop and runs coding agents you drive from outside. Compare it one to one: boat vs Blacksmith and boat vs Buildkite.
boat's own limits
- Plans start at $20 a month. The plan price comes back as VM time for that month. Unused time does not carry over. Source
- Starts (create, fork, resume): the $20 plan includes 12 a minute, 60 an hour and 200 a day. The $2,000 plan includes 90 a minute, 600 an hour and 2,400 a day. Extra starts cost $0.03 to $0.07. A start fails with 429 above 3 times the included starts per minute, or 6 times per hour or per day. Source
- A stopped sandbox has no compute charge. boat keeps its disk for at least 14 days. A saved snapshot keeps it longer; extra saved snapshots and the AI credentials proxy are paid. Source
- Auto-stop defaults to 1 hour. You can turn it off on a paid plan. The trial allows 2 hours at most. Source
- boat saves the disk every minute and at each stop. Memory, running programs and /tmp are not saved: a resume is like a reboot. A machine failure loses writes after the last good snapshot. Source
- Machines run in Germany, Finland and France only. Snapshot copies can be stored outside the EU. Source
- Public IPv6 and outbound IPv4. No dedicated IPv4. The IP changes at each resume. Source
- Hosts are AMD Ryzen 7 9700X and Ryzen 9 9950X. vCPUs are shared. No GPU. You cannot boot your own Docker image. Source
Questions people ask
Is Blacksmith cheaper than Buildkite?
In every scenario we priced, Blacksmith comes out lower, for example $207 against $715 a month for one machine running all month and $2,364 against $7,677 for 50 machines in parallel. Blacksmith is priced on ARM runners and Buildkite on its 4 vCPU / 16 GB Linux shape. Neither fits these agent scenarios cleanly; boat comes to $26 and $317 for the same two.
Which is faster, Blacksmith or Buildkite?
Neither is in the public benchmarks we use, so we cannot compare them. For reference, boat did 28.06 runs per second on the V8 Web Tooling Benchmark at 4 vCPU / 8 GB in StarSling's run of 3 October 2026.
Can Blacksmith and Buildkite run Docker?
Yes on Linux. Blacksmith also supports nested virtualization on x64 Linux runners but not on ARM. Buildkite's macOS agents have the Docker CLI, and you connect a daemon you provide. boat has Docker, root, systemd and /dev/kvm on every sandbox.
How long can a job run on each?
A Buildkite hosted agent runs up to 8 hours per job on Linux and 4 on macOS, longer on request. A Blacksmith runner lives until its GitHub Actions job ends, and Testbox stops after 30 idle minutes by default. On boat, auto-stop defaults to 1 hour; on a paid plan you can set up to 30 days or turn it off.
Can I run an AI coding agent on Blacksmith or Buildkite?
On Buildkite you install Claude Code in a pipeline step, and its Anthropic proxy lets it call the API with the job token. Blacksmith has a Testbox CLI in early beta for local agents and its own [code]smith agent; its Sandboxes product is in waitlist beta with a preview SDK. On boat, coding agents come preinstalled and you prompt and steer them through an API.
Do Blacksmith and Buildkite run in the EU?
Blacksmith runs in Amsterdam and Frankfurt as well as Ashburn and Phoenix. Buildkite hosted agents run in US East Coast data centers. boat runs sandboxes on machines in Germany, Finland and France, and you cannot pick a region; snapshot copies can be stored outside the EU.
Try boat
Install the CLI, then run boat new.
7-day free trial