Blacksmith vs Buildkite

Blacksmith runs your existing GitHub Actions jobs on its own runners, and you switch by changing one label; its Sandboxes product for agents is in a waitlist beta. Buildkite is a whole CI platform with its own pipelines, and its hosted agents are the machines it runs those jobs on.

The companies and products

Blacksmith

Blacksmith is a CI service: it runs your GitHub Actions jobs on its own runners instead of GitHub's, at a lower per-minute price.

You change the runner label in your workflow file, and each job runs in a fresh Firecracker microVM that boots in under 3 seconds and is destroyed when the job ends. It adds Docker layer caching and sticky disks that carry chosen folders from one run to the next, plus a Testbox CLI in early beta that lets a local coding agent run commands inside a CI job. Its Sandboxes product for coding agents is in a waitlist beta: the page shows a Sandbox.create() API and environment snapshots to boot from, and a preview SDK covers exec, files, exposed ports and pause/resume. Sandboxes have no published price yet.

Buildkite

Buildkite is a CI and software delivery platform, and hosted agents are the build machines it runs for you.

You write Buildkite pipelines, and each job runs on a new virtual machine that Buildkite starts for that job and destroys when it ends. Cache volumes and custom agent images carry files and packages to the next job. Hosted agents run Linux (AMD64 and ARM64) and macOS on Apple M4, and Windows is in public preview. You can also run Buildkite's agent on your own machines. It is a CI service, not an agent sandbox, so this page compares the part where they overlap: a machine where an agent or a job does its work.

Which use cases each one fits

For each use case, Battleships prices every sandbox provider on the same workload and sorts them from the cheapest monthly bill up. “84th cheapest of 374” means 83 of the other 373 providers run that workload, with every feature it needs, for less. “With limits” means the provider runs it but misses something, so it is listed after every full match. Open a use case for its whole ranking and what each provider misses.

Use case and its workloadBlacksmith: price rankBuildkite: price rankboat: price rank
Software factory
100 at once · 4 vCPU / 8 GB · 2,200 sessions of 8 h a month
An agent that builds a whole app needs the same machine for hours with its files intact. Both destroy the job machine when the job ends; you keep state in sticky disk folders, or in cache volumes and custom agent images. Blacksmith's Sandboxes beta lists exposed ports and environment snapshots.
262nd of 365, with limits
$3,607 a month
81st of 365, with limits
$11,648 a month
5th cheapest of 365
$634 a month
Always-on AI employees
50 machines 24/7 · 2 vCPU / 4 GB
An assistant that lives on its machine all month does not fit a CI job on either, and Buildkite also stops a Linux job after 8 hours (longer on request). boat has no session cap once auto-stop is off (default 1 h).
140th of 361, with limits
$5,463 a month
349th of 361, with limits
$17,564 a month
6th cheapest of 361
$657 a month
Code interpreter bursts
100 at once · 1 vCPU / 2 GB · 200,000 sessions of 1 min a month
278th of 336, with limits
$488 a month
309th of 336, with limits
$1,587 a month
189th of 336, with limits
$2,000 a month
Sandboxes for your users
300 at once · 2 vCPU / 4 GB · 30,000 sessions of 23 min a month
Giving each of your users a sandbox means starting machines from your own backend with a URL they can reach. boat does this by API today. Buildkite documents no inbound ports, and Blacksmith's Sandboxes API is still a waitlist beta in preview.
263rd of 349, with limits
$1,563 a month
320th of 349, with limits
$4,984 a month
3rd cheapest of 349
$209 a month
RL rollouts
1,000 at once · 2 vCPU / 4 GB · 1,000,000 sessions of 3 min a month
270th of 349, with limits
$7,488 a month
310th of 349, with limits
$23,881 a month
39th of 349, with limits
$2,000 a month
Agent evals
250 at once · 4 vCPU / 8 GB · 15,000 sessions of 20 min a month
350th of 359, with limits
$1,188 a month
322nd of 359, with limits
$3,884 a month
10th cheapest of 359
$200 a month
CI runners
40 at once · 8 vCPU / 16 GB · 6,000 sessions of 12 min a month
Blacksmith is the easy pick if your CI already lives in GitHub Actions, Buildkite if you want its own pipelines and test tooling. boat fits CI you start by API on full VMs with Docker and /dev/kvm, within its per-plan start limits.
84th cheapest of 374
$708 a month
202nd of 374, with limits
$2,318 a month
2nd cheapest of 374
$86 a month
24/7 dev boxes
1 machine 24/7 · 4 vCPU / 8 GB
A dev box has to stay up and keep its installed packages. On both services you set it up: sticky disks on Blacksmith, cache volumes or a custom agent image on Buildkite; each job still starts a new machine. boat keeps a stopped sandbox's disk for at least 14 days.
197th of 371, with limits
$207 a month
357th of 371, with limits
$715 a month
19th cheapest of 371
$26 a month
Computer-use agents
50 at once · 4 vCPU / 8 GB · 20,000 sessions of 15 min a month
Buildkite's desktop access is for a person who debugs a macOS job, and Blacksmith ships no desktop stream; you run Xvfb in the job and view it over debug SSH. boat streams a 1920x1080, 60 fps Linux desktop with computer tools.
273rd of 367, with limits
$1,088 a month
119th of 367, with limits
$3,500 a month
7th cheapest of 367
$200 a month
Browser-use agents
50 at once · 2 vCPU / 4 GB · 30,000 sessions of 10 min a month
188th of 356, with limits
$588 a month
210th of 356, with limits
$1,772 a month
7th cheapest of 356
$200 a month
Scraping & bots (startup scale)
50 at once · 1 vCPU / 2 GB · 100,000 sessions of 3 min a month
197th of 343, with limits
$719 a month
216th of 343, with limits
$2,354 a month
6th cheapest of 343
$90 a month
Mass scraping & bots (undetectable)
1,000 at once · 1 vCPU / 2 GB · 3,000,000 sessions of 2 min a month
262nd of 327, with limits
$14,613 a month
313th of 327, with limits
$46,814 a month
73rd of 327, with limits
$1,800 a month
Parallel red teaming
200 at once · 2 vCPU / 4 GB · 5,000 sessions of 30 min a month
115th of 351, with limits
$350 a month
143rd of 351, with limits
$1,184 a month
3rd cheapest of 351
$200 a month
GPU eval jobs
10 at once · 8 vCPU / 32 GB · 500 sessions of 30 min a month
None of the three fits: all hosted shapes on Blacksmith, Buildkite and boat are CPU only.
Not ranked: misses a requirementNot ranked: misses a requirementNot ranked: misses a requirement
macOS agents
5 at once · 4 vCPU / 8 GB · 1,000 sessions of 20 min a month
Both run macOS, Blacksmith on Apple Silicon and Buildkite on Apple M4 with Xcode; boat runs Linux only. Blacksmith ranks 26th here at $1,508 a month and Buildkite 34th at $2,310.
26th of 35, with limits
$1,508 a month
34th of 35, with limits
$2,310 a month
Not ranked: misses a requirement

Key features

FeatureBlacksmithBuildkiteboat
What it isDrop-in GitHub Actions runners; jobs start from your workflows. Sandboxes for agents in waitlist betaA CI platform; hosted agents run jobs from Buildkite pipelinesA full VM per agent that you create, stop and start by API, CLI or SDK
IsolationFirecracker microVM per job, destroyed with its state after the jobA virtual machine per job, destroyed after the job; Apple Virtualization framework on macOSFull VM with its own kernel on hosts dedicated to boat (AMD Ryzen 7 9700X and Ryzen 9 9950X); vCPUs are shared
How long a machine livesRunners: until the job ends; Testbox stops after 30 idle minutes by defaultUp to 8 hours per job on Linux and 4 on macOS, longer on requestUntil you stop it or auto-stop ends it (default 1 h, up to 30 days, or off on a paid plan)
Files after a runRunners: only sticky disk folders, up to 5 per job, evicted after 7 days unused; Sandboxes beta adds pause/resumeYou set up what carries over: cache volume paths, saved after successful jobs, best effort, up to 14 days; or packages baked into a custom agent imageThe disk, saved every minute and on stop; memory and /tmp are not saved. A stopped sandbox has no compute charge and keeps its disk for at least 14 days
ForkRunners: no fork. Sandboxes beta boots many sandboxes from one environment snapshot, not from a running oneNo VM fork; each job gets its own copy of the last committed cache volumeFork a running sandbox with its disk, from the last 1-minute snapshot
Docker and nested virtualizationDocker; nested virtualization on x64 Linux runners, not on ARMDocker on Linux; macOS has the Docker CLI, and you connect a daemon you provide; /dev/kvm not documentedDocker, root, systemd and /dev/kvm on every sandbox
Reaching the machineRunners: opt-in debug SSH, only for the user who triggered the job; no ports or URLs. Sandboxes beta (preview SDK) adds exposed portsNo inbound ports documented; a terminal into a running job through Buildkite's UI or bk job ssh; for a URL, you set up an outbound tunnelHTTPS URLs for up to 50 ports, token-gated by default; SSH; public IPv6; no static IP, and the IP changes at each resume
Desktop and computer useNo desktop stream; you set it up: run Xvfb in the job and view it over debug SSHBrowser-based desktop access for a person who debugs a running macOS job; no computer-use API documented1920x1080, 60 fps Linux desktop with Chrome and computer tools
Coding agentsIts own [code]smith agent from Slack, GitHub or a web UI; Testbox CLI in early beta for local agentsYou set it up: install Claude Code in a custom agent image or a step and run it as a pipeline jobClaude Code, Codex and five more preinstalled, with a prompt, steer and events API
Keeping API keys outGitHub Actions secrets and OIDC tokens available inside the jobAnthropic proxy: jobs call it with the job token and Buildkite sets the API key header; other secrets enter the jobOptional paid credentials proxy ($1 per day on, plus $1 per 1,000 requests) for supported model providers and your HTTPS APIs; other secrets still enter the sandbox
Operating systemsUbuntu x64 and ARM, Windows Server 2025 (beta), macOS on Apple SiliconLinux (AMD64 and ARM64), macOS on Apple M4 with Xcode, Windows Server 2022 in public previewLinux (x64) only; Windows programs through Wine
Biggest machineUp to 32 vCPU / 128 GB on Ubuntu x64, pay as you goLinux up to 8 vCPU / 32 GB on Pro and 64 vCPU / 256 GB on EnterpriseUp to 16 vCPU / 32 GB on the $100 plan and above
Running at onceDocs say Blacksmith imposes no concurrency limitsPro shares pools of 256 Linux AMD64 vCPU, 64 ARM64 vCPU and 48 macOS vCPU; jobs above a pool wait in the queue100 to 2,000 sandboxes by plan; included starts per minute, hour and day, extra starts $0.03 to $0.07; starts above 3 times the per-minute number fail, and boat never queues a start
RegionsUS east, US west, Amsterdam and FrankfurtUS East Coast data centers; no region choiceGermany, Finland and France; no US or Asia region, and you cannot pick one
Every feature Battleships tracks (43)
FeatureBlacksmithBuildkiteboat
Full VM (own kernel)YesYesYes
Docker insideYesYesYes
Nested virtualizationYesNo documented wayYes
RootYesNot publishedYes
SystemdNo documented wayNo documented wayYes
Desktop GUINo documented wayYesYes
BrowserYesYesYes
Computer-use APINo documented wayNo documented wayYes
SSHYesYesYes
Persistent diskYesYesYes
Pause/resumeNo documented wayNo documented wayYes
Automatic snapshotsNo documented wayYesYes
Fork/cloneNo documented wayYesYes
Live fork (no pause)No documented wayNo documented wayYes
Memory forkNo documented wayNo documented wayNo documented way
Custom image (Docker or snapshot)No documented wayYesYes
≥24 h sessionsYesNo documented wayYes
Idle auto-stopYesYesNo documented way
Wake on requestNo documented wayNo documented wayNo documented way
HTTPS preview URLsNo documented wayNo documented wayYes
Raw TCP inboundNo documented wayNo documented wayYes
Public IPv4No documented wayNo documented wayNo documented way
Egress allowlistNo documented wayNo documented wayNo documented way
Secret proxyNo documented wayYesYes
Secret proxy for any APINo documented wayNo documented wayYes
Preinstalled agentsNo documented wayNo documented wayYes
Agent harness APINo documented wayNo documented wayYes
MCP serverNo documented wayYesNo documented way
GPUNo documented wayNo documented wayNo documented way
WindowsYesYesNo documented way
Arm64YesYesNo documented way
Self-hostingNo documented wayYesNo documented way
BYOCNo documented wayNo documented wayNo documented way
Open sourceNo documented wayNo documented wayNo documented way
Scoped API keysNot publishedNot publishedYes
Spend limitsNot publishedNot publishedYes
Usage per sandbox (API)YesNo documented wayYes
Audit logsNot publishedNot publishedNo documented way
SSONo documented wayYesNo documented way
SOC 2YesYesNo documented way
HIPAANo documented wayNo documented wayNo documented way
EU data residencyYesNo documented wayNo documented way
Zero data retentionNot publishedNot publishedYes

Performance

BlacksmithNot measured
BuildkiteNot measured
boat28.06 runs/s

Blacksmith and Buildkite hosted agents are not in the public benchmarks we use, so we cannot say which of the three is faster. For reference, boat did 28.06 runs per second on the V8 Web Tooling Benchmark at 4 vCPU / 8 GB, second of 15 in StarSling's run of 3 October 2026 (run 37073680030). Every boat sandbox in that run ran on an AMD Ryzen 9 9950X host; boat also places sandboxes on Ryzen 7 9700X hosts and does not promise a CPU model.

V8 Web Tooling Benchmark at a 4 vCPU / 8 GB target, runs per second, higher is better. Source: StarSling's run of 3 October 2026 (run 37073680030), median of 5 sandboxes. Providers ran on different hardware in different regions. StarSling gives a shared rank when the run cannot separate two providers. Every boat sandbox in this run ran on an AMD Ryzen 9 9950X host. boat also places some sandboxes on AMD Ryzen 7 9700X hosts, which this run did not measure. Blacksmith and Buildkite are not in the public benchmarks we use, so we cannot say which is faster.

Pricing in five scenarios

Each bill is priced from the providers' public price lists by the open Battleships engine, with the cheapest plan that runs the scenario. Start limits, per-start fees and plan minimums are in the price, boat's included.

ScenarioBlacksmith, a monthBuildkite, a monthboat, a month
One coding agent, workdays
4 vCPU / 8 GB, 50 GB disk, 8 hours a day, 22 days a month, files kept between days
$36 · Pay-as-you-go · with limits$166 · Pro · with limits$20 · $20/mo
One agent that never stops
4 vCPU / 8 GB, 50 GB disk, running 24/7 all month
$207 · Pay-as-you-go · with limits$715 · Pro · with limits$26 · $20/mo
50 agents in parallel
50 machines of 4 vCPU / 8 GB at once, 8-hour sessions, 1,100 sessions a month, Docker inside
$2,364 · Pay-as-you-go · with limits$7,677 · Pro · with limits$317 · $20/mo × 3 seats
Sandboxes for your users
2 vCPU / 4 GB, 30,000 sessions of 10 minutes a month, up to 300 at once
$663 · Pay-as-you-go · with limits$2,204 · Pro · with limits$200 · $100/mo × 2 seats
1,000 short jobs at once
2 vCPU / 4 GB, 1,000 at once, 3 minutes each, 200,000 jobs a month
$1,488 · Pay-as-you-go · with limits$4,787 · Pro · with limits$8,800 · $2000/mo · with limits

Blacksmith: Blacksmith bills per minute of job time with no seat fee: Ubuntu x64 is $0.004 a minute for 2 vCPU, ARM $0.0025, Windows $0.008 and macOS $0.08 for 6 vCPU, with 3,000 free minutes a month. Sticky disks cost $0.50 per GB a month and a static IP $100 a month. Sandboxes have no published price.

Buildkite: Buildkite Pro costs $30 per active user a month and includes 4,000 Linux AMD64 vCPU minutes. After that, Linux is $0.004 per vCPU minute, Windows $0.008 and macOS $0.02, metered to the second with no minimums. Enterprise pricing is custom.

Buildkite: Plans cost $20, $100, $500 or $2,000 a month, and the whole plan price comes back as VM time for that month; unused time does not carry over. A card is required, also for the 7-day trial. A 4 vCPU / 8 GB sandbox costs $0.036 an hour, billed per second while it runs. Each plan includes a set number of starts (the $20 plan: 12 a minute, 200 a day, 100 running at once), and extra starts cost $0.03 to $0.07. A stopped sandbox has no compute charge and keeps its disk for at least 14 days; saved snapshots above the included count and the credentials proxy still charge.

Limits in those scenarios

None of the agent scenarios fit either service cleanly, because both are CI runners: packages installed outside sticky disks, cache volumes or a custom image reset after each job. Buildkite also stops a Linux job after 8 hours, so an agent that never stops needs restarts, and Pro's pool of 256 Linux vCPU makes 300 user sandboxes or 1,000 short jobs at once wait in a queue. Blacksmith's docs say it has no concurrency limit, but every runner start still has to be a GitHub Actions job. On boat, for the 1,000-job burst, the $2,000 plan charges for starts above 90 a minute and refuses them above 270; boat never queues a start. Auto-stop defaults to 1 hour.

ScenarioBlacksmithBuildkiteboat
One coding agent, workdays
  • different kind of product: CI runner
  • files survive a stop only in sticky disk paths: packages installed elsewhere reset
  • egress price not published
  • different kind of product: CI runner
  • files survive a stop only in cache volume paths: packages installed elsewhere reset
  • egress price not published
None for this scenario
One agent that never stops
  • different kind of product: CI runner
  • files survive a stop only in sticky disk paths: packages installed elsewhere reset
  • egress price not published
  • different kind of product: CI runner
  • sessions end after 8 h: needs restarts to run 24/7
  • files survive a stop only in cache volume paths: packages installed elsewhere reset
  • egress price not published
None for this scenario
50 agents in parallel
  • different kind of product: CI runner
  • files survive a stop only in sticky disk paths: packages installed elsewhere reset
  • egress price not published
  • how many machines can run at once: limit not published, probably fine at 50 (the typical published limit is 50)
  • different kind of product: CI runner
  • files survive a stop only in cache volume paths: packages installed elsewhere reset
  • egress price not published
None for this scenario
Sandboxes for your users
  • different kind of product: CI runner
  • not confirmed: how many machines can run at once (you need 300; the typical published limit is 50)
  • egress price not published
  • Pro: max 128 concurrent
  • Pro: max 256 vCPU running at once (needs 600)
  • different kind of product: CI runner
  • egress price not published
None for this scenario
1,000 short jobs at once
  • different kind of product: CI runner
  • not confirmed: how many machines can run at once (you need 1,000; the typical published limit is 50)
  • egress price not published
  • Pro: max 128 concurrent
  • Pro: max 256 vCPU running at once (needs 2,000)
  • different kind of product: CI runner
  • egress price not published
  • 2,400 starts/day included, paid above that (needs ~6,667)
  • 600 starts/hour included, paid above that (needs ~1,000)
  • starts above 90/min are paid, and starts above 270/min fail with 429

Which one should you use?

Use Blacksmith when

your CI runs on GitHub Actions and you want faster jobs by changing one label, with no concurrency cap, US or EU regions, and macOS or Windows runners.

  • A drop-in for GitHub Actions. If your CI already lives in GitHub Actions, you keep your workflows and switch by changing one runner label.
  • No concurrency cap and no seat fee. For big test matrices, Blacksmith's docs say it sets no concurrency limit, and you pay per job minute with no per-user fee.
  • US and EU regions. For teams that want runners near them, it runs in Ashburn, Phoenix, Amsterdam and Frankfurt.

Use Buildkite when

you want a full CI platform with its own pipelines and test tooling, self-hosted agents beside hosted ones, macOS on Apple M4, or Enterprise machines up to 64 vCPU.

  • A whole CI platform. For teams that want managed CI outside GitHub: pipelines, test tooling, cache volumes, and self-hosted agents next to hosted ones.
  • macOS on Apple M4 with Xcode. For iOS and Mac builds, its Mac hosted agents run current Xcode on Apple M4, up to 12 vCPU and 56 GB.
  • Very large build machines. For heavy builds on Enterprise, Linux shapes go up to 64 vCPU and 256 GB, in AMD64 and ARM64.

Where boat fits

Blacksmith and Buildkite are both CI services: a job machine exists only while a job runs, then it is destroyed. If what you are really running is agents that work for hours or days, start from your own backend and need to be reached from outside, boat gives each one a full Linux VM that stays. For plain CI on GitHub Actions or Buildkite pipelines, either of them is the better fit.

A machine that keeps its state

On both services the job machine is destroyed after each job. Only sticky disk folders, cache volume paths or a custom agent image carry over, so an agent sets up the rest again every run. boat saves the disk every minute, lets you fork a running sandbox, and has no session cap once auto-stop is off (default 1 h).

Reach in: URLs, ports and a desktop

Neither service exposes public ports today, only debug SSH or a terminal for a person, so opening what an agent built needs your own tunnel. boat gives token-gated HTTPS URLs on up to 50 ports and a 60 fps desktop for computer use.

Agents you drive from outside, keys kept out

On Blacksmith runners and Buildkite an agent is a job you start from a workflow and read through logs. Buildkite proxies the key for Anthropic. boat ships seven agents with a prompt, steer and events API, and an optional paid credentials proxy for supported model providers and your HTTPS APIs.

Use boat when what you run is agents, not CI jobs: one full Linux VM per agent that you start by API, that keeps its disk, hosts what it builds, shows a desktop and runs coding agents you drive from outside. Compare it one to one: boat vs Blacksmith and boat vs Buildkite.

boat's own limits
  • Plans start at $20 a month. The plan price comes back as VM time for that month. Unused time does not carry over. Source
  • Starts (create, fork, resume): the $20 plan includes 12 a minute, 60 an hour and 200 a day. The $2,000 plan includes 90 a minute, 600 an hour and 2,400 a day. Extra starts cost $0.03 to $0.07. A start fails with 429 above 3 times the included starts per minute, or 6 times per hour or per day. Source
  • A stopped sandbox has no compute charge. boat keeps its disk for at least 14 days. A saved snapshot keeps it longer; extra saved snapshots and the AI credentials proxy are paid. Source
  • Auto-stop defaults to 1 hour. You can turn it off on a paid plan. The trial allows 2 hours at most. Source
  • boat saves the disk every minute and at each stop. Memory, running programs and /tmp are not saved: a resume is like a reboot. A machine failure loses writes after the last good snapshot. Source
  • Machines run in Germany, Finland and France only. Snapshot copies can be stored outside the EU. Source
  • Public IPv6 and outbound IPv4. No dedicated IPv4. The IP changes at each resume. Source
  • Hosts are AMD Ryzen 7 9700X and Ryzen 9 9950X. vCPUs are shared. No GPU. You cannot boot your own Docker image. Source

Questions people ask

Is Blacksmith cheaper than Buildkite?

In every scenario we priced, Blacksmith comes out lower, for example $207 against $715 a month for one machine running all month and $2,364 against $7,677 for 50 machines in parallel. Blacksmith is priced on ARM runners and Buildkite on its 4 vCPU / 16 GB Linux shape. Neither fits these agent scenarios cleanly; boat comes to $26 and $317 for the same two.

Which is faster, Blacksmith or Buildkite?

Neither is in the public benchmarks we use, so we cannot compare them. For reference, boat did 28.06 runs per second on the V8 Web Tooling Benchmark at 4 vCPU / 8 GB in StarSling's run of 3 October 2026.

Can Blacksmith and Buildkite run Docker?

Yes on Linux. Blacksmith also supports nested virtualization on x64 Linux runners but not on ARM. Buildkite's macOS agents have the Docker CLI, and you connect a daemon you provide. boat has Docker, root, systemd and /dev/kvm on every sandbox.

How long can a job run on each?

A Buildkite hosted agent runs up to 8 hours per job on Linux and 4 on macOS, longer on request. A Blacksmith runner lives until its GitHub Actions job ends, and Testbox stops after 30 idle minutes by default. On boat, auto-stop defaults to 1 hour; on a paid plan you can set up to 30 days or turn it off.

Can I run an AI coding agent on Blacksmith or Buildkite?

On Buildkite you install Claude Code in a pipeline step, and its Anthropic proxy lets it call the API with the job token. Blacksmith has a Testbox CLI in early beta for local agents and its own [code]smith agent; its Sandboxes product is in waitlist beta with a preview SDK. On boat, coding agents come preinstalled and you prompt and steer them through an API.

Do Blacksmith and Buildkite run in the EU?

Blacksmith runs in Amsterdam and Frankfurt as well as Ashburn and Phoenix. Buildkite hosted agents run in US East Coast data centers. boat runs sandboxes on machines in Germany, Finland and France, and you cannot pick a region; snapshot copies can be stored outside the EU.

Try boat

Install the CLI, then run boat new.

7-day free trial

More comparisons