Runloop vs Fly.io Machines
Runloop is a sandbox product made for AI coding agents, with agent gateways, built-in agents and hostname egress rules. Fly.io Machines are fast microVMs for hosting your own apps near users, so you build the agent sandbox on top yourself.
The companies and products
Runloop
Runloop is a US platform of sandboxes, called devboxes, for building, running and testing AI coding agents.
You get Linux devboxes built from Dockerfile-style blueprints, disk snapshots, HTTPS tunnels, agent gateways that hide API keys, and Axons, event streams that drive agents such as Claude Code. Each devbox runs in a micro-VM on Runloop's own bare-metal hypervisor, with a container layer inside. It is built for teams shipping coding agents, plus evaluation and RL fine-tuning work on the Enterprise plan.
- Compliance
- SOC 2 Type 2
- Architectures
- x86_64 and arm64 base blueprints
- Own-cloud deploy
- Deploy to VPC on AWS, GCP and Azure (Enterprise)
- Regions
- Not documented or selectable
- Company
- Runloop AI, Inc.
Fly.io Machines
Fly.io is an app hosting platform, and Machines are the microVMs underneath it that you can also start and stop through a REST API.
A Fly Machine boots your Docker image as a Firecracker microVM on Fly's own servers, in one of 17 regions. You pick a shared or performance CPU preset, attach at most one Volume for data that must last, and Fly Proxy can stop, suspend and wake Machines as traffic comes and goes. It is built for teams that host web apps and services worldwide; it is not a sandbox product, so this page compares the parts that overlap with agent work.
Which use cases each one fits
For each use case, Battleships prices every sandbox provider on the same workload and sorts them from the cheapest monthly bill up. “20th cheapest of 367” means 19 of the other 366 providers run that workload, with every feature it needs, for less. “With limits” means the provider runs it but misses something, so it is listed after every full match. Open a use case for its whole ranking and what each provider misses.
| Use case and its workload | Runloop: price rank | Fly.io Machines: price rank | boat: price rank |
|---|---|---|---|
| Software factory 100 at once · 4 vCPU / 8 GB · 2,200 sessions of 8 h a month Runloop fits fully with Docker-in-Docker blueprints, while a Fly Machine resets its root disk on restart, so each agent needs its own Volume; boat keeps the whole disk and adds /dev/kvm. | 20th cheapest of 367 $8,152 a month | 92nd of 367, with limits $836 a month | 3rd cheapest of 367 $634 a month |
| Always-on AI employees 50 machines 24/7 · 2 vCPU / 4 GB | 32nd cheapest of 366 $10,194 a month | 125th of 366, with limits $816 a month | 5th cheapest of 366 $657 a month |
| Code interpreter bursts 100 at once · 1 vCPU / 2 GB · 200,000 sessions of 1 min a month Fly bills per second with no plan fee and cheap shared CPU, and Runloop fits fully; boat's per-plan start limits queue very large bursts. | 44th cheapest of 345 $525 a month | 75th of 345, with limits $94 a month | 186th of 345, with limits $2,000 a month |
| Sandboxes for your users 300 at once · 2 vCPU / 4 GB · 30,000 sessions of 23 min a month Runloop fits fully, while Fly's API creates about one Machine per second per app, so big sign-up waves need pre-created Machines or several apps. | 26th cheapest of 361 $3,301 a month | 69th of 361, with limits $600 a month | Cheapest of 361 $209 a month |
| RL rollouts 1,000 at once · 2 vCPU / 4 GB · 1,000,000 sessions of 3 min a month Runloop is the strongest of the three here: it starts large waves without published start limits and sells RL fine-tuning on Enterprise. | 4th cheapest of 360 $15,922 a month | 48th of 360, with limits $2,838 a month | 34th of 360, with limits $2,000 a month |
| Agent evals 250 at once · 4 vCPU / 8 GB · 15,000 sessions of 20 min a month | 81st cheapest of 364 $2,562 a month | 117th of 364, with limits $457 a month | 10th cheapest of 364 $200 a month |
| CI runners 40 at once · 8 vCPU / 16 GB · 6,000 sessions of 12 min a month Builds keep the CPU busy, which drains a shared Fly vCPU's burst balance, while boat ran about three times faster than Runloop on the CPU test. | 121st cheapest of 377 $1,554 a month | 152nd of 377, with limits $288 a month | 4th cheapest of 377 $86 a month |
| 24/7 dev boxes 1 machine 24/7 · 4 vCPU / 8 GB Runloop can run all month with an idle policy but has no public IP or raw TCP, and Fly's cheapest shared CPU is throttled under steady load. | 113th cheapest of 368 $410 a month | 118th of 368, with limits $53 a month | 18th cheapest of 368 $26 a month |
| Computer-use agents 50 at once · 4 vCPU / 8 GB · 20,000 sessions of 15 min a month Runloop and boat ship a desktop and a browser your agent can control, with boat streaming at 60fps; on Fly you build the desktop into your own image. | 39th cheapest of 369 $2,306 a month | 95th of 369, with limits $173 a month | 6th cheapest of 369 $200 a month |
| Browser-use agents 50 at once · 2 vCPU / 4 GB · 30,000 sessions of 10 min a month | 40th cheapest of 367 $1,153 a month | 160th of 367, with limits $90 a month | 7th cheapest of 367 $200 a month |
| Scraping & bots (startup scale) 50 at once · 1 vCPU / 2 GB · 100,000 sessions of 3 min a month | 68th cheapest of 346 $781 a month | 161st of 346, with limits $67 a month | 6th cheapest of 346 $90 a month |
| Mass scraping & bots (undetectable) 1,000 at once · 1 vCPU / 2 GB · 3,000,000 sessions of 2 min a month Fly fits best of the three here: it can spread scrapers across 17 regions and pay only for the seconds they run. | 266th of 338, with limits $15,611 a month | 44th of 338, with limits $1,232 a month | 71st of 338, with limits $1,800 a month |
| Parallel red teaming 200 at once · 2 vCPU / 4 GB · 5,000 sessions of 30 min a month | 10th cheapest of 360 $770 a month | 118th of 360, with limits $63 a month | 2nd cheapest of 360 $200 a month |
| GPU eval jobs 10 at once · 8 vCPU / 32 GB · 500 sessions of 30 min a month | Can't run it | Can't run it | Can't run it |
| macOS agents 5 at once · 4 vCPU / 8 GB · 1,000 sessions of 20 min a month | Can't run it | Can't run it | Can't run it |
Key features
| Feature | Runloop | Fly.io Machines | boat |
|---|---|---|---|
| Kind of product | Agent sandbox: devboxes for building, running and testing coding agents | App hosting platform; you boot your own image as a microVM | Agent sandbox: a full VM per agent, ready to use |
| Isolation | Micro-VM on a custom bare-metal hypervisor, plus a container layer | Firecracker microVM with its own kernel | Full VM with its own kernel on dedicated AMD Ryzen 9 9950X hosts |
| Docker, systemd and nested virtualization | Docker on the dnd blueprints, as root; nested virtualization not documented | Docker you install yourself; Fly's own init, no systemd; /dev/kvm not offered | Docker, root, systemd and /dev/kvm on every machine |
| Images | Blueprints built from Dockerfiles, x86_64 and arm64 | Your own Docker image | Templates from named snapshots, x86_64 only, no Dockerfile builds |
| Files after a stop | Disk kept through suspend (Pro plan) or disk snapshots; shutdown ends the devbox | Root file system is ephemeral; one Volume per Machine, billed for its full size all month | Whole disk kept, snapshots every minute and on stop; a stopped sandbox costs nothing |
| Long runs | Fixed lifetime 1 hour by default, 48 hours max; an idle policy replaces it | No documented maximum runtime | No session cap on paid plans; default TTL 1 hour, changeable |
| Sleep and wake | Suspend after idle, wake on an HTTP request to the tunnel (Pro) | Fly Proxy stops or suspends idle Machines and wakes them on a request; memory kept up to 2GB | Start and stop through the API; resume reboots from disk |
| Fork | Disk snapshots you can start new devboxes from | Clone copies a Machine's config; a cloned Volume starts empty | Fork a running sandbox, disk included |
| Desktop and computer use | Desktop and browser your agent can control | None built in; you add a desktop and browser to your image | 60fps desktop with Chromium, computer use and browser control, embeddable stream |
| Coding agents | Claude Code, Codex, OpenCode, Gemini CLI, DeepAgents, driven through Axons and Broker | None built in; you install and run agents yourself | Claude Code, Codex, OpenCode and more, with a prompt, steer and events API |
| Keeping keys out of the sandbox | Agent gateways for any API endpoint with bearer, header or basic auth | Secrets as environment variables; open-source tokenizer proxy you deploy yourself | AI credentials proxy for any HTTPS API; switch a credential to another account when credits run out |
| Reaching the sandbox | One HTTPS tunnel per devbox; SSH through a TLS proxy; no public IP | HTTPS and TCP services through Fly Proxy, Anycast IPs, custom domains, private network | HTTPS URLs for up to 50 ports, token-protected; raw TCP; dedicated IPv6 or IPv4 |
| Outbound rules | Hostname allowlist with wildcards, or block all | Network policies allow chosen ports and protocols | Open internet, no provider-side allowlist |
| Where it runs | Regions not documented; Deploy to VPC on AWS, GCP or Azure (Enterprise) | 17 regions on six continents | EU only (Germany, Finland, France), EU data residency |
Every feature Battleships tracks (43)
| Feature | Runloop | Fly.io Machines | boat |
|---|---|---|---|
| Full VM (own kernel) | Yes | Yes | Yes |
| Docker inside | Yes | Yes | Yes |
| Nested virtualization | No | No | Yes |
| Root | Yes | Yes | Yes |
| Systemd | Not published | No | Yes |
| Desktop GUI | Yes | No | Yes |
| Browser | Yes | No | Yes |
| Computer-use API | Yes | No | Yes |
| SSH | Yes | Yes | Yes |
| Persistent disk | Yes | Yes | Yes |
| Pause/resume | Yes | Yes | Yes |
| Automatic snapshots | Yes | Yes | Yes |
| Fork/clone | Yes | Yes | Yes |
| Live fork (no pause) | Yes | Yes | Yes |
| Memory fork | No | No | No |
| Custom image (Docker or snapshot) | Yes | Yes | Yes |
| ≥24 h sessions | Yes | Yes | Yes |
| Idle auto-stop | Yes | Yes | No |
| Wake on request | Yes | Yes | No |
| HTTPS preview URLs | Yes | Yes | Yes |
| Raw TCP inbound | No | Yes | Yes |
| Public IPv4 | No | Yes | No |
| Egress allowlist | Yes | Yes | No |
| Secret proxy | Yes | No | Yes |
| Secret proxy for any API | Yes | No | Yes |
| Preinstalled agents | Yes | No | Yes |
| Agent harness API | Yes | No | Yes |
| MCP server | Yes | Yes | No |
| GPU | No | No | No |
| Windows | No | No | No |
| Arm64 | Yes | No | No |
| Self-hosting | No | No | No |
| BYOC | Yes | No | No |
| Open source | No | Yes | No |
| Scoped API keys | Yes | Not published | Yes |
| Spend limits | Not published | No | Yes |
| Usage per sandbox (API) | No | No | Yes |
| Audit logs | Not published | Not published | No |
| SSO | Yes | No | No |
| SOC 2 | Yes | Yes | No |
| HIPAA | Not published | Yes | No |
| EU data residency | No | Yes | Yes |
| Zero data retention | Not published | Not published | Yes |
Performance
On the StarSling Node.js build-and-test loop at 4 vCPU / 8 GB (run 36498291449, 2026-09-29), boat scored 26.69 runs a second and Runloop 8.29, about 3.2 times apart. Fly.io Machines were not part of that run, so there is no comparable measurement for Fly; its results would depend on whether you pick shared vCPUs, which guarantee 6.25% of a core, or performance vCPUs, which get the full core.
Node.js build-and-test loop at 4 vCPU / 8 GB, higher is better. Measured with the open hpc-sandbox-benchmarks suite. Fly.io Machines has no published run in this suite yet, so there is no like-for-like number to compare.
Pricing in five scenarios
Each bill is priced from the providers' public price lists by the open Battleships engine, with the cheapest plan that runs the scenario.
| Scenario | Runloop, a month | Fly.io Machines, a month | boat, a month |
|---|---|---|---|
| One coding agent, workdays 4 vCPU / 8 GB, 50 GB disk, 8 hours a day, 22 days a month, files kept between days | $113 · Basic | $15 · Pay As You Go · with limits | $20 · $20/mo |
| One agent that never stops 4 vCPU / 8 GB, 50 GB disk, running 24/7 all month | $429 · Basic | $36 · Pay As You Go · with limits | $26 · $20/mo |
| 50 agents in parallel 50 machines of 4 vCPU / 8 GB at once, 8-hour sessions, 1,100 sessions a month, Docker inside | $5,268 · Basic | $487 · Pay As You Go · with limits | $317 · $20/mo × 3 seats |
| Sandboxes for your users 2 vCPU / 4 GB, 30,000 sessions of 10 minutes a month, up to 300 at once | $1,441 · Basic | $110 · Pay As You Go · with limits | $200 · $100/mo × 2 seats |
| 1,000 short jobs at once 2 vCPU / 4 GB, 1,000 at once, 3 minutes each, 200,000 jobs a month | $3,184 · Basic | $232 · Pay As You Go · with limits | $2,000 · $2000/mo · with limits |
Runloop: Runloop bills per second for the CPU and memory you allocate while a devbox is starting, running, suspending or resuming, and bills its disk for the devbox's whole life. Basic has no plan fee; Pro adds $250 a month that is not usage credit and unlocks suspend.
Fly.io Machines: Fly bills each Machine per second while it runs, by CPU preset and memory, with no plan fee; Volumes are billed for their full size all month. The figures here use Fly's lowest rate: shared vCPUs on a 1-year reservation.
Fly.io Machines: Plans start at $20 a month and the whole fee is usage credit, billed per second while a sandbox runs. A stopped sandbox costs nothing and keeps its disk.
Limits in those scenarios
Runloop fits every scenario, but publishes no egress price, caps disk at 64 GiB and needs an idle policy to run past 48 hours. Fly is marked a compromise in all five: its lowest figures need a 1-year commitment and use shared vCPUs throttled above 6.25% of each core, and its API starts about one Machine per second per app. boat's limit that bites is start rate, so 1,000 jobs at once gets queued even on the $2,000 plan.
| Scenario | Runloop | Fly.io Machines | boat |
|---|---|---|---|
| One coding agent, workdays |
|
| None that bite |
| One agent that never stops |
|
| None that bite |
| 50 agents in parallel |
|
| None that bite |
| Sandboxes for your users |
|
| None that bite |
| 1,000 short jobs at once |
|
|
|
Which one should you use?
Use Runloop when
you want a ready agent sandbox with hostname egress rules, arm64 or Dockerfile images, deploys into your own cloud, or very large bursts of short jobs.
- Made for coding agents. Useful if you want agent tooling out of the box: Runloop ships built-in agents, Axons event streams and agent gateways that hide API keys.
- Hostname egress rules. Useful if a sandbox must only reach a fixed list of hosts: Runloop allows listed hostnames, with wildcards, or blocks everything.
- Your own cloud and arm64. On Enterprise, Runloop runs devboxes in your own AWS, GCP or Azure account, and it builds x86_64 or arm64 images from Dockerfiles.
Use Fly.io Machines when
you host web apps or light services that must run near users worldwide, sleep between requests and wake on traffic.
- 17 regions worldwide. Useful if your users are spread out: Fly runs Machines in 17 regions, from the US to Sydney and Johannesburg.
- Wake on request with memory kept. Useful for services that sleep between requests: Fly Proxy wakes a Machine on traffic, and small Machines resume with memory kept.
- A full app platform. Useful if you host whole apps next to your sandboxes: Fly adds raw TCP, private networking, Anycast IPs and custom domains.
Where boat fits
If your agents spend hours installing, building and testing, look at boat. Each sandbox is a full VM with Docker, systemd and /dev/kvm on fast dedicated CPUs, and its whole disk is kept through every stop, which neither Runloop nor Fly gives you on its basic terms.
Files kept on every stop
A Runloop devbox is gone once it shuts down unless you pay for Pro suspend or write snapshot code, and a Fly Machine's root disk resets, so work must live on a Volume billed all month. boat keeps the whole disk.
A full Linux machine
Neither Runloop nor Fly offers /dev/kvm, so Android emulators and nested VMs do not start, and Fly has no systemd. Runloop also has no raw TCP way in. boat has all of these on every machine.
CPU that does not slow down
boat ran 26.69 build-and-test loops a second against Runloop's 8.29, so agents wait about a third as long. Fly's cheapest shared vCPUs guarantee 6.25% of a core and throttle long builds; full cores cost several times more.
Use boat when your agents do real work for hours: installs, builds, tests, Docker and emulators on a full VM whose disk survives every stop, with a desktop, coding agents and a keys proxy built in. Compare it one to one: boat vs Runloop and boat vs Fly.io Machines.
Questions people ask
Is Runloop cheaper than Fly.io Machines?
No, on our scenarios Fly costs far less, if your work fits its cheapest shared CPU. One coding agent on workdays is $113 a month on Runloop and $15 on Fly, and 50 agents in parallel are $5,268 versus $487. boat is $20 and $317 for the same two.
Which is faster, Runloop or Fly.io Machines?
We cannot say: Runloop scored 8.29 runs a second on the StarSling CPU test, and Fly was not measured. Fly's shared vCPUs guarantee only 6.25% of a core, so long builds get throttled unless you pay for performance vCPUs. boat scored 26.69 on the same test.
Can Runloop and Fly.io Machines run Docker?
Yes, both. Runloop has Docker-in-Docker blueprints, and on Fly you install Docker in your own image. Neither offers /dev/kvm for nested VMs or Android emulators; boat has Docker, systemd and /dev/kvm on every machine.
Do they keep my files when a sandbox stops?
A Runloop devbox keeps its disk only while suspended on Pro or in a snapshot you take, and shutdown ends it. A Fly Machine keeps only what is on its one Volume. boat keeps the whole disk, with snapshots every minute and on stop.
Can I use Fly.io Machines as an AI agent sandbox?
Yes, but you build the sandbox yourself: the image with tools, browser and agents, a Volume for files, and the code that drives each agent. Runloop and boat ship agents, a desktop and key-hiding proxies built in.
Where do they run?
Fly runs Machines in 17 regions on six continents. Runloop does not document its regions, but Enterprise can deploy into your own AWS, GCP or Azure account. boat runs only in the EU, with EU data residency.
Try boat
Install the CLI, then run boat new.
7-day free trial