Runloop vs Fly.io Machines
Runloop is a sandbox product for AI coding agents, with agent gateways, agent wrappers and hostname egress rules. Fly.io Machines are microVMs for hosting your own apps near users. On Fly you build the agent sandbox on top yourself, or use Fly's separate Sprites product.
The companies and products
Runloop
Runloop is a US platform of sandboxes, called devboxes, for building, running and testing AI coding agents.
You get Linux devboxes built from Dockerfile-style blueprints, disk snapshots, HTTPS tunnels, agent gateways that hide API keys, browser and computer use, and Axons, event streams that drive agents such as Claude Code. Each devbox runs in a micro-VM on Runloop's own bare-metal hypervisor, with a container layer inside. It is built for teams that ship coding agents, plus evaluation and RL fine-tuning work on the Enterprise plan.
- Architectures
- x86_64 and arm64 base blueprints
- Own-cloud deploy
- Deploy to VPC on AWS, GCP and Azure (Enterprise)
- Regions
- Not documented or selectable
- Company
- Runloop AI, Inc.
Fly.io Machines
Fly.io is an app hosting platform, and Machines are the microVMs underneath it that you can also start and stop through a REST API.
A Fly Machine boots your Docker image as a Firecracker microVM on Fly's own servers, in one of 17 regions. You pick a shared or performance CPU preset and mount one Volume at a time for data that must last. Fly Proxy can stop, suspend and wake Machines as traffic comes and goes. Machines themselves are an app-hosting building block; for agent sandboxes Fly also sells Sprites (see boat vs Sprites), so this page compares only the Machines product.
Which use cases each one fits
For each use case, Battleships prices every sandbox provider on the same workload and sorts them from the cheapest monthly bill up. “24th cheapest of 365” means 23 of the other 364 providers run that workload, with every feature it needs, for less. “With limits” means the provider runs it but misses something, so it is listed after every full match. Open a use case for its whole ranking and what each provider misses.
| Use case and its workload | Runloop: price rank | Fly.io Machines: price rank | boat: price rank |
|---|---|---|---|
| Software factory 100 at once · 4 vCPU / 8 GB · 2,200 sessions of 8 h a month Runloop fits with its Docker-in-Docker blueprints. On Fly you set it up: install Docker in your image, and keep files on a Volume or turn on persist_rootfs. boat saves its disk every minute and at each stop and adds /dev/kvm. | 24th cheapest of 365 $8,152 a month | 123rd of 365, with limits $836 a month | 5th cheapest of 365 $634 a month |
| Always-on AI employees 50 machines 24/7 · 2 vCPU / 4 GB | 38th cheapest of 361 $10,194 a month | 186th of 361, with limits $816 a month | 6th cheapest of 361 $657 a month |
| Code interpreter bursts 100 at once · 1 vCPU / 2 GB · 200,000 sessions of 1 min a month Runloop fits and ranks higher than boat here. Fly bills per second with no plan fee, but its lowest rate needs a 1-year reservation. boat includes 12 / 60 / 200 starts per minute / hour / day on $20, and extra starts cost $0.03 to $0.07. | 52nd cheapest of 336 $525 a month | 83rd of 336, with limits $94 a month | 189th of 336, with limits $2,000 a month |
| Sandboxes for your users 300 at once · 2 vCPU / 4 GB · 30,000 sessions of 23 min a month Runloop fits. Fly's Machines API takes about one create request per second per app, so big sign-up waves need pre-created Machines or several apps. boat allows 100 sandboxes at once on $20 and 2,000 on $2,000. | 37th cheapest of 349 $3,301 a month | 75th of 349, with limits $600 a month | 3rd cheapest of 349 $209 a month |
| RL rollouts 1,000 at once · 2 vCPU / 4 GB · 1,000,000 sessions of 3 min a month Runloop ranks highest of the three here. It publishes no start-rate limit for paid plans and sells RL fine-tuning on Enterprise. | 8th cheapest of 349 $15,922 a month | 52nd of 349, with limits $2,838 a month | 39th of 349, with limits $2,000 a month |
| Agent evals 250 at once · 4 vCPU / 8 GB · 15,000 sessions of 20 min a month | 91st cheapest of 359 $2,562 a month | 124th of 359, with limits $457 a month | 10th cheapest of 359 $200 a month |
| CI runners 40 at once · 8 vCPU / 16 GB · 6,000 sessions of 12 min a month Builds keep the CPU busy. Fly's lowest-priced shared vCPUs guarantee 6.25% of a core, and full cores cost more. In StarSling's run of 3 October 2026, boat did 28.06 runs a second and Runloop 8.93; Fly was not measured. | 125th cheapest of 374 $1,554 a month | 154th of 374, with limits $288 a month | 2nd cheapest of 374 $86 a month |
| 24/7 dev boxes 1 machine 24/7 · 4 vCPU / 8 GB Runloop can run 24/7 with an idle policy, reached through HTTPS tunnels and SSH. Fly keeps one Volume per Machine, and the root file system only with opt-in persist_rootfs. On boat, turn auto-stop off (default 1 hour) on a paid plan. | 118th cheapest of 371 $410 a month | 160th of 371, with limits $35 a month | 19th cheapest of 371 $26 a month |
| Computer-use agents 50 at once · 4 vCPU / 8 GB · 20,000 sessions of 15 min a month Runloop and boat ship a desktop and a browser that an agent can control; boat streams at 60 fps. On Fly you set it up: add a desktop, a browser and a VNC stream to your own image. | 41st cheapest of 367 $2,306 a month | 140th of 367, with limits $173 a month | 7th cheapest of 367 $200 a month |
| Browser-use agents 50 at once · 2 vCPU / 4 GB · 30,000 sessions of 10 min a month | 38th cheapest of 356 $1,153 a month | 229th of 356, with limits $90 a month | 7th cheapest of 356 $200 a month |
| Scraping & bots (startup scale) 50 at once · 1 vCPU / 2 GB · 100,000 sessions of 3 min a month | 73rd cheapest of 343 $781 a month | 233rd of 343, with limits $67 a month | 6th cheapest of 343 $90 a month |
| Mass scraping & bots (undetectable) 1,000 at once · 1 vCPU / 2 GB · 3,000,000 sessions of 2 min a month None of the three is a full match. Fly ranks highest of the three: it runs in 17 regions and bills per second. None ships anti-bot stealth, CAPTCHA solving or residential IPs. | 267th of 327, with limits $15,611 a month | 47th of 327, with limits $1,232 a month | 73rd of 327, with limits $1,800 a month |
| Parallel red teaming 200 at once · 2 vCPU / 4 GB · 5,000 sessions of 30 min a month | 16th cheapest of 351 $770 a month | 168th of 351, with limits $63 a month | 3rd cheapest of 351 $200 a month |
| GPU eval jobs 10 at once · 8 vCPU / 32 GB · 500 sessions of 30 min a month | Not ranked: misses a requirement | Not ranked: misses a requirement | Not ranked: misses a requirement |
| macOS agents 5 at once · 4 vCPU / 8 GB · 1,000 sessions of 20 min a month | Not ranked: misses a requirement | Not ranked: misses a requirement | Not ranked: misses a requirement |
Key features
| Feature | Runloop | Fly.io Machines | boat |
|---|---|---|---|
| Kind of product | Agent sandbox: devboxes for building, running and testing coding agents | App hosting platform; you boot your own image as a microVM. Fly sells Sprites separately for agent sandboxes | Agent sandbox: a full VM per agent, ready to use |
| Isolation | Micro-VM on a custom bare-metal hypervisor, plus a container layer | Firecracker microVM with its own kernel | Full VM with its own kernel, on physical hosts dedicated to boat (AMD Ryzen 7 9700X and Ryzen 9 9950X); vCPUs are shared |
| Docker, systemd and nested virtualization | Docker on the Docker-in-Docker blueprints, as root; nested virtualization not documented | Root and Docker you install yourself; Fly's own init by default; /dev/kvm not documented | Docker, root, real systemd and /dev/kvm on every sandbox |
| Images | Blueprints built from Dockerfiles, x86_64 and arm64 | Your own Docker image | boat's image or a saved snapshot, x86_64 only; you cannot boot your own Docker or OCI image |
| Files after a stop | Disk snapshots on every plan; idle suspend keeps the disk on Pro; shutdown deletes the disk | Root file system resets by default; opt-in persist_rootfs keeps it, but Fly staff say not to trust it with critical data. Lasting data goes on one Volume, billed for its full size | Disk saved every minute and at each stop, kept at least 14 days; memory and processes are not saved; a machine failure loses writes after the last good snapshot |
| Long runs | Keep-alive timer 1 hour by default, 48 hours max; an idle policy replaces it, with no documented cap | No documented maximum runtime | Auto-stop 1 hour by default, up to 30 days, or off on a paid plan (then no session cap) |
| Sleep and wake | Suspend after idle, wake on an HTTP request to the tunnel (Pro) | Fly Proxy stops or suspends idle Machines and wakes them on a request; memory kept for Machines up to 2GB | No idle timer; auto-stop counts from creation or resume; resume reboots from disk |
| Snapshots and fork | Disk snapshots you can start new devboxes from | Clone copies a Machine's config; restore a Volume snapshot or fork the Volume; the root file system is not copied | Automatic disk snapshots; fork a running sandbox from its last 1-minute snapshot |
| Desktop and computer use | Desktop and browser that an agent can control | Not built in; you set it up: add a desktop, a browser and a VNC stream to your own image | 1920x1080, 60 fps desktop with Chrome and computer tools; embeddable stream |
| Coding agents | Claude Code, Codex, OpenCode, Gemini CLI and DeepAgents through public agent wrappers | Not built in on Machines; you install agents in your image. Fly's separate Sprites product ships Claude Code, Codex and Gemini | Seven agents installed, with a prompt, steer and events API |
| Keeping keys out of the sandbox | Agent gateways for any API endpoint with bearer, header or basic auth | Secrets arrive as environment variables; you set up key injection by deploying Fly's open-source tokenizer proxy | Optional AI credentials proxy, off by default, $1 a day plus $1 per 1,000 requests; keys of supported model providers stay outside; other secrets still enter the sandbox |
| Reaching the sandbox | One HTTPS tunnel per devbox, open or bearer token; SSH through a TLS proxy; no public IP documented | Public HTTPS and TCP services through Fly Proxy, Anycast IPs, custom domains, private network between Machines | Up to 50 HTTPS URLs, token-gated by default; SSH; public IPv6 address with raw ports (some sandboxes have no IPv4 of their own) |
| Outbound rules | Hostname allowlist with wildcards, or block all | Network policies allow chosen ports and protocols | Open internet; no provider-side allowlist. You set it up: firewall rules inside the VM, as root |
| Where it runs | Regions not documented; Deploy to VPC on AWS, GCP or Azure (Enterprise) | 17 regions on six continents, on Fly's own servers | Germany, Finland and France; you cannot pick a region; no US or Asia region |
Every feature Battleships tracks (43)
| Feature | Runloop | Fly.io Machines | boat |
|---|---|---|---|
| Full VM (own kernel) | Yes | Yes | Yes |
| Docker inside | Yes | Yes | Yes |
| Nested virtualization | No documented way | Not published | Yes |
| Root | Yes | Yes | Yes |
| Systemd | Not published | Not published | Yes |
| Desktop GUI | Yes | No documented way | Yes |
| Browser | Yes | No documented way | Yes |
| Computer-use API | Yes | No documented way | Yes |
| SSH | Yes | Yes | Yes |
| Persistent disk | Yes | Yes | Yes |
| Pause/resume | Yes | Yes | Yes |
| Automatic snapshots | Yes | Yes | Yes |
| Fork/clone | Yes | Yes | Yes |
| Live fork (no pause) | Yes | Yes | Yes |
| Memory fork | No documented way | No documented way | No documented way |
| Custom image (Docker or snapshot) | Yes | Yes | Yes |
| ≥24 h sessions | Yes | Yes | Yes |
| Idle auto-stop | Yes | Yes | No documented way |
| Wake on request | Yes | Yes | No documented way |
| HTTPS preview URLs | Yes | Yes | Yes |
| Raw TCP inbound | No documented way | Yes | Yes |
| Public IPv4 | No documented way | Yes | No documented way |
| Egress allowlist | Yes | Yes | No documented way |
| Secret proxy | Yes | No documented way | Yes |
| Secret proxy for any API | Yes | No documented way | Yes |
| Preinstalled agents | Yes | No documented way | Yes |
| Agent harness API | Yes | No documented way | Yes |
| MCP server | Yes | Yes | No documented way |
| GPU | No documented way | No documented way | No documented way |
| Windows | No documented way | No documented way | No documented way |
| Arm64 | Yes | No documented way | No documented way |
| Self-hosting | No documented way | No documented way | No documented way |
| BYOC | Yes | No documented way | No documented way |
| Open source | No documented way | Yes | No documented way |
| Scoped API keys | Yes | Not published | Yes |
| Spend limits | Not published | No documented way | Yes |
| Usage per sandbox (API) | Yes | No documented way | Yes |
| Audit logs | Not published | Not published | No documented way |
| SSO | Yes | No documented way | No documented way |
| SOC 2 | Yes | Yes | No documented way |
| HIPAA | Not published | Yes | No documented way |
| EU data residency | No documented way | Yes | No documented way |
| Zero data retention | Not published | Not published | Yes |
Performance
In StarSling's run of 3 October 2026 (run 37073680030), on the V8 Web Tooling Benchmark at 4 vCPU / 8 GB, boat did 28.06 runs a second (second of 15) and Runloop 8.93 (15th of 15), about 3.1 times apart. Fly.io Machines are not in the public benchmarks we use, so we cannot say how Fly compares. Fly results would also depend on the CPU class: shared vCPUs guarantee 6.25% of a core, and performance vCPUs get the full core. Every boat sandbox in this run ran on an AMD Ryzen 9 9950X host. boat also places some sandboxes on AMD Ryzen 7 9700X hosts, which this run did not measure. Providers ran on different hardware in different regions.
V8 Web Tooling Benchmark at a 4 vCPU / 8 GB target, runs per second, higher is better. Source: StarSling's run of 3 October 2026 (run 37073680030), median of 5 sandboxes. Providers ran on different hardware in different regions. StarSling gives a shared rank when the run cannot separate two providers. Every boat sandbox in this run ran on an AMD Ryzen 9 9950X host. boat also places some sandboxes on AMD Ryzen 7 9700X hosts, which this run did not measure. Fly.io Machines is not in the public benchmarks we use, so we cannot say which is faster.
Pricing in five scenarios
Each bill is priced from the providers' public price lists by the open Battleships engine, with the cheapest plan that runs the scenario. Start limits, per-start fees and plan minimums are in the price, boat's included.
| Scenario | Runloop, a month | Fly.io Machines, a month | boat, a month |
|---|---|---|---|
| One coding agent, workdays 4 vCPU / 8 GB, 50 GB disk, 8 hours a day, 22 days a month, files kept between days | $113 · Basic | $15 · Pay As You Go · with limits | $20 · $20/mo |
| One agent that never stops 4 vCPU / 8 GB, 50 GB disk, running 24/7 all month | $429 · Basic | $36 · Pay As You Go · with limits | $26 · $20/mo |
| 50 agents in parallel 50 machines of 4 vCPU / 8 GB at once, 8-hour sessions, 1,100 sessions a month, Docker inside | $5,268 · Basic | $487 · Pay As You Go · with limits | $317 · $20/mo × 3 seats |
| Sandboxes for your users 2 vCPU / 4 GB, 30,000 sessions of 10 minutes a month, up to 300 at once | $1,441 · Basic | $110 · Pay As You Go · with limits | $200 · $100/mo × 2 seats |
| 1,000 short jobs at once 2 vCPU / 4 GB, 1,000 at once, 3 minutes each, 200,000 jobs a month | $3,184 · Basic | $232 · Pay As You Go · with limits | $8,800 · $2000/mo · with limits |
Runloop: Runloop bills per second for allocated CPU ($0.108 per vCPU-hour) and memory ($0.0252 per GB-hour) while a devbox is initializing, running, suspending or resuming: $0.63 an hour at 4 vCPU / 8 GB. Its disk bills $0.2499 per GB-month for the devbox's whole life. Basic has no plan fee. Pro adds $250 a month that is not usage credit, and unlocks suspend and resume. New accounts get $50 of credit with no card.
Fly.io Machines: Fly bills each Machine per second while it runs, by CPU preset and memory, with no plan fee. Stopped Machines pay for their root file system, and Volumes are billed for their full size at $0.15 per GB a month. The table uses Fly's lowest rate: shared vCPUs in 1-year reservation blocks for 40% off. A performance-4x Machine with 8GB, which gets the full cores, is $0.1833 an hour on demand.
Limits in those scenarios
Runloop fits all five scenarios, but publishes no egress price, caps disk at 64 GiB, and needs an idle policy to run past 48 hours; suspend on idle needs Pro. The table marks Fly as a compromise in all five: its lowest figures need a 1-year reservation and use shared vCPUs that guarantee 6.25% of each core, and its Machines API takes about one create request per second per app. boat's limits: plans start at $20 a month, and the included time expires at the end of the month. A stopped sandbox has no compute charge, but saved snapshots above the included count and the AI credentials proxy still charge. For 1,000 short jobs at once, even the $2,000 plan charges $0.03 to $0.07 for starts above 90 a minute and refuses them above 270, so the burst does not fit boat.
| Scenario | Runloop | Fly.io Machines | boat |
|---|---|---|---|
| One coding agent, workdays |
|
| None for this scenario |
| One agent that never stops |
|
| None for this scenario |
| 50 agents in parallel |
|
| None for this scenario |
| Sandboxes for your users |
|
| None for this scenario |
| 1,000 short jobs at once |
|
|
|
Which one should you use?
Use Runloop when
you want a ready agent sandbox with hostname egress rules, arm64 or Dockerfile images, deploys into your own cloud, or very large bursts of short jobs.
- Made for coding agents. For agent tooling with no setup: Runloop ships agent wrappers, Axons event streams and agent gateways that keep API keys out of the devbox.
- Hostname egress rules. For a sandbox that must reach only a fixed list of hosts: Runloop allows listed hostnames, with wildcards, or blocks everything.
- Your own cloud and arm64. On Enterprise, Runloop runs devboxes in your own AWS, GCP or Azure account. It builds x86_64 or arm64 images from Dockerfiles.
Use Fly.io Machines when
you host web apps or light services near users worldwide that sleep between requests and wake on traffic, and you are happy to build the sandbox image yourself.
- Lower bills in these scenarios. On a 1-year shared-CPU reservation, Fly costs less than Runloop in all five scenarios on this page. Its shared CPU guarantees 6.25% of each core.
- 17 regions and wake on request. For services near users that sleep between requests: Fly runs in 17 regions, wakes a Machine on traffic, and resumes Machines up to 2GB with memory kept.
- A full app platform. For hosting whole apps next to your sandboxes: Fly adds public TCP services, private networking, Anycast IPs and custom domains.
Where boat fits
Look at boat if your agents spend hours installing, building and testing on one machine. Each sandbox is a full VM with Docker, systemd and /dev/kvm, and boat saves its disk every minute and at each stop with no code from you.
The disk comes back with no setup
Runloop deletes the disk at shutdown, so you keep work with snapshot calls or Pro suspend. Fly keeps one Volume, and the root file system only with opt-in persist_rootfs. boat saves the disk every minute and at each stop and keeps it at least 14 days. Memory and running programs are not saved.
A fuller Linux machine
boat has root, real systemd, Docker and /dev/kvm in every sandbox, so Android emulators and nested VMs start. Neither Runloop nor Fly documents /dev/kvm. boat cannot boot your own image and runs x86_64 only.
More CPU work per hour, at a lower list price
In StarSling's run of 3 October 2026, boat did 28.06 runs a second and Runloop 8.93 on the V8 Web Tooling Benchmark; Fly was not measured. boat lists $0.036 an hour at 4 vCPU / 8 GB, from a $20 plan. Every boat sandbox in that run ran on an AMD Ryzen 9 9950X host; boat also uses other hosts.
Use boat when your agents do real work for hours: installs, builds, tests, Docker and emulators on a full VM whose disk is saved every minute and at each stop. boat runs only in Germany, Finland and France and has no GPU. Compare it one to one: boat vs Runloop and boat vs Fly.io Machines.
boat's own limits
- Plans start at $20 a month. The plan price comes back as VM time for that month. Unused time does not carry over. Source
- Starts (create, fork, resume): the $20 plan includes 12 a minute, 60 an hour and 200 a day. The $2,000 plan includes 90 a minute, 600 an hour and 2,400 a day. Extra starts cost $0.03 to $0.07. A start fails with 429 above 3 times the included starts per minute, or 6 times per hour or per day. Source
- A stopped sandbox has no compute charge. boat keeps its disk for at least 14 days. A saved snapshot keeps it longer; extra saved snapshots and the AI credentials proxy are paid. Source
- Auto-stop defaults to 1 hour. You can turn it off on a paid plan. The trial allows 2 hours at most. Source
- boat saves the disk every minute and at each stop. Memory, running programs and /tmp are not saved: a resume is like a reboot. A machine failure loses writes after the last good snapshot. Source
- Machines run in Germany, Finland and France only. Snapshot copies can be stored outside the EU. Source
- Public IPv6 and outbound IPv4. No dedicated IPv4. The IP changes at each resume. Source
- Hosts are AMD Ryzen 7 9700X and Ryzen 9 9950X. vCPUs are shared. No GPU. You cannot boot your own Docker image. Source
Questions people ask
Is Runloop cheaper than Fly.io Machines?
No. At Fly's lowest rate, a 1-year shared-CPU reservation, Fly costs less in all five scenarios on this page. One coding agent on workdays is $113 a month on Runloop and $15 on Fly. 50 agents in parallel are $5,268 and $487. Those Fly shared vCPUs guarantee 6.25% of each core. boat is $20 and $317 for the same two.
Which is faster, Runloop or Fly.io Machines?
We cannot say: Fly is not in the public benchmarks we use. In StarSling's run of 3 October 2026, Runloop did 8.93 runs a second on the V8 Web Tooling Benchmark and boat 28.06. On Fly, speed also depends on the CPU class you buy.
Can Runloop and Fly.io Machines run Docker?
Yes. Runloop has Docker-in-Docker blueprints, and on Fly you install Docker in your own image. Neither documents /dev/kvm for nested VMs or Android emulators. boat has Docker, systemd and /dev/kvm on every sandbox.
Do they keep my files when a sandbox stops?
Runloop keeps the disk through idle suspend on Pro or in disk snapshots on every plan; shutdown deletes it. Fly keeps one Volume, and the root file system with opt-in persist_rootfs, which Fly staff say not to trust with critical data. boat saves the disk every minute and at each stop and keeps it at least 14 days; memory and running programs are not saved.
Can I use Fly.io Machines as an AI agent sandbox?
Yes, but you build the sandbox yourself: the image with tools, browser and agents, a Volume for files, and the code that drives each agent. Fly's own agent sandbox is Sprites, sold separately. Runloop and boat ship agents and a desktop.
Where do they run?
Fly runs Machines in 17 regions on six continents. Runloop does not document its regions, but Enterprise can deploy into your own AWS, GCP or Azure account. boat runs in Germany, Finland and France, with no US or Asia region.
Try boat
Install the CLI, then run boat new.
7-day free trial