Runloop vs Upstash Box
Runloop gives each agent a micro-VM devbox that can run Docker and is built from a Dockerfile. Upstash Box gives it a light container that pauses when idle, bills only the CPU it uses, and wakes on the next request.
The companies and products
Runloop
Runloop is a US platform of sandboxes, called devboxes, for building, running and testing AI coding agents.
You get Linux devboxes built from Dockerfile-style blueprints, disk snapshots, HTTPS tunnels, agent gateways that hide API keys, and Axons, event streams that drive agents such as Claude Code. Each devbox runs in a micro-VM on Runloop's own bare-metal hypervisor, with a container layer inside. It is built for teams shipping coding agents, plus evaluation and RL fine-tuning work on the Enterprise plan.
- Compliance
- SOC 2 Type 2
- Architectures
- x86_64 and arm64 base blueprints
- Own-cloud deploy
- Deploy to VPC on AWS, GCP and Azure (Enterprise)
- Regions
- Not documented or selectable
- Company
- Runloop AI, Inc.
Upstash Box
Upstash Box is a cloud computer for AI agents from Upstash, the company known for serverless Redis, Vector and QStash.
Each box is a Docker container with its own filesystem, shell, git and an optional coding agent such as Claude Code, Codex or OpenCode, which you drive from TypeScript or Python SDKs. A box pauses when it sits idle, keeps its files, and you pay for the CPU it actually uses. Upstash built it to run its own Context7 service and opened it to everyone in March 2026.
Which use cases each one fits
For each use case, Battleships prices every sandbox provider on the same workload and sorts them from the cheapest monthly bill up. “20th cheapest of 367” means 19 of the other 366 providers run that workload, with every feature it needs, for less. “With limits” means the provider runs it but misses something, so it is listed after every full match. Open a use case for its whole ranking and what each provider misses.
| Use case and its workload | Runloop: price rank | Upstash Box: price rank | boat: price rank |
|---|---|---|---|
| Software factory 100 at once · 4 vCPU / 8 GB · 2,200 sessions of 8 h a month Runloop fits fully because Docker runs inside its devboxes, while Upstash Box is a compromise with no Docker inside; boat ranks 3rd with Docker and /dev/kvm. | 20th cheapest of 367 $8,152 a month | 61st of 367, with limits $464 a month | 3rd cheapest of 367 $634 a month |
| Always-on AI employees 50 machines 24/7 · 2 vCPU / 4 GB | 32nd cheapest of 366 $10,194 a month | 103rd of 366, with limits $475 a month | 5th cheapest of 366 $657 a month |
| Code interpreter bursts 100 at once · 1 vCPU / 2 GB · 200,000 sessions of 1 min a month Upstash Box fits short code runs well, since an idle box costs no CPU, and boat is a compromise here because its start limits queue very large bursts. | 44th cheapest of 345 $525 a month | 10th cheapest of 345 $119 a month | 186th of 345, with limits $2,000 a month |
| Sandboxes for your users 300 at once · 2 vCPU / 4 GB · 30,000 sessions of 23 min a month Upstash Box and boat sit at the top for sandboxes you hand to your users, and Runloop fits too at a higher bill. | 26th cheapest of 361 $3,301 a month | 2nd cheapest of 361 $222 a month | Cheapest of 361 $209 a month |
| RL rollouts 1,000 at once · 2 vCPU / 4 GB · 1,000,000 sessions of 3 min a month Runloop ranks 4th for RL rollouts, with large waves and RL fine-tuning on Enterprise, while Upstash Box is a compromise without Docker inside. | 4th cheapest of 360 $15,922 a month | 352nd of 360, with limits $8,068 a month | 34th of 360, with limits $2,000 a month |
| Agent evals 250 at once · 4 vCPU / 8 GB · 15,000 sessions of 20 min a month | 81st cheapest of 364 $2,562 a month | 52nd cheapest of 364 $1,011 a month | 10th cheapest of 364 $200 a month |
| CI runners 40 at once · 8 vCPU / 16 GB · 6,000 sessions of 12 min a month All three fit CI, and boat's faster measured CPUs finish builds and tests sooner than Runloop's. | 121st cheapest of 377 $1,554 a month | 120th cheapest of 377 $1,520 a month | 4th cheapest of 377 $86 a month |
| 24/7 dev boxes 1 machine 24/7 · 4 vCPU / 8 GB A flat-price keep-alive box makes Upstash Box a cheap light dev box, while a Runloop devbox needs an idle policy to run all month. | 113th cheapest of 368 $410 a month | 10th cheapest of 368 $20 a month | 18th cheapest of 368 $26 a month |
| Computer-use agents 50 at once · 4 vCPU / 8 GB · 20,000 sessions of 15 min a month Runloop ships a desktop your agent can control, Upstash Box only a headless browser, and boat streams a full Linux desktop at 60fps. | 39th cheapest of 369 $2,306 a month | 73rd of 369, with limits $76 a month | 6th cheapest of 369 $200 a month |
| Browser-use agents 50 at once · 2 vCPU / 4 GB · 30,000 sessions of 10 min a month | 40th cheapest of 367 $1,153 a month | 49th of 367, with limits $38 a month | 7th cheapest of 367 $200 a month |
| Scraping & bots (startup scale) 50 at once · 1 vCPU / 2 GB · 100,000 sessions of 3 min a month | 68th cheapest of 346 $781 a month | 37th cheapest of 346 $400 a month | 6th cheapest of 346 $90 a month |
| Mass scraping & bots (undetectable) 1,000 at once · 1 vCPU / 2 GB · 3,000,000 sessions of 2 min a month | 266th of 338, with limits $15,611 a month | 200th of 338, with limits $8,000 a month | 71st of 338, with limits $1,800 a month |
| Parallel red teaming 200 at once · 2 vCPU / 4 GB · 5,000 sessions of 30 min a month Runloop fits with hostname allowlists and Docker inside, Upstash Box is a compromise, and boat gives each sandbox a full kernel with nested virtualization. | 10th cheapest of 360 $770 a month | 225th of 360, with limits $259 a month | 2nd cheapest of 360 $200 a month |
| GPU eval jobs 10 at once · 8 vCPU / 32 GB · 500 sessions of 30 min a month | Can't run it | Can't run it | Can't run it |
| macOS agents 5 at once · 4 vCPU / 8 GB · 1,000 sessions of 20 min a month | Can't run it | Can't run it | Can't run it |
Key features
| Feature | Runloop | Upstash Box | boat |
|---|---|---|---|
| Isolation | Micro-VM on a custom bare-metal hypervisor, plus a container layer | One Docker container per box on AWS | Full VM with its own kernel on dedicated AMD Ryzen 9 9950X hosts |
| Docker inside | Yes, on the Docker-in-Docker blueprints, as root | Not documented; the box is itself a Docker container | Yes, with root and systemd, as on any Linux server |
| Nested virtualization | Not documented; Docker-in-Docker only | No; boxes are containers | Yes: /dev/kvm for Android emulators and VMs inside |
| Images | Blueprints built from Dockerfiles, x86_64 or arm64, plus disk snapshots | Custom images coming soon; reuse a setup through filesystem snapshots | Templates from named snapshots, no Dockerfile builds, x86_64 only |
| Disk | Up to 64 GiB | 5, 10 or 20 GB by size, billed per GB-month | 50 GB at the default 4 vCPU / 8 GB size, 12 to 251 GB by size |
| Keeping files | Kept through suspend (Pro plan) or disk snapshots you take; shutdown ends the devbox | Pauses itself when idle and keeps its files; storage keeps billing while paused | Disk saved every minute and on stop, on every plan; a stopped sandbox costs nothing |
| Long runs | Fixed lifetime 1 hour by default, 48 hours max; an idle policy replaces it | No session cap documented; a keep-alive box stays on all month | No session cap on paid plans; default TTL 1 hour, changeable |
| Wake on request | Suspend after idle and wake on an HTTP request to the tunnel (Pro plan) | A request to a public URL resumes a paused box, no plan fee needed | No; a stopped sandbox is started by API |
| Inbound access | One HTTPS tunnel per devbox, open or bearer token; SSH through a TLS proxy; no public IP | HTTPS URL per port, public unless you add a token or basic auth; SSH through a gateway; no raw TCP | HTTPS URLs for up to 50 ports, token-protected by default, raw TCP, dedicated IPv6 or IPv4 |
| Outbound rules | Hostname allowlist with wildcards, or block all | Allow all, deny all, or allow and deny by domain and CIDR, changeable on a running box | Open internet, no provider-side allowlist |
| Desktop and browser | Desktop and browser your agent can control | Managed headless Chromium with AI page actions and CDP; no desktop | Linux desktop streamed at 60fps, embeddable in your app, with Chromium and computer-use tools |
| Hidden API keys | Agent gateways for any API endpoint with bearer, header or basic auth | Host proxy injects headers into outbound HTTPS on port 443, set when the box is created | Credentials proxy for model providers and any HTTPS API; switch a credential to another account when one runs out |
| Coding agents | Claude Code, Codex, OpenCode, Gemini CLI, DeepAgents, driven through Axons and Broker | Claude Code, Codex and OpenCode built in, with run and stream APIs, webhooks and scheduled prompts | Claude Code, Codex, pi, OpenCode, Prime Agent, Kimi Code, Mistral Vibe, with a prompt, steer and events API |
| Where it runs | Regions not documented; Deploy to VPC on AWS, GCP or Azure (Enterprise) | US only (AWS us-east-1), more regions planned | EU only (Germany, Finland, France), EU data residency |
Every feature Battleships tracks (43)
| Feature | Runloop | Upstash Box | boat |
|---|---|---|---|
| Full VM (own kernel) | Yes | No | Yes |
| Docker inside | Yes | No | Yes |
| Nested virtualization | No | No | Yes |
| Root | Yes | Not published | Yes |
| Systemd | Not published | Not published | Yes |
| Desktop GUI | Yes | No | Yes |
| Browser | Yes | Yes | Yes |
| Computer-use API | Yes | No | Yes |
| SSH | Yes | Yes | Yes |
| Persistent disk | Yes | Yes | Yes |
| Pause/resume | Yes | Yes | Yes |
| Automatic snapshots | Yes | No | Yes |
| Fork/clone | Yes | Yes | Yes |
| Live fork (no pause) | Yes | Yes | Yes |
| Memory fork | No | No | No |
| Custom image (Docker or snapshot) | Yes | Yes | Yes |
| ≥24 h sessions | Yes | Yes | Yes |
| Idle auto-stop | Yes | Yes | No |
| Wake on request | Yes | Yes | No |
| HTTPS preview URLs | Yes | Yes | Yes |
| Raw TCP inbound | No | No | Yes |
| Public IPv4 | No | No | No |
| Egress allowlist | Yes | Yes | No |
| Secret proxy | Yes | Yes | Yes |
| Secret proxy for any API | Yes | Yes | Yes |
| Preinstalled agents | Yes | Yes | Yes |
| Agent harness API | Yes | Yes | Yes |
| MCP server | Yes | Yes | No |
| GPU | No | No | No |
| Windows | No | No | No |
| Arm64 | Yes | Yes | No |
| Self-hosting | No | No | No |
| BYOC | Yes | No | No |
| Open source | No | No | No |
| Scoped API keys | Yes | Not published | Yes |
| Spend limits | Not published | Not published | Yes |
| Usage per sandbox (API) | No | Yes | Yes |
| Audit logs | Not published | Not published | No |
| SSO | Yes | No | No |
| SOC 2 | Yes | Yes | No |
| HIPAA | Not published | No | No |
| EU data residency | No | No | Yes |
| Zero data retention | Not published | Not published | Yes |
Performance
On the hpc-sandbox-benchmarks Node.js build-and-test loop at 4 vCPU / 8 GB (run 36498291449, 2026-09-29), boat scored 26.69 runs a second and Runloop 8.29, about 3.2 times apart. Upstash Box has no result in that test, so there is no comparable measurement for it. The test measures CPU work only, not start time or network speed.
Node.js build-and-test loop at 4 vCPU / 8 GB, higher is better. Measured with the open hpc-sandbox-benchmarks suite. Upstash Box has no published run in this suite yet, so there is no like-for-like number to compare.
Pricing in five scenarios
Each bill is priced from the providers' public price lists by the open Battleships engine, with the cheapest plan that runs the scenario.
| Scenario | Runloop, a month | Upstash Box, a month | boat, a month |
|---|---|---|---|
| One coding agent, workdays 4 vCPU / 8 GB, 50 GB disk, 8 hours a day, 22 days a month, files kept between days | $113 · Basic | $21 · Pay as You Go | $20 · $20/mo |
| One agent that never stops 4 vCPU / 8 GB, 50 GB disk, running 24/7 all month | $429 · Basic | $21 · Pay as You Go | $26 · $20/mo |
| 50 agents in parallel 50 machines of 4 vCPU / 8 GB at once, 8-hour sessions, 1,100 sessions a month, Docker inside | $5,268 · Basic | $900 · Pay as You Go · with limits | $317 · $20/mo × 3 seats |
| Sandboxes for your users 2 vCPU / 4 GB, 30,000 sessions of 10 minutes a month, up to 300 at once | $1,441 · Basic | $306 · Pay as You Go | $200 · $100/mo × 2 seats |
| 1,000 short jobs at once 2 vCPU / 4 GB, 1,000 at once, 3 minutes each, 200,000 jobs a month | $3,184 · Basic | $1,822 · Pay as You Go | $2,000 · $2000/mo · with limits |
Runloop: Runloop bills per second for the CPU and memory you allocate while a devbox is starting, running, suspending or resuming, and bills its disk for the devbox's whole life. Basic has no plan fee; Pro adds $250 a month that is not usage credit and unlocks suspend and resume.
Upstash Box: A standard box is billed only for the CPU it actually uses, at $0.10, $0.20 or $0.40 per active CPU hour for Small, Medium and Large, plus $0.10 per GB-month of storage, and RAM is free. A keep-alive box costs a flat $8, $16 or $32 a month with CPU and storage included.
Upstash Box: Plans start at $20 a month and the whole fee is usage credit, billed per second while a sandbox runs. A stopped sandbox costs nothing and keeps its disk.
Limits in those scenarios
Upstash Box has no Docker inside, so the 50-agent scenario does not fit there, and its disks stop at 5, 10 or 20 GB by size. Runloop fits every scenario, but its fixed lifetime tops out at 48 hours unless an idle policy replaces it, its disk tops out at 64 GiB, and suspend needs Pro. Neither publishes an egress price. boat's limit is start rate: for 1,000 short jobs at once, even the $2,000 plan starts at most 90 a minute, so the burst is queued.
| Scenario | Runloop | Upstash Box | boat |
|---|---|---|---|
| One coding agent, workdays |
|
| None that bite |
| One agent that never stops |
|
| None that bite |
| 50 agents in parallel |
|
| None that bite |
| Sandboxes for your users |
|
| None that bite |
| 1,000 short jobs at once |
|
|
|
Which one should you use?
Use Runloop when
your agents need Docker, Dockerfile images or arm64, when you launch large RL or eval waves, or when devboxes must run inside your own AWS, GCP or Azure account.
- Docker and Dockerfile images. Useful when agents run databases or build containers: Runloop runs Docker inside a devbox and builds images from Dockerfiles.
- Run it in your own cloud. On the Enterprise plan, Runloop can run devboxes inside your own AWS, GCP or Azure account.
- Large RL and eval waves. Good for RL rollouts: Runloop starts large waves of Docker-capable devboxes and sells RL fine-tuning on Enterprise.
Use Upstash Box when
your agents are light, mostly wait, need to wake on an incoming request or stay on all month at a flat price, and can run without Docker in the US.
- Idle costs no CPU, and a request wakes the box. Good for agents that wait a lot: an idle box costs no CPU, and a request to its URL wakes it, with no plan fee.
- Flat price for an always-on box. A keep-alive box stays on all month for a flat $8, $16 or $32 by size, whether it is busy or not.
- Outbound rules by domain and IP range. Useful when an agent may only reach a short list of domains or CIDR ranges, changed on a running box.
Where boat fits
If your agents do real work for hours, with installs, builds, Docker, emulators or a desktop, look at boat. Each sandbox is a full VM on dedicated AMD Ryzen 9 9950X hosts that keeps its disk on every plan, and it ran about three times faster than Runloop on the same CPU test.
Docker and /dev/kvm in every sandbox
An Upstash box is a container with no Docker inside, so the 50-agent scenario does not fit there. Runloop runs Docker but documents no nested virtualization, so Android emulators and VMs do not start.
Files kept every minute, on every plan
A Runloop devbox is gone once it shuts down, so keeping work means the Pro plan's $250 fee for suspend or snapshot code you maintain. Upstash keeps files, but disks stop at 20 GB.
A real desktop and real ports
Upstash has only a headless browser, so a computer-use agent has no screen to see. Runloop reaches devboxes through one HTTPS tunnel with no raw TCP; boat streams a 60fps desktop and opens 50 ports plus raw TCP.
Use boat when your agents need a real machine for hours: Docker, emulators, a desktop and open ports, on CPUs about three times faster than Runloop's, with files kept on every plan. Compare it one to one: boat vs Runloop and boat vs Upstash Box.
Questions people ask
Is Runloop cheaper than Upstash Box?
No, Upstash Box costs less in every scenario on this page. One agent that never stops costs $21 a month on Upstash Box and $429 on Runloop, and 1,000 short jobs at once cost $1,822 against $3,184. But Upstash cannot run the 50-agent scenario, because it has no Docker inside.
Which is faster, Runloop or Upstash Box?
Runloop scored 8.29 runs a second on the hpc-sandbox-benchmarks Node.js test at 4 vCPU / 8 GB, and Upstash Box has no result in that test, so they cannot be compared. boat scored 26.69 on the same test.
Can Upstash Box run Docker?
Its docs do not offer it: each box is itself a Docker container, and custom images are listed as coming soon. Runloop runs Docker on its Docker-in-Docker blueprints, and boat runs Docker and systemd as on any Linux server.
Can a Runloop devbox or an Upstash box run all month?
Yes, both. Runloop's fixed lifetime stops at 48 hours, but an idle policy replaces it, and Upstash keeps a keep-alive box on all month at a flat price. boat has no session cap on paid plans.
Which one has a desktop for computer use?
Runloop ships a desktop and a browser your agent can control. Upstash Box has a headless Chromium with AI page actions and no desktop. boat streams a full Linux desktop at 60fps that you can embed in your app.
Where do Runloop and Upstash Box run?
Upstash Box runs in AWS us-east-1, with more regions planned. Runloop does not document its regions, but on Enterprise it deploys into your own AWS, GCP or Azure account. boat runs in the EU, in Germany, Finland and France.
Try boat
Install the CLI, then run boat new.
7-day free trial