How to run Harbor and Terminal-Bench trials in cloud sandboxes
Install withruntime[harbor] and run harbor run -e withruntime.harbor:RuntimeEnvironment; each trial gets its own microVM.
On Runtime every Harbor trial runs in its own microVM, built from the task's own Dockerfile, and you pay for the CPU it uses. Harbor is the framework behind Terminal-Bench 2.0: it runs an agent on a task inside an environment, then runs the task's verifier and records a score. Docker on one machine is its default environment, so a run's parallelism stops where that machine's cores and disk do. With Runtime (withruntime.com) each trial is a machine with its own kernel, and a paid account runs 100 at once. Harbor 0.24.0 was the current PyPI release on 11 October 2026.
Run a task, then a benchmark
Terminalpip install "withruntime[harbor]"harbor run -t hello-world/hello-world -a oracle -e withruntime.harbor:RuntimeEnvironmentharbor run -d terminal-bench@2.0 -a oracle -e withruntime.harbor:RuntimeEnvironment --n-concurrent 8Harbor loads any environment by its import path (module.path:ClassName), so
this works with the Harbor you already have. The oracle agent runs each
task's reference solution, which checks the environment without paying for a
model; swap in your agent and model with -a and -m once it passes. The key
comes from RUNTIME_API_KEY or the one npx withruntime login saved.
What happens to a task's Dockerfile?
It is built once as a Runtime image and reused. The image is
named harbor-<task>, tagged with a hash of the environment, so a second run of
the same task starts from it at once and a changed Dockerfile builds a new one.
--force-build builds it again. To skip building altogether, pass a ready
image with --ek image=<name:tag>.
| Task setting | In the trial's sandbox |
|---|---|
Dockerfile or docker_image |
A Runtime image, built once and kept |
cpus, memory_mb, storage_mb |
The sandbox's vCPU, memory and disk |
workdir or the Dockerfile's WORKDIR |
Where every command starts |
| Commands as root | Run through the sandbox user's passwordless sudo |
| Network policies | no-network and host allow-lists, changeable mid-trial |
| A long trial | The sandbox runs as long as the trial does |
How many trials at once?
As many as --n-concurrent asks, up to your account's limit. An account without
credit runs four sandboxes at once; with credit it runs
100. Builds run one at a time until the account holds credit,
then four, so build a benchmark's images once before a wide run. Harbor stops
each sandbox when its trial ends, and runtime images ls lists the images the
run kept.
What does a Terminal-Bench pass cost?
Runtime bills the CPU a trial uses, not the CPUs it holds, plus memory while it runs. A trial that keeps 2 vCPUs half busy for 10 minutes with 4 GiB costs about $0.01, so 100 such trials cost about $0.92. Agents spend most of a trial waiting on the model, which costs the memory alone (pricing).
When a task's image lacks sudo
Harbor runs commands as root; a Runtime sandbox runs them as uid 1000 through
sudo. Runtime adds sudo to every image it builds, so this matters only for
a ready image passed with --ek image=. The environment checks at the start of
the trial and stops it with the line to add to the Dockerfile:
dockerfileRUN apt-get update && apt-get install -y sudo && echo '#1000 ALL=(ALL:ALL) NOPASSWD: ALL' > /etc/sudoers.d/runtimeDocker Compose tasks with sidecar services are not supported yet; a task with one image runs as it is.
Related
- Runtime for Harbor and Terminal-Bench, every option
- SWE-bench-style evals in parallel sandboxes
- What is SWE-bench?
- Inspect AI evals on Runtime
- RL environments in sandboxes
Sources
- harbor on PyPI, version 0.24.0,
harbor run --helpread 11 October 2026 - Terminal-Bench, read 11 October 2026
On 2 October 2026, with a key on production, harbor run of the hello-world
task with the oracle agent on this environment scored 1.0.
Facts on this page were checked on 11 October 2026.