# How to run Harbor and Terminal-Bench trials in cloud sandboxes

Install `withruntime[harbor]` and run `harbor run -e withruntime.harbor:RuntimeEnvironment`; each trial gets its own microVM.

**On Runtime every Harbor trial runs in its own microVM, built from the task's own Dockerfile, and you pay for the CPU it uses.**
Harbor is the framework behind Terminal-Bench 2.0: it runs an agent on a task
inside an environment, then runs the task's verifier and records a score.
Docker on one machine is its default environment, so a run's parallelism stops
where that machine's cores and disk do. With Runtime (withruntime.com) each
trial is a machine with its own kernel, and a paid account runs
100 at once. Harbor 0.24.0 was the current PyPI release on
11 October 2026.

## Run a task, then a benchmark

```bash no-run
pip install "withruntime[harbor]"
harbor run -t hello-world/hello-world -a oracle -e withruntime.harbor:RuntimeEnvironment
harbor run -d terminal-bench@2.0 -a oracle -e withruntime.harbor:RuntimeEnvironment --n-concurrent 8
```

Harbor loads any environment by its import path (`module.path:ClassName`), so
this works with the Harbor you already have. The `oracle` agent runs each
task's reference solution, which checks the environment without paying for a
model; swap in your agent and model with `-a` and `-m` once it passes. The key
comes from `RUNTIME_API_KEY` or the one `npx withruntime login` saved.

## What happens to a task's Dockerfile?

It is built once as a Runtime [image](/docs/images) and reused. The image is
named `harbor-<task>`, tagged with a hash of the environment, so a second run of
the same task starts from it at once and a changed Dockerfile builds a new one.
`--force-build` builds it again. To skip building altogether, pass a ready
image with `--ek image=<name:tag>`.

| Task setting                            | In the trial's sandbox                                  |
| --------------------------------------- | ------------------------------------------------------- |
| `Dockerfile` or `docker_image`          | A Runtime image, built once and kept                    |
| `cpus`, `memory_mb`, `storage_mb`       | The sandbox's vCPU, memory and disk                     |
| `workdir` or the Dockerfile's `WORKDIR` | Where every command starts                              |
| Commands as root                        | Run through the sandbox user's passwordless `sudo`      |
| Network policies                        | `no-network` and host allow-lists, changeable mid-trial |
| A long trial                            | The sandbox runs as long as the trial does              |

## How many trials at once?

As many as `--n-concurrent` asks, up to your account's limit. An account without
credit runs four sandboxes at once; with credit it runs
100. Builds run one at a time until the account holds credit,
then four, so build a benchmark's images once before a wide run. Harbor stops
each sandbox when its trial ends, and `runtime images ls` lists the images the
run kept.

## What does a Terminal-Bench pass cost?

Runtime bills the CPU a trial uses, not the CPUs it holds, plus memory while
it runs. A trial that keeps 2 vCPUs half busy for 10 minutes with 4 GiB costs
about $0.01, so 100
such trials cost about
$0.92. Agents
spend most of a trial waiting on the model, which costs the memory alone
([pricing](/docs/pricing)).

## When a task's image lacks sudo

Harbor runs commands as root; a Runtime sandbox runs them as uid 1000 through
`sudo`. Runtime adds `sudo` to every image it builds, so this matters only for
a ready image passed with `--ek image=`. The environment checks at the start of
the trial and stops it with the line to add to the Dockerfile:

```dockerfile no-run
RUN apt-get update && apt-get install -y sudo && echo '#1000 ALL=(ALL:ALL) NOPASSWD: ALL' > /etc/sudoers.d/runtime
```

Docker Compose tasks with sidecar services are not supported yet; a task with
one image runs as it is.

## Related

- [Runtime for Harbor and Terminal-Bench](/docs/harbor), every option
- [SWE-bench-style evals in parallel sandboxes](/use-cases/agent-evals-and-swe-bench)
- [What is SWE-bench?](/glossary/swe-bench)
- [Inspect AI evals on Runtime](/integrations/inspect-ai)
- [RL environments in sandboxes](/use-cases/rl-environments)

## Sources

- [harbor on PyPI](https://pypi.org/project/harbor/), version 0.24.0, `harbor run --help` read 11 October 2026
- [Terminal-Bench](https://www.tbench.ai/), read 11 October 2026

On 2 October 2026, with a key on production, `harbor run` of the hello-world
task with the oracle agent on this environment scored 1.0.

Facts on this page were checked on 11 October 2026.