Runtime for Harbor and Terminal-Bench

Run Harbor trials, Terminal-Bench 2.0 among them, each in its own Runtime microVM, with the task and the agent unchanged.

RuntimeEnvironment is a Harbor environment, like its Docker, E2B or Modal environments. Harbor loads it by import path, so it works with the Harbor you already have. Each trial gets a Firecracker microVM with its own kernel, built from the task's own Dockerfile or docker_image, and you pay for the CPU it uses rather than the CPUs it holds.

Updated October 25, 2026

On this page

Install#

Terminalpip install "withruntime[harbor]"

Harbor needs Python 3.12 or newer. The key comes from RUNTIME_API_KEY, or from npx withruntime login on this machine, as in the rest of the SDK.

Run a benchmark#

Pass the environment's import path to -e, and its options with --ek:

Terminalharbor run -t hello-world/hello-world -a oracle -e withruntime.harbor:RuntimeEnvironmentharbor run -d terminal-bench@2.0 -a oracle -e withruntime.harbor:RuntimeEnvironment --n-concurrent 8

The oracle agent runs each task's reference solution against its tests, so it checks the environment without calling a model. Swap in your own agent and model with -a and -m.

In a job config, or from Python:

Pythonimport asynciofrom harbor.job import Jobfrom harbor.models.job.config import JobConfigfrom harbor.models.trial.config import AgentConfig, EnvironmentConfig, TaskConfigasync def main():    job = await Job.create(        JobConfig(            environment=EnvironmentConfig(                import_path="withruntime.harbor:RuntimeEnvironment",            ),            agents=[AgentConfig(name="oracle")],            tasks=[TaskConfig(name="hello-world/hello-world")],        )    )    await job.run()asyncio.run(main())

Options#

Every option is optional.

`--ek`	What it does
`funding`	Accepted and ignored: the free usage is spent first, then prepaid credit
`region`	The region to run in; left out, the default
`image`	A Runtime image (id, name or `name:tag`) to start every trial from, instead of building the task's
`build`	Build limits, such as `build='{"disk_mib": 16384}'`: `vcpu`, `memory_mib`, `disk_mib`, `max_image_mib`, `timeout_seconds`
`labels`	Your own labels on each sandbox and image
`base_url`	Another API address; `RUNTIME_API_URL` does the same

What runs in the sandbox#

Harbor	Runtime
`environment/Dockerfile`, `docker_image`	A custom image, built once and kept as `harbor-<task>` with a tag from a hash of the environment. Reruns start from it; `--force-build` builds it again
`cpus`, `memory_mb`, `storage_mb`	`vcpu`, `memory_mib`, `disk_mib`. A sandbox without credit is at most 2 vCPU, 4 GiB and 10 GiB of disk; a larger request without credit runs at that size, with a warning
Commands	`bash -c` (`sh -c` in an image without bash), in the task's `workdir` or the Dockerfile's `WORKDIR`, with the task's and the agent's environment. Output streams, however large
Users	Root by default, as in Docker, and any named user, through `sudo`
Files	Uploads and downloads of any size, anywhere in the machine, as root
Network policies	`no-network`, and allow-lists of hostnames, `*.domain` wildcards, IPv4 addresses and IPv4 CIDR ranges, changed while the trial runs
Long trials	The sandbox runs while the trial works, however long it takes

Root and sudo. Commands in a Runtime sandbox run as the sandbox user, uid 1000, and Harbor runs them as root. The environment runs root commands through the sandbox user's passwordless sudo, which Runtime's base image has. Runtime adds it to every image it builds, including images built from a public image. An image built before 25 September 2026 may lack it. Rebuilding adds it. The environment checks when the trial starts. If sudo is missing, it stops the trial and shows this line to add to the task's Dockerfile:

dockerfileRUN apt-get update && apt-get install -y sudo && echo '#1000 ALL=(ALL:ALL) NOPASSWD: ALL' > /etc/sudoers.d/runtime

Not supported: Docker Compose tasks with sidecar services, GPUs, Windows tasks, IPv6 allow-list entries, and Harbor's stream mode, which Harbor offers only to its built-in environments.

Limits to plan for#

The trial runs four sandboxes at the same time, so keep --n-concurrent at 4 or below without credit. A paid account runs more; see pricing.
Each task is its own image. Builds run one at a time until your account holds credit, then four. An account without credit keeps its first three images free; after that they are billed as stored images (pricing). runtime images ls lists them and runtime images rm removes them.
Harbor stops each sandbox when its trial ends.

What was verified#

On 25 September 2026 the environment ran under Harbor 0.23.0 against a fake Runtime API whose sandbox is a folder on the test machine: Harbor loading it by import path, building an image once and reusing it, --force-build, root and uid 1000 commands through a real sudo, the Dockerfile's WORKDIR, task and per-command variables, streamed output, a 40 KB command with 80 variables, timeouts, directory and file transfers, the refusal when sudo is missing, the trial's sizes, network policies and cleanup. On the same fake, a whole harbor run -a oracle of Harbor's hello-world example task, with its verifier changed to check the file without installing packages on the test machine, scored 1.

Try it on your own work. Sign up, then paste the setup prompt into your coding agent.

Start free, no card