How to run Inspect AI evals in cloud sandboxes
Install withruntime[inspect-ai] and set sandbox="runtime" on a task; every sample's tools then run in their own Firecracker microVM.
On Runtime each Inspect sample gets its own microVM, built from the Dockerfile or compose file the eval already has.
Inspect AI, the UK AI Security Institute's eval framework, runs tool calls in a
sandbox: local, or docker on your own machine. A wide agentic eval on Docker
is bounded by that one machine. With Runtime (withruntime.com) each sample is a
machine with its own kernel, an account with credit runs 100 at
once, and a sandbox waiting on the model costs $0.03125 an hour at 2 vCPU,
4 GiB. inspect-ai 0.3.278 was the current PyPI release on 11 October 2026.
Switch an eval to Runtime
Terminalpip install "withruntime[inspect-ai]"inspect eval task.py --sandbox runtimeInstalling the package registers runtime with Inspect through an entry
point, beside docker and local; nothing needs importing. In the task
itself:
Pythonfrom inspect_ai import Task, taskfrom inspect_ai.dataset import Samplefrom inspect_ai.scorer import includesfrom inspect_ai.solver import generate, use_toolsfrom inspect_ai.tool import bash, python@taskdef ctf_warmup(): return Task( dataset=[Sample(input="Find the flag in /workspace/flag.txt and print it.", target="FLAG{", files={"/workspace/flag.txt": "FLAG{runtime}"})], solver=[use_tools(bash(), python()), generate()], scorer=includes(), sandbox=("runtime", "Dockerfile"), )sandbox="runtime" alone uses Runtime's base image, Ubuntu with Python,
Node.js and git. Naming a Dockerfile or a single-service compose.yaml, as
for the Docker sandbox, builds it once as a Runtime image; every
sample starts from it, and it is rebuilt only when the Dockerfile or its build
context changes.
How many samples run at once?
Inspect starts as many sandboxes as --max-sandboxes allows. Without credit
an account runs four at once, and the sandbox asks Inspect for
that many by default; with credit raise it:
Terminalinspect eval task.py --sandbox runtime --max-sandboxes 64 --max-connections 64--max-connections bounds calls to the model; raise both together, or the
model becomes the queue.
Which settings carry over from Docker?
| Docker sandbox setting | On Runtime |
|---|---|
Dockerfile, compose build or image |
Built once as a Runtime image |
compose cpus, mem_limit |
The sandbox's vCPU and memory |
working_dir, environment, user |
Carried over |
network_mode: none |
No internet |
| Commands as root (Inspect's default) | Run through the sandbox user's passwordless sudo |
inspect sandbox cleanup docker |
inspect sandbox cleanup runtime |
Other options, such as a ready Runtime image or labels, go in
RuntimeSandboxEnvironmentConfig or under x-runtime on the compose service
(every option).
A compose file with more than one service is not supported yet: an eval with a sidecar, such as a separate victim host, still needs Docker.
Why a microVM for agentic evals?
Capability evals ask models to break out, escalate and exfiltrate on purpose.
A container shares the host's kernel, so a kernel exploit inside one reaches
the machine (container escape). A Firecracker
microVM has its own kernel behind a hypervisor, and a sample with
network_mode: none has no route out at all.
Related
Sources
- Inspect AI: Sandboxing, read 11 October 2026
- inspect-ai on PyPI, version 0.3.278, read 11 October 2026
On 1 October 2026, with a key on production, inspect-ai 0.3.275 passed 47 of 47 live checks on this sandbox, including a Dockerfile eval.
Facts on this page were checked on 11 October 2026.