# How to run Inspect AI evals in cloud sandboxes

Install `withruntime[inspect-ai]` and set `sandbox="runtime"` on a task; every sample's tools then run in their own Firecracker microVM.

**On Runtime each Inspect sample gets its own microVM, built from the Dockerfile or compose file the eval already has.**
Inspect AI, the UK AI Security Institute's eval framework, runs tool calls in a
sandbox: `local`, or `docker` on your own machine. A wide agentic eval on Docker
is bounded by that one machine. With Runtime (withruntime.com) each sample is a
machine with its own kernel, an account with credit runs 100 at
once, and a sandbox waiting on the model costs $0.03125 an hour at 2 vCPU,
4 GiB. inspect-ai 0.3.278 was the current PyPI release on 11 October 2026.

## Switch an eval to Runtime

```bash no-run
pip install "withruntime[inspect-ai]"
inspect eval task.py --sandbox runtime
```

Installing the package registers `runtime` with Inspect through an entry
point, beside `docker` and `local`; nothing needs importing. In the task
itself:

```python check
from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import includes
from inspect_ai.solver import generate, use_tools
from inspect_ai.tool import bash, python


@task
def ctf_warmup():
    return Task(
        dataset=[Sample(input="Find the flag in /workspace/flag.txt and print it.", target="FLAG{",
                        files={"/workspace/flag.txt": "FLAG{runtime}"})],
        solver=[use_tools(bash(), python()), generate()],
        scorer=includes(),
        sandbox=("runtime", "Dockerfile"),
    )
```

`sandbox="runtime"` alone uses Runtime's base image, Ubuntu with Python,
Node.js and git. Naming a `Dockerfile` or a single-service `compose.yaml`, as
for the Docker sandbox, builds it once as a Runtime [image](/docs/images); every
sample starts from it, and it is rebuilt only when the Dockerfile or its build
context changes.

## How many samples run at once?

Inspect starts as many sandboxes as `--max-sandboxes` allows. Without credit
an account runs four at once, and the sandbox asks Inspect for
that many by default; with credit raise it:

```bash no-run
inspect eval task.py --sandbox runtime --max-sandboxes 64 --max-connections 64
```

`--max-connections` bounds calls to the model; raise both together, or the
model becomes the queue.

## Which settings carry over from Docker?

| Docker sandbox setting                   | On Runtime                                         |
| ---------------------------------------- | -------------------------------------------------- |
| `Dockerfile`, compose `build` or `image` | Built once as a Runtime image                      |
| compose `cpus`, `mem_limit`              | The sandbox's vCPU and memory                      |
| `working_dir`, `environment`, `user`     | Carried over                                       |
| `network_mode: none`                     | No internet                                        |
| Commands as root (Inspect's default)     | Run through the sandbox user's passwordless `sudo` |
| `inspect sandbox cleanup docker`         | `inspect sandbox cleanup runtime`                  |

Other options, such as a ready Runtime image or labels, go in
`RuntimeSandboxEnvironmentConfig` or under `x-runtime` on the compose service
([every option](/docs/inspect)).

A compose file with more than one service is not supported yet: an eval with a
sidecar, such as a separate victim host, still needs Docker.

## Why a microVM for agentic evals?

Capability evals ask models to break out, escalate and exfiltrate on purpose.
A container shares the host's kernel, so a kernel exploit inside one reaches
the machine ([container escape](/glossary/container-escape)). A Firecracker
microVM has its own kernel behind a hypervisor, and a sample with
`network_mode: none` has no route out at all.

## Related

- [Runtime for Inspect AI](/docs/inspect), every option
- [Run untrusted LLM code](/use-cases/run-untrusted-llm-code)
- [Harbor and Terminal-Bench on Runtime](/integrations/harbor)

## Sources

- [Inspect AI: Sandboxing](https://inspect.aisi.org.uk/sandboxing.html), read 11 October 2026
- [inspect-ai on PyPI](https://pypi.org/project/inspect-ai/), version 0.3.278, read 11 October 2026

On 1 October 2026, with a key on production, inspect-ai 0.3.275 passed 47 of
47 live checks on this sandbox, including a Dockerfile eval.

Facts on this page were checked on 11 October 2026.