# How to run an agent's pytest suite in an isolated sandbox

Upload the project to a microVM, install its requirements, run pytest there and download the JUnit XML and coverage report.

**Runtime starts the machine in 351 ms at the median and bills only the CPU the
tests burn.** That start time is from create request to first Python result,
measured over 20 runs on 24 September 2026. A 2 vCPU, 4 GiB sandbox costs
$0.08 an hour with both CPUs busy, so a five-minute suite that keeps them busy
costs under a cent. Code an agent wrote runs behind its own Linux kernel, never
on your CI runner or your laptop.

## Run the suite and bring the reports back

The project folder travels in as one compressed archive. Install first, then
cut the network so the tests themselves reach nothing:

```ts check
import { Sandbox } from "withruntime";

await using sbx = await Sandbox.create({ timeoutSeconds: 1800 });
await sbx.files.upload("./project", "/workspace/project");
await sbx.exec("cd project && pip install -q -r requirements.txt pytest==9.1.1 pytest-cov==7.1.0", {
  check: true,
  timeoutMs: 600_000,
});

await sbx.network.set({ internet: false });
const run = await sbx.exec(
  "cd project && python3 -m pytest -q --junitxml=report.xml --cov=. --cov-report=html",
  { timeoutMs: 1_200_000, onStdout: (text) => process.stdout.write(text) },
);

await sbx.files.download("/workspace/project/report.xml", "./report.xml");
await sbx.files.download("/workspace/project/htmlcov", "./htmlcov");
process.exitCode = run.exitCode ?? 1;
```

```python check
import sys
from withruntime import Sandbox

with Sandbox.create(timeout_seconds=1800) as sbx:
    sbx.files.upload("./project", "/workspace/project")
    sbx.exec("cd project && pip install -q -r requirements.txt pytest==9.1.1 pytest-cov==7.1.0",
             check=True, timeout_ms=600_000)

    sbx.network.set(internet=False)
    run = sbx.exec("cd project && python3 -m pytest -q --junitxml=report.xml --cov=. --cov-report=html",
                   timeout_ms=1_200_000, on_stdout=sys.stdout.write)

    sbx.files.download("/workspace/project/report.xml", "report.xml")
    sbx.files.download("/workspace/project/htmlcov", "htmlcov")

sys.exit(1 if run.exit_code is None else run.exit_code)
```

- **Streaming:** `onStdout` prints each line as pytest writes it, and the
  result then keeps the whole output instead of the first 64 KiB.
- **Timeouts:** a command gets 60 seconds unless you pass `timeoutMs`. A suite
  that runs longer returns `timedOut: true` with the output so far; it never
  throws.
- **Coverage:** `--cov-report=html` writes the `htmlcov` folder, which comes
  back in one download call.

## Why the network goes off after the install

Tests written by a model sometimes call real services, read environment
variables or download fixtures. Once `network.set({ internet: false })` runs,
every outbound connection is refused, so a test that depends on the internet
fails in the report instead of quietly reaching it. The rule is enforced on the
host: `sudo` inside the sandbox cannot undo it
([turn off sandbox internet](/how-to/turn-off-sandbox-internet)).

If the suite needs one service, such as a package index or your staging API,
use an allow list instead of switching the internet off:
`network.set({ internet: true, allow: ["staging.example.com"] })`.

## Read the result from the exit code

pytest's exit code says what happened without parsing any output. Pass it to
the agent along with the failing tests from `report.xml`:

| Exit code | pytest's meaning                                     | What to tell the agent                     |
| --------- | ---------------------------------------------------- | ------------------------------------------ |
| 0         | All tests collected and passed                       | Done                                       |
| 1         | Tests ran and some failed                            | Fix the failures listed in the report      |
| 2         | Execution was interrupted                            | Rerun; check for a hung test               |
| 3         | Internal error, or a plugin raised while importing   | Fix the environment, not the code          |
| 4         | Usage error, or a `conftest.py` that fails to import | Fix the command or the conftest            |
| 5         | No tests were collected                              | The agent wrote no tests, or misnamed them |
| 6         | Too many warnings (`--max-warnings`)                 | Clean up warnings                          |

Exit code 5 matters for agents: a model asked to "make the tests pass" can
delete every test, and pytest then reports 5, not 0.

## Use more than one CPU

`pytest-xdist` spreads tests over worker processes with `-n auto`. Ask for the
vCPUs to match when you create the sandbox:

```ts check
import { Sandbox } from "withruntime";

await using sbx = await Sandbox.create({ vcpu: 4, memoryMiB: 8192, timeoutSeconds: 1800 });
await sbx.files.upload("./project", "/workspace/project");
await sbx.exec(
  "cd project && pip install -q -r requirements.txt pytest==9.1.1 pytest-xdist==3.8.0",
  {
    check: true,
    timeoutMs: 600_000,
  },
);
await sbx.exec("cd project && python3 -m pytest -n auto -q", {
  timeoutMs: 1_200_000,
  onStdout: (text) => process.stdout.write(text),
});
```

CPU is billed on what the workers use, so four workers that finish in a
quarter of the time cost about the same CPU as one worker that runs four times
longer. Trial sandboxes go up to 2 vCPU and 4 GiB.

## Skip the install on every run

When the same dependencies install for every run, build them into a
[custom image](/docs/images). Building is free, and each later sandbox starts
with pytest and the project's packages in place:

```python check
from withruntime import Runtime

runtime = Runtime()
with open("project/requirements.txt") as handle:
    requirements = [line.strip() for line in handle if line.strip() and not line.startswith("#")]

runtime.images.build(
    name="pytest-env",
    recipe={"pip": ["pytest==9.1.1", "pytest-cov==7.1.0", "pytest-xdist==3.8.0", *requirements]},
    on_log=lambda line: print(line["text"]),
)

with runtime.sandboxes.create(image="pytest-env") as sbx:
    sbx.files.upload("./project", "/workspace/project")
    print(sbx.exec("cd project && python3 -m pytest -q", timeout_ms=600_000).stdout)
```

Ubuntu's own `python3-pytest` package on noble is pytest 7.4.4. Installing from
PyPI gets the current 9.1.1.

## What is in the sandbox already

| Item         | Detail                                                               |
| ------------ | -------------------------------------------------------------------- |
| Python       | 3.12, with `pip`, `venv` and uv; `pip install` works without a venv  |
| Preinstalled | NumPy, pandas and matplotlib in the default image                    |
| Disk         | 4 GiB by default, about 2.5 GiB free; raise it with `diskMiB`        |
| Root         | `sudo` works, for `apt-get install` of system libraries a test needs |

See [the sandbox environment](/docs/sandbox-environment) for the full list.

## Related

- [Run untrusted Python code safely](/languages/python)
- [A sandbox for a coding agent](/use-cases/coding-agent-sandbox)
- [Agent evals and SWE-bench](/use-cases/agent-evals-and-swe-bench)
- [Jest tests across parallel sandboxes](/integrations/jest)

## Sources

Checked 25 September 2026.

- [pytest exit codes](https://docs.pytest.org/en/stable/reference/exit-codes.html):
  the seven codes, 0 to 6
- [pytest on PyPI](https://pypi.org/project/pytest/): version 9.1.1, Python 3.10
  or later
- [pytest-xdist distribution](https://pytest-xdist.readthedocs.io/en/stable/distribution.html):
  `-n auto`; version 3.8.0 on [PyPI](https://pypi.org/project/pytest-xdist/)
- [pytest-cov reporting](https://pytest-cov.readthedocs.io/en/latest/reporting.html):
  `--cov-report html`; version 7.1.0 on [PyPI](https://pypi.org/project/pytest-cov/)
- [python3-pytest in Ubuntu noble](https://packages.ubuntu.com/noble/python3-pytest):
  7.4.4

Facts on this page were checked on 25 September 2026.