How to run an agent's pytest suite in an isolated sandbox

Upload the project to a microVM, install its requirements, run pytest there and download the JUnit XML and coverage report.

Runtime starts the machine in 351 ms at the median and bills only the CPU the tests burn. That start time is from create request to first Python result, measured over 20 runs on 24 September 2026. A 2 vCPU, 4 GiB sandbox costs $0.08 an hour with both CPUs busy, so a five-minute suite that keeps them busy costs under a cent. Code an agent wrote runs behind its own Linux kernel, never on your CI runner or your laptop.

Run the suite and bring the reports back

The project folder travels in as one compressed archive. Install first, then cut the network so the tests themselves reach nothing:

TypeScriptimport { Sandbox } from "withruntime";await using sbx = await Sandbox.create({ timeoutSeconds: 1800 });await sbx.files.upload("./project", "/workspace/project");await sbx.exec("cd project && pip install -q -r requirements.txt pytest==9.1.1 pytest-cov==7.1.0", {  check: true,  timeoutMs: 600_000,});await sbx.network.set({ internet: false });const run = await sbx.exec(  "cd project && python3 -m pytest -q --junitxml=report.xml --cov=. --cov-report=html",  { timeoutMs: 1_200_000, onStdout: (text) => process.stdout.write(text) },);await sbx.files.download("/workspace/project/report.xml", "./report.xml");await sbx.files.download("/workspace/project/htmlcov", "./htmlcov");process.exitCode = run.exitCode ?? 1;

Pythonimport sysfrom withruntime import Sandboxwith Sandbox.create(timeout_seconds=1800) as sbx:    sbx.files.upload("./project", "/workspace/project")    sbx.exec("cd project && pip install -q -r requirements.txt pytest==9.1.1 pytest-cov==7.1.0",             check=True, timeout_ms=600_000)    sbx.network.set(internet=False)    run = sbx.exec("cd project && python3 -m pytest -q --junitxml=report.xml --cov=. --cov-report=html",                   timeout_ms=1_200_000, on_stdout=sys.stdout.write)    sbx.files.download("/workspace/project/report.xml", "report.xml")    sbx.files.download("/workspace/project/htmlcov", "htmlcov")sys.exit(1 if run.exit_code is None else run.exit_code)

• Streaming: onStdout prints each line as pytest writes it, and the result then keeps the whole output instead of the first 64 KiB.
• Timeouts: a command gets 60 seconds unless you pass timeoutMs. A suite that runs longer returns timedOut: true with the output so far; it never throws.
• Coverage: --cov-report=html writes the htmlcov folder, which comes back in one download call.

Why the network goes off after the install

Tests written by a model sometimes call real services, read environment variables or download fixtures. Once network.set({ internet: false }) runs, every outbound connection is refused, so a test that depends on the internet fails in the report instead of quietly reaching it. The rule is enforced on the host: sudo inside the sandbox cannot undo it (turn off sandbox internet).

If the suite needs one service, such as a package index or your staging API, use an allow list instead of switching the internet off: network.set({ internet: true, allow: ["staging.example.com"] }).

Read the result from the exit code

pytest's exit code says what happened without parsing any output. Pass it to the agent along with the failing tests from report.xml:

Exit code 5 matters for agents: a model asked to "make the tests pass" can delete every test, and pytest then reports 5, not 0.

Use more than one CPU

pytest-xdist spreads tests over worker processes with -n auto. Ask for the vCPUs to match when you create the sandbox:

TypeScriptimport { Sandbox } from "withruntime";await using sbx = await Sandbox.create({ vcpu: 4, memoryMiB: 8192, timeoutSeconds: 1800 });await sbx.files.upload("./project", "/workspace/project");await sbx.exec(  "cd project && pip install -q -r requirements.txt pytest==9.1.1 pytest-xdist==3.8.0",  {    check: true,    timeoutMs: 600_000,  },);await sbx.exec("cd project && python3 -m pytest -n auto -q", {  timeoutMs: 1_200_000,  onStdout: (text) => process.stdout.write(text),});

CPU is billed on what the workers use, so four workers that finish in a quarter of the time cost about the same CPU as one worker that runs four times longer. Trial sandboxes go up to 2 vCPU and 4 GiB.

Skip the install on every run

When the same dependencies install for every run, build them into a custom image. Building is free, and each later sandbox starts with pytest and the project's packages in place:

Pythonfrom withruntime import Runtimeruntime = Runtime()with open("project/requirements.txt") as handle:    requirements = [line.strip() for line in handle if line.strip() and not line.startswith("#")]runtime.images.build(    name="pytest-env",    recipe={"pip": ["pytest==9.1.1", "pytest-cov==7.1.0", "pytest-xdist==3.8.0", *requirements]},    on_log=lambda line: print(line["text"]),)with runtime.sandboxes.create(image="pytest-env") as sbx:    sbx.files.upload("./project", "/workspace/project")    print(sbx.exec("cd project && python3 -m pytest -q", timeout_ms=600_000).stdout)

Ubuntu's own python3-pytest package on noble is pytest 7.4.4. Installing from PyPI gets the current 9.1.1.

What is in the sandbox already

See the sandbox environment for the full list.

Related

• Run untrusted Python code safely
• A sandbox for a coding agent
• Agent evals and SWE-bench
• Jest tests across parallel sandboxes

Sources

Checked 25 September 2026.

• pytest exit codes: the seven codes, 0 to 6
• pytest on PyPI: version 9.1.1, Python 3.10 or later
• pytest-xdist distribution: -n auto; version 3.8.0 on PyPI
• pytest-cov reporting: --cov-report html; version 7.1.0 on PyPI
• python3-pytest in Ubuntu noble: 7.4.4

Facts on this page were checked on 25 September 2026.