How to run an agent's pytest suite in an isolated sandbox

It bills only the CPU the tests use. That start time runs from create request to first Python result. A 2 vCPU, 4 GiB sandbox costs $0.08 an hour with both CPUs busy. A five-minute suite that keeps them busy costs under a cent. Code an agent wrote runs behind its own Linux kernel, never on your CI runner or your laptop.

Run the suite and bring the reports back

The project folder is uploaded as one compressed archive. Install the requirements first. Then cut the network, so the tests themselves reach nothing:

TypeScriptimport { Sandbox } from "withruntime";await using sbx = await Sandbox.create({ timeoutSeconds: 1800 });await sbx.files.upload("./project", "/workspace/project");await sbx.exec("cd project && pip install -q -r requirements.txt pytest==9.1.1 pytest-cov==7.1.0", {  check: true,  timeoutMs: 600_000,});await sbx.network.set({ internet: false });const run = await sbx.exec(  "cd project && python3 -m pytest -q --junitxml=report.xml --cov=. --cov-report=html",  { timeoutMs: 1_200_000, onStdout: (text) => process.stdout.write(text) },);await sbx.files.download("/workspace/project/report.xml", "./report.xml");await sbx.files.download("/workspace/project/htmlcov", "./htmlcov");process.exitCode = run.exitCode ?? 1;

Pythonimport sysfrom withruntime import Sandboxwith Sandbox.create(timeout_seconds=1800) as sbx:    sbx.files.upload("./project", "/workspace/project")    sbx.exec("cd project && pip install -q -r requirements.txt pytest==9.1.1 pytest-cov==7.1.0",             check=True, timeout_ms=600_000)    sbx.network.set(internet=False)    run = sbx.exec("cd project && python3 -m pytest -q --junitxml=report.xml --cov=. --cov-report=html",                   timeout_ms=1_200_000, on_stdout=sys.stdout.write)    sbx.files.download("/workspace/project/report.xml", "report.xml")    sbx.files.download("/workspace/project/htmlcov", "htmlcov")sys.exit(1 if run.exit_code is None else run.exit_code)

• Streaming: onStdout prints each line as pytest writes it. The result then keeps the whole output, not only the first 64 KiB.
• Timeouts: a command gets 60 seconds unless you pass timeoutMs. A suite that runs longer returns timedOut: true with the output so far. It never throws.
• Coverage: --cov-report=html writes the htmlcov folder. One download call returns it.

Why the network goes off after the install

Tests written by a model sometimes call real services, read environment variables or download fixtures. After network.set({ internet: false }) runs, every outbound connection is refused. A test that depends on the internet therefore fails in the report instead of quietly reaching it. The host enforces the rule. sudo inside the sandbox cannot undo it (turn off sandbox internet).

If the suite needs one service, such as a package index or your staging API, use an allow list instead of switching the internet off: network.set({ internet: true, allow: ["staging.example.com"] }).

Read the result from the exit code

pytest's exit code says what happened without parsing any output. Pass it to the agent with the failing tests from report.xml:

Exit code 5 matters for agents. A model asked to "make the tests pass" can delete every test. pytest then reports 5, not 0.

Use more than one CPU

pytest-xdist spreads tests over worker processes with -n auto. Set the vCPU count to match when you create the sandbox:

TypeScriptimport { Sandbox } from "withruntime";await using sbx = await Sandbox.create({ vcpu: 4, memoryMiB: 8192, timeoutSeconds: 1800 });await sbx.files.upload("./project", "/workspace/project");await sbx.exec(  "cd project && pip install -q -r requirements.txt pytest==9.1.1 pytest-xdist==3.8.0",  {    check: true,    timeoutMs: 600_000,  },);await sbx.exec("cd project && python3 -m pytest -n auto -q", {  timeoutMs: 1_200_000,  onStdout: (text) => process.stdout.write(text),});

CPU is billed on what the workers use. Four workers that finish in a quarter of the time cost about the same CPU as one worker that runs four times longer. Sandboxes without credit go up to 2 vCPU and 4 GiB.

Skip the install on every run

When the same dependencies install for every run, build them into a custom image. Building is free. Each later sandbox starts with pytest and the project's packages already in place:

Pythonfrom withruntime import Runtimeruntime = Runtime()with open("project/requirements.txt") as handle:    requirements = [line.strip() for line in handle if line.strip() and not line.startswith("#")]runtime.images.build(    name="pytest-env",    recipe={"pip": ["pytest==9.1.1", "pytest-cov==7.1.0", "pytest-xdist==3.8.0", *requirements]},    on_log=lambda line: print(line["text"]),)with runtime.sandboxes.create(image="pytest-env") as sbx:    sbx.files.upload("./project", "/workspace/project")    print(sbx.exec("cd project && python3 -m pytest -q", timeout_ms=600_000).stdout)

Ubuntu's own python3-pytest package on noble is pytest 7.4.4. Installing from PyPI gets the current 9.1.1.

What is in the sandbox already

See the sandbox environment for the full list.

Related

• Run untrusted Python code safely
• A sandbox for a coding agent
• Agent evals and SWE-bench
• Jest tests across parallel sandboxes

Sources

Checked 25 October 2026.

• pytest exit codes: the seven codes, 0 to 6
• pytest on PyPI: version 9.1.1, Python 3.10 or later
• pytest-xdist distribution: -n auto; version 3.8.0 on PyPI
• pytest-cov reporting: --cov-report html; version 7.1.0 on PyPI
• python3-pytest in the Ubuntu noble universe index: 7.4.4

Facts on this page were checked on 25 October 2026.