How to let an AI agent test your own API in a sandbox

Run your API and the agent's tests in one microVM, allow only your own hosts, and send each failing test back to the agent to fix or report.

On Runtime the service under test, its tests and the agent's mistakes stay on one disposable machine. A 2 vCPU, 4 GiB sandbox costs $0.08 an hour with both CPUs busy and $0.03125 while the agent waits on its model, because Runtime bills measured CPU (pricing, 25 September 2026). Testing every pull request of a busy service comes to about $1.28 a month, worked out below.

This page is about testing your own APIs. Runtime's acceptable use policy forbids probing or vulnerability testing against any system that is not yours, even with its owner's permission, and the network rules below keep the agent on your own hosts.

Two ways to point an agent at an API

Running the service in the sandbox is the stronger choice for an agent that writes tests: a test that deletes every record deletes a copy that stops with the sandbox.

The short answer

Upload the service, start it with spawn, wait until it answers, then run the test file the agent wrote and give it the result:

TypeScriptimport { Sandbox } from "withruntime";type Agent = (feedback: string) => Promise<string | null>; // your model: a new test file, or null when doneexport async function testApi(serviceDir: string, agent: Agent, rounds = 5) {  await using sbx = await Sandbox.create({    network: { internet: true, allow: ["pypi.org", "*.pythonhosted.org"] },    timeoutSeconds: 1800,    onLeaseEnd: "stop",  });  await sbx.files.upload(serviceDir, "/workspace/service");  await sbx.exec("pip install --quiet -r requirements.txt pytest httpx", {    cwd: "/workspace/service",    check: true,    timeoutMs: 300_000,  });  await sbx.network.set({ internet: false }); // from here on, only localhost  await sbx.spawn("python3 -m uvicorn app:app --port 8000", { cwd: "/workspace/service" });  await sbx.exec(    [      "bash",      "-c",      "for i in $(seq 60); do curl -sf localhost:8000/health && exit 0; sleep 1; done; exit 1",    ],    {      check: true,      timeoutMs: 90_000,    },  );  let feedback =    "Write pytest tests for the API at http://localhost:8000. The source is in service/.";  for (let round = 0; round < rounds; round++) {    const tests = await agent(feedback);    if (tests === null) break;    await sbx.files.write("/workspace/tests/test_api.py", tests);    const run = await sbx.exec(["python3", "-m", "pytest", "-q", "-x", "/workspace/tests"], {      cwd: "/workspace/service",      timeoutMs: 300_000,    });    feedback = `exit ${run.exitCode}${run.timedOut ? " (timed out)" : ""}\n${run.stdout.slice(-6000)}`;  }  return feedback;}

Pythonfrom withruntime import SandboxWAIT = "for i in $(seq 60); do curl -sf localhost:8000/health && exit 0; sleep 1; done; exit 1"def test_api(service_dir: str, agent, rounds: int = 5) -> str:    """agent(feedback) returns a new test file, or None when it is done."""    with Sandbox.create(network={"internet": True, "allow": ["pypi.org", "*.pythonhosted.org"]},                        timeout_seconds=1800, on_lease_end="stop") as sbx:        sbx.files.upload(service_dir, "/workspace/service")        sbx.exec("pip install --quiet -r requirements.txt pytest httpx",                 cwd="/workspace/service", check=True, timeout_ms=300_000)        sbx.network.set(internet=False)  # from here on, only localhost        sbx.spawn("python3 -m uvicorn app:app --port 8000", cwd="/workspace/service")        sbx.exec(["bash", "-c", WAIT], check=True, timeout_ms=90_000)        feedback = "Write pytest tests for the API at http://localhost:8000. The source is in service/."        for _ in range(rounds):            tests = agent(feedback)            if tests is None:                break            sbx.files.write("/workspace/tests/test_api.py", tests)            run = sbx.exec(["python3", "-m", "pytest", "-q", "-x", "/workspace/tests"],                           cwd="/workspace/service", timeout_ms=300_000)            feedback = f"exit {run.exit_code}{' (timed out)' if run.timed_out else ''}\n{run.stdout[-6000:]}"        return feedback

Replace the uvicorn line with your own start command, such as npm start or go run ./cmd/api, and the health check with a route your service has. The server keeps running between rounds because spawn started it; a process started by exec would end with its command.

Test your staging API with a key the agent never sees

To check the deployed service, allow the sandbox to reach your staging host and nothing else, and store the staging token as a Runtime secret. With header, the host's proxy sets that header on every HTTPS request to the host, so the test code carries no credential at all:

TypeScriptimport { Runtime } from "withruntime";const runtime = new Runtime();await runtime.secrets.set("STAGING_API_TOKEN", {  value: process.env.STAGING_API_TOKEN ?? "",  hosts: ["staging-api.example.com"],  header: "Authorization",  format: "Bearer {value}",});await using sbx = await runtime.sandboxes.create({  network: { internet: true, allow: ["staging-api.example.com"] },});const probe = await sbx.exec([  "curl",  "-s",  "-o",  "/dev/null",  "-w",  "%{http_code}",  "https://staging-api.example.com/health",]);console.log(probe.stdout); // the proxy added the Authorization header

The value is sealed when you store it: no API returns it, and the sandbox holds only a placeholder, so a test that prints its environment or a prompt injection in an API response leaks nothing usable (security). Name only hosts you own: a host that echoes requests back would show the value to the sandbox.

What an API-testing agent needs

For a service that needs Postgres or Redis, see Docker in a sandbox: Compose services reach each other by name inside the sandbox, as they do anywhere.

What it costs

Take 300 pull requests a month. Each run keeps a 2 vCPU, 4 GiB sandbox running for 6 minutes, install and agent rounds included, at 0.5 of a vCPU on average:

TextCPU:    300 × 6 min / 60 × 0.5 vCPU × $0.025   = $0.38Memory: 300 × 6 min / 60 × 4 GiB × $0.0075     = $0.90Total:                                           $1.28

Put the install in a custom image and each run skips it. Model tokens are billed by your model provider. New accounts get 50 free sandbox hours, no card; the trial reaches ports 80 and 443.

Start

Terminalnpx withruntime sandbox run --trial -- python3 -c 'print(6 * 7)'

The first run prints a link to approve in your browser. Then use the JavaScript or Python SDK as above.

Related: end-to-end testing in sandboxes, egress control, a coding agent sandbox, run Docker in a sandbox.

Facts on this page were checked on 25 September 2026.