How to let an AI agent test your own API in a sandbox

On Runtime, the service under test, its tests and the agent's mistakes stay on one disposable machine. A 2 vCPU, 4 GiB sandbox costs $0.08 per hour with both CPUs busy. It costs $0.03125 per hour while the agent waits on its model. Runtime bills measured CPU use (pricing, 26 October 2026). Testing every pull request of a busy service costs about $1.28 a month. The calculation is below.

This page covers testing your own APIs. Runtime's acceptable use policy prohibits probing or vulnerability testing of any system you do not own, even with the owner's permission. The network rules below keep the agent on your own hosts.

Two ways to point an agent at an API

For an agent that writes tests, running the service in the sandbox is the stronger choice. A test that deletes every record deletes only a copy, and that copy stops with the sandbox.

The short answer

Upload the service. Start it with spawn. Wait until it responds. Then run the test file the agent wrote, and return the result to the agent:

TypeScriptimport { Sandbox } from "withruntime";type Agent = (feedback: string) => Promise<string | null>; // your model: a new test file, or null when doneexport async function testApi(serviceDir: string, agent: Agent, rounds = 5) {  await using sbx = await Sandbox.create({    network: { internet: true, allow: ["pypi.org", "*.pythonhosted.org"] },    timeoutSeconds: 1800,    onLeaseEnd: "stop",  });  await sbx.files.upload(serviceDir, "/workspace/service");  await sbx.exec("pip install --quiet -r requirements.txt pytest httpx", {    cwd: "/workspace/service",    check: true,    timeoutMs: 300_000,  });  await sbx.network.set({ internet: false }); // from here on, only localhost  await sbx.spawn("python3 -m uvicorn app:app --port 8000", { cwd: "/workspace/service" });  await sbx.exec(    [      "bash",      "-c",      "for i in $(seq 60); do curl -sf localhost:8000/health && exit 0; sleep 1; done; exit 1",    ],    {      check: true,      timeoutMs: 90_000,    },  );  let feedback =    "Write pytest tests for the API at http://localhost:8000. The source is in service/.";  for (let round = 0; round < rounds; round++) {    const tests = await agent(feedback);    if (tests === null) break;    await sbx.files.write("/workspace/tests/test_api.py", tests);    const run = await sbx.exec(["python3", "-m", "pytest", "-q", "-x", "/workspace/tests"], {      cwd: "/workspace/service",      timeoutMs: 300_000,    });    feedback = `exit ${run.exitCode}${run.timedOut ? " (timed out)" : ""}\n${run.stdout.slice(-6000)}`;  }  return feedback;}

Pythonfrom withruntime import SandboxWAIT = "for i in $(seq 60); do curl -sf localhost:8000/health && exit 0; sleep 1; done; exit 1"def test_api(service_dir: str, agent, rounds: int = 5) -> str:    """agent(feedback) returns a new test file, or None when it is done."""    with Sandbox.create(network={"internet": True, "allow": ["pypi.org", "*.pythonhosted.org"]},                        timeout_seconds=1800, on_lease_end="stop") as sbx:        sbx.files.upload(service_dir, "/workspace/service")        sbx.exec("pip install --quiet -r requirements.txt pytest httpx",                 cwd="/workspace/service", check=True, timeout_ms=300_000)        sbx.network.set(internet=False)  # from here on, only localhost        sbx.spawn("python3 -m uvicorn app:app --port 8000", cwd="/workspace/service")        sbx.exec(["bash", "-c", WAIT], check=True, timeout_ms=90_000)        feedback = "Write pytest tests for the API at http://localhost:8000. The source is in service/."        for _ in range(rounds):            tests = agent(feedback)            if tests is None:                break            sbx.files.write("/workspace/tests/test_api.py", tests)            run = sbx.exec(["python3", "-m", "pytest", "-q", "-x", "/workspace/tests"],                           cwd="/workspace/service", timeout_ms=300_000)            feedback = f"exit {run.exit_code}{' (timed out)' if run.timed_out else ''}\n{run.stdout[-6000:]}"        return feedback

Replace the uvicorn line with your own start command, such as npm start or go run ./cmd/api. Replace the health check with a route your service has. The server keeps running between rounds because spawn started it. A process started by exec ends when its command ends.

Test your staging API with a key the agent never sees

To check the deployed service, allow the sandbox to reach your staging host and no other host. Store the staging token as a Runtime secret. With header, the proxy for that host sets the header on every HTTPS request to the host. The test code holds no credential.

TypeScriptimport { Runtime } from "withruntime";const runtime = new Runtime();await runtime.secrets.set("STAGING_API_TOKEN", {  value: process.env.STAGING_API_TOKEN ?? "",  hosts: ["staging-api.example.com"],  header: "Authorization",  format: "Bearer {value}",});await using sbx = await runtime.sandboxes.create({  network: { internet: true, allow: ["staging-api.example.com"] },});const probe = await sbx.exec([  "curl",  "-s",  "-o",  "/dev/null",  "-w",  "%{http_code}",  "https://staging-api.example.com/health",]);console.log(probe.stdout); // the proxy added the Authorization header

The value is sealed when you store it. No API returns it. The sandbox holds only a placeholder. So a test that prints its environment, or a prompt injection in an API response, leaks nothing usable (security). Name only hosts you own. A host that echoes requests back would show the value to the sandbox.

What an API-testing agent needs

For a service that needs Postgres or Redis, see Docker in a sandbox: Compose services reach each other by name inside the sandbox, as they do anywhere.

What it costs

Assume 300 pull requests a month. Each run keeps a 2 vCPU, 4 GiB sandbox running for 6 minutes, including the install and the agent rounds. On average, the sandbox uses 0.5 of a vCPU:

Put the install in a custom image so each run skips it. Your model provider bills the model tokens. Every account gets 100 free hours of a 1 GB machine in its first 30 days, with no card. Without credit, a sandbox reaches ports 80 and 443.

Start

Terminalprintf '%s\n' 'print(sum([125, 250, 375]))' > invoice.pynpx withruntime sandboxes run -- python3 - < invoice.py

The first run prints a link. Open it in your browser to approve the run. Then use the JavaScript or Python SDK as shown above.

Related: end-to-end testing in sandboxes, egress control, a coding agent sandbox, run Docker in a sandbox.

Facts on this page were checked on 26 October 2026.