# How to let an AI agent test your own API in a sandbox

Run your API and the agent's tests in one microVM, allow only your own hosts, and send each failing test back to the agent to fix or report.

**On Runtime the service under test, its tests and the agent's mistakes stay
on one disposable machine.** A 2 vCPU, 4 GiB sandbox costs $0.08 an hour with
both CPUs busy and $0.03125 while the agent waits on its model, because Runtime
bills measured CPU ([pricing](/docs/pricing), 25 September 2026). Testing every
pull request of a busy service comes to about $1.28 a month, worked out below.

This page is about testing your own APIs. Runtime's
[acceptable use policy](https://withruntime.com/legal/acceptable-use) forbids
probing or vulnerability testing against any system that is not yours, even
with its owner's permission, and the network rules below keep the agent on
your own hosts.

## Two ways to point an agent at an API

| Setup                              | What the agent reaches             | Use it for                                     |
| ---------------------------------- | ---------------------------------- | ---------------------------------------------- |
| The API runs inside the sandbox    | `localhost` only; internet off     | Every pull request; destructive tests are safe |
| Your staging API, from the sandbox | Your staging host and nothing else | Contract checks against the deployed service   |

Running the service in the sandbox is the stronger choice for an agent that
writes tests: a test that deletes every record deletes a copy that stops with
the sandbox.

## The short answer

Upload the service, start it with `spawn`, wait until it answers, then run the
test file the agent wrote and give it the result:

```ts check
import { Sandbox } from "withruntime";

type Agent = (feedback: string) => Promise<string | null>; // your model: a new test file, or null when done

export async function testApi(serviceDir: string, agent: Agent, rounds = 5) {
  await using sbx = await Sandbox.create({
    network: { internet: true, allow: ["pypi.org", "*.pythonhosted.org"] },
    timeoutSeconds: 1800,
    onLeaseEnd: "stop",
  });
  await sbx.files.upload(serviceDir, "/workspace/service");
  await sbx.exec("pip install --quiet -r requirements.txt pytest httpx", {
    cwd: "/workspace/service",
    check: true,
    timeoutMs: 300_000,
  });
  await sbx.network.set({ internet: false }); // from here on, only localhost

  await sbx.spawn("python3 -m uvicorn app:app --port 8000", { cwd: "/workspace/service" });
  await sbx.exec(
    [
      "bash",
      "-c",
      "for i in $(seq 60); do curl -sf localhost:8000/health && exit 0; sleep 1; done; exit 1",
    ],
    {
      check: true,
      timeoutMs: 90_000,
    },
  );

  let feedback =
    "Write pytest tests for the API at http://localhost:8000. The source is in service/.";
  for (let round = 0; round < rounds; round++) {
    const tests = await agent(feedback);
    if (tests === null) break;
    await sbx.files.write("/workspace/tests/test_api.py", tests);
    const run = await sbx.exec(["python3", "-m", "pytest", "-q", "-x", "/workspace/tests"], {
      cwd: "/workspace/service",
      timeoutMs: 300_000,
    });
    feedback = `exit ${run.exitCode}${run.timedOut ? " (timed out)" : ""}\n${run.stdout.slice(-6000)}`;
  }
  return feedback;
}
```

```python check
from withruntime import Sandbox

WAIT = "for i in $(seq 60); do curl -sf localhost:8000/health && exit 0; sleep 1; done; exit 1"


def test_api(service_dir: str, agent, rounds: int = 5) -> str:
    """agent(feedback) returns a new test file, or None when it is done."""
    with Sandbox.create(network={"internet": True, "allow": ["pypi.org", "*.pythonhosted.org"]},
                        timeout_seconds=1800, on_lease_end="stop") as sbx:
        sbx.files.upload(service_dir, "/workspace/service")
        sbx.exec("pip install --quiet -r requirements.txt pytest httpx",
                 cwd="/workspace/service", check=True, timeout_ms=300_000)
        sbx.network.set(internet=False)  # from here on, only localhost

        sbx.spawn("python3 -m uvicorn app:app --port 8000", cwd="/workspace/service")
        sbx.exec(["bash", "-c", WAIT], check=True, timeout_ms=90_000)

        feedback = "Write pytest tests for the API at http://localhost:8000. The source is in service/."
        for _ in range(rounds):
            tests = agent(feedback)
            if tests is None:
                break
            sbx.files.write("/workspace/tests/test_api.py", tests)
            run = sbx.exec(["python3", "-m", "pytest", "-q", "-x", "/workspace/tests"],
                           cwd="/workspace/service", timeout_ms=300_000)
            feedback = f"exit {run.exit_code}{' (timed out)' if run.timed_out else ''}\n{run.stdout[-6000:]}"
        return feedback
```

Replace the `uvicorn` line with your own start command, such as `npm start` or
`go run ./cmd/api`, and the health check with a route your service has. The
server keeps running between rounds because `spawn` started it; a process
started by `exec` would end with its command.

## Test your staging API with a key the agent never sees

To check the deployed service, allow the sandbox to reach your staging host
and nothing else, and store the staging token as a Runtime secret. With
`header`, the host's proxy sets that header on every HTTPS request to the
host, so the test code carries no credential at all:

```ts check
import { Runtime } from "withruntime";

const runtime = new Runtime();
await runtime.secrets.set("STAGING_API_TOKEN", {
  value: process.env.STAGING_API_TOKEN ?? "",
  hosts: ["staging-api.example.com"],
  header: "Authorization",
  format: "Bearer {value}",
});

await using sbx = await runtime.sandboxes.create({
  network: { internet: true, allow: ["staging-api.example.com"] },
});
const probe = await sbx.exec([
  "curl",
  "-s",
  "-o",
  "/dev/null",
  "-w",
  "%{http_code}",
  "https://staging-api.example.com/health",
]);
console.log(probe.stdout); // the proxy added the Authorization header
```

The value is sealed when you store it: no API returns it, and the sandbox
holds only a placeholder, so a test that prints its environment or a prompt
injection in an API response leaks nothing usable
([security](/docs/security#secrets-sandboxes-never-see)). Name only hosts you
own: a host that echoes requests back would show the value to the sandbox.

## What an API-testing agent needs

| Need                                  | How Runtime covers it                                                       |
| ------------------------------------- | --------------------------------------------------------------------------- |
| The service running beside its tests  | `spawn` keeps the server up; tests call `localhost`                         |
| Tests that cannot reach anyone else   | `internet: false`, or `allow` with only your hosts, enforced on the host    |
| Credentials the agent cannot read     | Secrets added by the proxy with `header` and `format`                       |
| A test that hangs                     | `timeoutMs` returns `timedOut: true` with the output so far                 |
| Databases and queues the API uses     | `sudo enable-docker`, then `docker compose up -d` in the same sandbox       |
| One run per pull request, in parallel | 100 sandboxes at once on a paid account to start                            |
| A record of what ran                  | stdout, stderr and exit code per command; files read back with `files.read` |

For a service that needs Postgres or Redis, see
[Docker in a sandbox](/how-to/run-docker-in-a-sandbox): Compose services reach
each other by name inside the sandbox, as they do anywhere.

## What it costs

Take 300 pull requests a month. Each run keeps a 2 vCPU, 4 GiB sandbox running
for 6 minutes, install and agent rounds included, at 0.5 of a vCPU on average:

```
CPU:    300 × 6 min / 60 × 0.5 vCPU × $0.025   = $0.38
Memory: 300 × 6 min / 60 × 4 GiB × $0.0075     = $0.90
Total:                                           $1.28
```

Put the install in a [custom image](/docs/images) and each run skips it.
Model tokens are billed by your model provider. New accounts get 50 free
sandbox hours, no card; the trial reaches ports 80 and 443.

## Start

```bash no-run
npx withruntime sandbox run --trial -- python3 -c 'print(6 * 7)'
```

The first run prints a link to approve in your browser. Then use the
[JavaScript](/docs/javascript#background-processes) or
[Python](/docs/python#background-processes) SDK as above.

Related: [end-to-end testing in sandboxes](/use-cases/end-to-end-testing),
[egress control](/glossary/egress-control),
[a coding agent sandbox](/use-cases/coding-agent-sandbox),
[run Docker in a sandbox](/how-to/run-docker-in-a-sandbox).

Facts on this page were checked on 25 September 2026.