How to run an agent's CI on every pull request

Upload each pull request's code to a fresh sandbox, run install and tests there with no secrets inside, and report the exit code.

On Runtime a burst of agent pull requests queues for a sandbox instead of failing, and a paid account runs 100 at once to start. Each run is a Firecracker microVM with its own kernel, so code nobody has reviewed yet runs nowhere near your CI secrets. A CI key can carry a daily spending limit that only a person can raise, and Runtime was 42% to 88% cheaper than eleven other sandbox providers on a mostly-idle job at rates checked 23 September 2026 (the comparison).

Why agent pull requests need different CI

Coding agents open many more pull requests than people do, and every one carries code that no human has read when its checks start. On a shared CI runner that code runs next to whatever the runner can reach: its tokens, its cache, its network. GitHub already treats pull requests from forks as untrusted for this reason: secrets other than GITHUB_TOKEN are not passed to the runner, and GITHUB_TOKEN is read-only (GitHub's documentation, read 25 September 2026).

Running the install and the tests in a sandbox gives the same protection to every pull request, fork or not. The runner holds only a Runtime key; the agent's code runs somewhere that key does not exist.

The script your CI job runs

Upload the checked-out tree, install, test, and exit with the tests' code. Output streams into the CI log as it happens.

TypeScriptimport { Runtime } from "withruntime";// In CI a burst can queue behind other runs for up to ten minutes.const runtime = new Runtime({ waitForCapacityMs: 600_000 });await using sbx = await runtime.sandboxes.create({  image: "ci-node", // your dependencies, built once  diskMiB: 8192,  timeoutSeconds: 1800,  onLeaseEnd: "stop",  labels: { ci: "agent-prs", pr: process.env.PR_NUMBER ?? "local" },  network: { internet: true, allow: ["registry.npmjs.org"] },});await sbx.files.upload(".", "/workspace/repo");const stream = {  onStdout: (t: string) => process.stdout.write(t),  onStderr: (t: string) => process.stderr.write(t),};await sbx.exec("npm ci", { cwd: "/workspace/repo", timeoutMs: 600_000, check: true, ...stream });const tests = await sbx.exec("npm test", {  cwd: "/workspace/repo",  timeoutMs: 1_200_000,  ...stream,});process.exitCode = tests.timedOut ? 124 : (tests.exitCode ?? 1);

Pythonimport osimport sysfrom withruntime import Runtimeruntime = Runtime(wait_for_capacity=600)  # queue up to ten minutes in a burstwith runtime.sandboxes.create(    image="ci-python",    disk_mib=8192,    timeout_seconds=1800,    on_lease_end="stop",    labels={"ci": "agent-prs", "pr": os.environ.get("PR_NUMBER", "local")},    network={"internet": True, "allow": ["pypi.org", "*.pythonhosted.org"]},) as sbx:    sbx.files.upload(".", "/workspace/repo")    sbx.exec("pip install -e '.[test]'", cwd="/workspace/repo", timeout_ms=600_000, check=True,             on_stdout=sys.stdout.write, on_stderr=sys.stderr.write)    tests = sbx.exec("python3 -m pytest -q", cwd="/workspace/repo", timeout_ms=1_200_000,                     on_stdout=sys.stdout.write, on_stderr=sys.stderr.write)    code = 124 if tests.timed_out else (1 if tests.exit_code is None else tests.exit_code)sys.exit(code)

The allow list lets the install reach the package registry and nothing else, so a test that tries to send the repository somewhere fails to connect. Rules are enforced on the host, and root in the sandbox cannot change them (the sandbox environment).

Wire it into GitHub Actions

pull_request workflows run on opened, synchronize and reopened by default. The job needs Node or Python and one secret:

YAMLon: pull_requestjobs:  sandbox-tests:    runs-on: ubuntu-latest    steps:      - uses: actions/checkout@v4      - run: pip install withruntime      - run: python3 ci_sandbox.py        env:          RUNTIME_API_KEY: ${{ secrets.RUNTIME_API_KEY }}          PR_NUMBER: ${{ github.event.pull_request.number }}

Make that key from your own terminal. It prints a link for an owner, admin or developer to approve, then goes straight into the repository's secrets:

Terminalruntime keys create --name agent-ci --daily-limit 25 | gh secret set RUNTIME_API_KEY

With --daily-limit 25, no 24-hour window can spend more than $25. Past it, a create fails with spending_limit_reached and nothing is charged (keys for CI).

Pull requests from forks

A fork's workflow does not get your secrets, so it cannot call Runtime from GitHub's runner. Run the check from your own service instead: receive GitHub's pull request webhook, fetch the pull request's head inside a sandbox, and post the result as a check or a comment.

TypeScriptimport { Runtime } from "withruntime";const runtime = new Runtime({ waitForCapacityMs: 600_000 });export async function checkPullRequest(cloneUrl: string, number: number) {  await using sbx = await runtime.sandboxes.create({    image: "ci-node",    timeoutSeconds: 1800,    network: { internet: true, allow: ["github.com", "*.github.com", "registry.npmjs.org"] },  });  await sbx.exec(["git", "clone", "--depth", "50", cloneUrl, "repo"], {    check: true,    timeoutMs: 300_000,  });  await sbx.exec(["git", "fetch", "origin", `pull/${number}/head:pr`], {    cwd: "/workspace/repo",    check: true,  });  await sbx.exec("git checkout pr && npm ci", {    cwd: "/workspace/repo",    timeoutMs: 600_000,    check: true,  });  const tests = await sbx.exec("npm test", { cwd: "/workspace/repo", timeoutMs: 1_200_000 });  return { passed: tests.exitCode === 0, log: tests.stdout.slice(-4000) };}

git fetch origin pull/ID/head:BRANCH is GitHub's documented way to fetch a pull request by number.

What agent CI needs

What it costs

Take 3,000 agent pull requests a month. Each run keeps a 2 vCPU, 4 GiB sandbox for 8 minutes, and install plus tests use 600 CPU-seconds:

TextCPU:    3,000 × 600 s / 3,600 × $0.025            = $12.50Memory: 3,000 × 480 s / 3,600 × 4 GiB × $0.0075   = $12.00Total:                                              $24.50

About 0.8 cents a pull request, at $0.025 per vCPU-hour of measured CPU and $0.0075 per GiB-hour of memory (pricing). The GitHub runner that starts the job only uploads files and waits. Try it first on the free trial: 50 hours, eight sandboxes at once, no card.

Sources

• Events that trigger workflows: pull_request activity types and what fork workflows receive, read 25 September 2026.
• Checking out pull requests locally: git fetch origin pull/ID/head:BRANCH, read 25 September 2026.

Related: automated code review, coding agent sandbox, run Docker in a sandbox, test generation, GitHub Codespaces vs an agent sandbox.

Facts on this page were checked on 25 September 2026.