How to run SWE-bench-style agent evals in parallel sandboxes

On Runtime, every attempt gets its own machine. A paid account runs 100 sandboxes at once. Each sandbox is a Firecracker microVM with its own kernel, so one task's tests cannot disturb another task's. Runtime bills the CPU the tests use, not the CPUs they hold. 500 ten-minute attempts on 2 vCPUs and 4 GiB cost $2.92 at Runtime's rates. The calculation is below.

What a SWE-bench-style eval is

SWE-bench gives a language model "a codebase and an issue". It asks the model to generate "a patch that resolves the described problem". The benchmark's harness applies the patch to the repository. Then it runs the repository's tests, in Docker, to decide whether the issue is resolved. Each task names a repository, a base commit, the problem statement, and two lists of tests. FAIL_TO_PASS lists the tests that the fix must make pass. PASS_TO_PASS lists the tests that must keep passing. The full dataset has 2,294 tasks. SWE-bench Verified has 500 (as SWE-bench's documentation states, read 26 October 2026).

SWE-bench's own Docker setup guide asks for at least 120 GB of free disk. It recommends 16 GB of RAM and at least 8 CPUs for its evaluation. The harness runs tasks side by side with --max_workers. On one machine, parallelism stops where that machine's disk and cores do. In sandboxes, each attempt has its own disk and cores.

The same pattern fits any eval in which an agent changes code and tests judge the result. Each eval needs one environment per task, one attempt, one check and one score.

The core loop

Give every task an image with its repository checked out at the base commit and its dependencies installed. Then start one sandbox per attempt:

TypeScriptimport { Runtime } from "withruntime";type Task = { id: string; image: string; patch: string; testCommand: string };const tasks: Task[] = []; // load your task list here// Past the account's limit, a create waits for room instead of failing.const runtime = new Runtime({ waitForCapacityMs: 600_000 });async function evaluate(task: Task) {  await using sbx = await runtime.sandboxes.create({    image: task.image,    labels: { run: "eval-2026-09-25", task: task.id },    network: { internet: false },    timeoutSeconds: 1800,    onLeaseEnd: "stop",  });  await sbx.files.write("/workspace/agent.patch", task.patch);  const apply = await sbx.exec("git apply /workspace/agent.patch", { cwd: "/workspace/repo" });  if (apply.exitCode !== 0) return { id: task.id, resolved: false, reason: apply.stderr };  const tests = await sbx.exec(task.testCommand, { cwd: "/workspace/repo", timeoutMs: 900_000 });  return { id: task.id, resolved: tests.exitCode === 0, timedOut: tests.timedOut };}const results = await Promise.all(tasks.map(evaluate));console.log(results.filter((r) => r.resolved).length, "of", results.length);

Pythonfrom concurrent.futures import ThreadPoolExecutorfrom withruntime import Runtimetasks = []  # dicts with id, image, patch, test_commandruntime = Runtime(wait_for_capacity=600)  # past the limit, a create waits for roomdef evaluate(task):    with runtime.sandboxes.create(        image=task["image"],        labels={"run": "eval-2026-09-25", "task": task["id"]},        network={"internet": False},        timeout_seconds=1800,        on_lease_end="stop",    ) as sbx:        sbx.files.write("/workspace/agent.patch", task["patch"])        apply = sbx.exec("git apply /workspace/agent.patch", cwd="/workspace/repo")        if apply.exit_code != 0:            return {"id": task["id"], "resolved": False, "reason": apply.stderr}        tests = sbx.exec(task["test_command"], cwd="/workspace/repo", timeout_ms=900_000)        return {"id": task["id"], "resolved": tests.exit_code == 0, "timed_out": tests.timed_out}with ThreadPoolExecutor(max_workers=100) as pool:    results = list(pool.map(evaluate, tasks))print(sum(r["resolved"] for r in results), "of", len(results))

• The sandbox is deleted when its block ends, even when a test crashes.
• A test that hangs returns timedOut: true with its output so far. It is a result, not an exception.
• With the internet off, a task cannot fetch its answer or leak the test set.
• Labels mark every sandbox with its run and task. Then runtime.sandboxes.list({ labels: { run: "eval-2026-09-25" } }) finds them.

Build the task environments

A task image can come from any container image, public or private. It can also come from a Dockerfile with its build context, or from a short recipe. Build each image once. Every attempt starts from it without reinstalling:

TypeScriptimport { Runtime } from "withruntime";const runtime = new Runtime();await runtime.images.build({  name: "task-env",  recipe: {    apt: ["libpq-dev"],    commands: [      "git clone https://github.com/your-org/your-repo.git repo",      "cd repo && git checkout 0123abc && pip install -e .",    ],  },});

Building is free. A stored image is charged by its size, at $0.08 per GB per 30-day month. An image can be up to 20 GiB (custom images). If your harness already ships a Docker image for each task, pass it as image and use it as it is.

Many attempts at one task

For pass@k, let the agent work in one prepared sandbox. Or start it from the task image, then fork it. fork copies the machine as it is (files, memory and running processes) into 1 to 100 running sandboxes:

TypeScriptimport { Sandbox } from "withruntime";await using base = await Sandbox.create({ image: "task-env", network: { internet: false } });const copies = await base.fork({ count: 5 });// ... run one agent attempt in each copy, then score each ...await Promise.all(copies.map((copy) => copy.stop()));

To start attempts later, or from other processes, keep the prepared machine as a snapshot. Then create sandboxes from it with snapshot: id.

What matters for evals

To run more than 100 sandboxes at once, contact support with the numbers you need (pricing).

What it costs

Take 500 attempts. Each keeps a 2 vCPU, 4 GiB sandbox running for 10 minutes. Each uses 120 CPU-seconds of test work:

Runtime charges $0.025 per vCPU-hour of measured CPU, with a floor of a twentieth of a vCPU. It charges $0.0075 per GiB-hour of memory (pricing). A sandbox waiting on the agent's model costs $0.03125 an hour at this size. Your model provider bills the model calls. Runtime does not. A new account gets 100 free hours of a 1 GB machine in its first 30 days and needs no card. It runs four sandboxes at once.

Sources

• SWE-bench documentation: what the benchmark asks, and its containerized harness, read 26 October 2026.
• SWE-bench datasets: task counts and fields, read 26 October 2026.
• SWE-bench evaluation and Docker setup: how patches are scored, --max_workers, and resource requirements, read 26 October 2026.

Related: RL environments, coding agent sandbox, sandbox forks, sandbox cost calculator.

Facts on this page were checked on 26 October 2026.