How to run SWE-bench-style agent evals in parallel sandboxes

Build each task's environment once, start one isolated sandbox per attempt, apply the agent's patch, run the tests and record pass or fail.

On Runtime every attempt gets its own machine, and a paid account runs 100 at once to start. Each sandbox is a Firecracker microVM with its own kernel, so one task's tests cannot disturb another's. Runtime bills the CPU the tests use, not the CPUs they hold: 500 ten-minute attempts on 2 vCPUs and 4 GiB cost $2.92 at Runtime's rates, worked out below.

What a SWE-bench-style eval is

SWE-bench gives a language model "a codebase and an issue" and asks it to generate "a patch that resolves the described problem". Its harness applies the patch to the repository and runs the repository's tests, in Docker, to decide whether the issue is resolved. Each task names a repository, a base commit, the problem statement, and two lists of tests: FAIL_TO_PASS, which the fix must make pass, and PASS_TO_PASS, which must keep passing. The full dataset has 2,294 tasks and SWE-bench Verified has 500 (as SWE-bench's documentation states, read 25 September 2026).

SWE-bench's own Docker setup guide asks for at least 120 GB of free disk and recommends 16 GB of RAM and at least 8 CPUs for its evaluation, and the harness runs tasks side by side with --max_workers. On one machine, parallelism stops where that machine's disk and cores do. In sandboxes, each attempt has its own.

The same shape fits any eval where an agent changes code and tests judge it: an environment per task, an attempt, a check, a score.

The core loop

Give every task an image with its repository checked out at the base commit and its dependencies installed. Then each attempt is one sandbox:

TypeScriptimport { Runtime } from "withruntime";type Task = { id: string; image: string; patch: string; testCommand: string };const tasks: Task[] = []; // load your task list here// Past the account's limit, a create waits for room instead of failing.const runtime = new Runtime({ waitForCapacityMs: 600_000 });async function evaluate(task: Task) {  await using sbx = await runtime.sandboxes.create({    image: task.image,    labels: { run: "eval-2026-09-25", task: task.id },    network: { internet: false },    timeoutSeconds: 1800,    onLeaseEnd: "stop",  });  await sbx.files.write("/workspace/agent.patch", task.patch);  const apply = await sbx.exec("git apply /workspace/agent.patch", { cwd: "/workspace/repo" });  if (apply.exitCode !== 0) return { id: task.id, resolved: false, reason: apply.stderr };  const tests = await sbx.exec(task.testCommand, { cwd: "/workspace/repo", timeoutMs: 900_000 });  return { id: task.id, resolved: tests.exitCode === 0, timedOut: tests.timedOut };}const results = await Promise.all(tasks.map(evaluate));console.log(results.filter((r) => r.resolved).length, "of", results.length);

Pythonfrom concurrent.futures import ThreadPoolExecutorfrom withruntime import Runtimetasks = []  # dicts with id, image, patch, test_commandruntime = Runtime(wait_for_capacity=600)  # past the limit, a create waits for roomdef evaluate(task):    with runtime.sandboxes.create(        image=task["image"],        labels={"run": "eval-2026-09-25", "task": task["id"]},        network={"internet": False},        timeout_seconds=1800,        on_lease_end="stop",    ) as sbx:        sbx.files.write("/workspace/agent.patch", task["patch"])        apply = sbx.exec("git apply /workspace/agent.patch", cwd="/workspace/repo")        if apply.exit_code != 0:            return {"id": task["id"], "resolved": False, "reason": apply.stderr}        tests = sbx.exec(task["test_command"], cwd="/workspace/repo", timeout_ms=900_000)        return {"id": task["id"], "resolved": tests.exit_code == 0, "timed_out": tests.timed_out}with ThreadPoolExecutor(max_workers=100) as pool:    results = list(pool.map(evaluate, tasks))print(sum(r["resolved"] for r in results), "of", len(results))

The sandbox stops when its block ends, even when a test crashes.
A test that hangs returns timedOut: true with its output so far; it is a result, not an exception.
With the internet off, a task cannot fetch its answer or leak the test set.
Labels mark every sandbox with its run and task, so runtime.sandboxes.list({ labels: { run: "eval-2026-09-25" } }) finds them.

Build the task environments

A task image comes from any container image, public or private, a Dockerfile with its build context, or a short recipe. Build each once; every attempt starts from it without reinstalling:

TypeScriptimport { Runtime } from "withruntime";const runtime = new Runtime();await runtime.images.build({  name: "task-env",  recipe: {    apt: ["libpq-dev"],    commands: [      "git clone https://github.com/your-org/your-repo.git repo",      "cd repo && git checkout 0123abc && pip install -e .",    ],  },});

Building is free. A stored image is charged on its size at $0.08 per GB per 30-day month, and an image can be up to 20 GiB (custom images). If your harness already ships a Docker image per task, pass it as image and build it as it is.

Many attempts at one task

For pass@k, let the agent work in one prepared sandbox, or start it from the task image, then fork it. fork copies the machine as it is, files, memory and running processes, into 1 to 10 running sandboxes:

TypeScriptimport { Sandbox } from "withruntime";await using base = await Sandbox.create({ image: "task-env", network: { internet: false } });const copies = await base.fork({ count: 5 });// ... run one agent attempt in each copy, then score each ...await Promise.all(copies.map((copy) => copy.stop()));

To start attempts later, or from other processes, keep the prepared machine as a snapshot and create sandboxes from it with snapshot: id.

What matters for evals

Need	How Runtime covers it
Tasks that cannot affect each other	A Firecracker microVM with its own kernel and disk per attempt
The same environment every time	Versioned images: `name@version` pins one build
Many at once	100 sandboxes, 200 vCPUs and 400 GiB across running ones, on a paid account to start
Bursts past the limit	Creates wait for room, up to two minutes by default or as long as you set
Hung tests	Per-command `timeoutMs`, up to 24 hours; host-side leases
No leaked answers	`network: { internet: false }`, enforced outside the guest
Retries that do not double-count	Idempotency keys on every write
Several samples per task	Forks of 1 to 10 running copies; snapshots kept 1 to 365 days
Cost you can predict	Measured CPU plus reserved memory; `maxCostMicros` per create

To run more than 100 at once, write to support with the numbers you need (pricing).

What it costs

Take 500 attempts, each keeping a 2 vCPU, 4 GiB sandbox running for 10 minutes and using 120 CPU-seconds of test work:

TextCPU:    500 × 120 s / 3,600 × $0.025            = $0.42Memory: 500 × 600 s / 3,600 × 4 GiB × $0.0075   = $2.50Total:                                            $2.92

Runtime charges $0.025 per vCPU-hour of measured CPU, with a floor of a twentieth of a vCPU, and $0.0075 per GiB-hour of memory (pricing). A sandbox waiting on the agent's model costs $0.03125 an hour at this size. Model calls are billed by your model provider, not here. New accounts get 50 free sandbox hours, no card, with eight sandboxes at once.

Sources

SWE-bench documentation: what the benchmark asks and its containerized harness, read 25 September 2026.
SWE-bench datasets: task counts and fields, read 25 September 2026.
SWE-bench evaluation and Docker setup: how patches are scored, --max_workers, and resource requirements, read 25 September 2026.

Facts on this page were checked on 25 September 2026.