# How to run SWE-bench-style agent evals in parallel sandboxes

Build each task's environment once, start one isolated sandbox per attempt, apply the agent's patch, run the tests and record pass or fail.

**On Runtime every attempt gets its own machine, and a paid account runs 100
at once to start.** Each sandbox is a Firecracker microVM with its own kernel,
so one task's tests cannot disturb another's. Runtime bills the CPU the tests
use, not the CPUs they hold: 500 ten-minute attempts on 2 vCPUs and 4 GiB cost
$2.92 at Runtime's rates, worked out below.

## What a SWE-bench-style eval is

SWE-bench gives a language model "a codebase and an issue" and asks it to
generate "a patch that resolves the described problem". Its harness applies the
patch to the repository and runs the repository's tests, in Docker, to decide
whether the issue is resolved. Each task names a repository, a base commit, the
problem statement, and two lists of tests: `FAIL_TO_PASS`, which the fix must
make pass, and `PASS_TO_PASS`, which must keep passing. The full dataset has
2,294 tasks and SWE-bench Verified has 500 (as SWE-bench's documentation states,
read 25 September 2026).

SWE-bench's own Docker setup guide asks for at least 120 GB of free disk and
recommends 16 GB of RAM and at least 8 CPUs for its evaluation, and the harness
runs tasks side by side with `--max_workers`. On one machine, parallelism stops
where that machine's disk and cores do. In sandboxes, each attempt has its own.

The same shape fits any eval where an agent changes code and tests judge it:
an environment per task, an attempt, a check, a score.

## The core loop

Give every task an image with its repository checked out at the base commit
and its dependencies installed. Then each attempt is one sandbox:

```ts check
import { Runtime } from "withruntime";

type Task = { id: string; image: string; patch: string; testCommand: string };
const tasks: Task[] = []; // load your task list here

// Past the account's limit, a create waits for room instead of failing.
const runtime = new Runtime({ waitForCapacityMs: 600_000 });

async function evaluate(task: Task) {
  await using sbx = await runtime.sandboxes.create({
    image: task.image,
    labels: { run: "eval-2026-09-25", task: task.id },
    network: { internet: false },
    timeoutSeconds: 1800,
    onLeaseEnd: "stop",
  });
  await sbx.files.write("/workspace/agent.patch", task.patch);
  const apply = await sbx.exec("git apply /workspace/agent.patch", { cwd: "/workspace/repo" });
  if (apply.exitCode !== 0) return { id: task.id, resolved: false, reason: apply.stderr };
  const tests = await sbx.exec(task.testCommand, { cwd: "/workspace/repo", timeoutMs: 900_000 });
  return { id: task.id, resolved: tests.exitCode === 0, timedOut: tests.timedOut };
}

const results = await Promise.all(tasks.map(evaluate));
console.log(results.filter((r) => r.resolved).length, "of", results.length);
```

```python check
from concurrent.futures import ThreadPoolExecutor
from withruntime import Runtime

tasks = []  # dicts with id, image, patch, test_command
runtime = Runtime(wait_for_capacity=600)  # past the limit, a create waits for room


def evaluate(task):
    with runtime.sandboxes.create(
        image=task["image"],
        labels={"run": "eval-2026-09-25", "task": task["id"]},
        network={"internet": False},
        timeout_seconds=1800,
        on_lease_end="stop",
    ) as sbx:
        sbx.files.write("/workspace/agent.patch", task["patch"])
        apply = sbx.exec("git apply /workspace/agent.patch", cwd="/workspace/repo")
        if apply.exit_code != 0:
            return {"id": task["id"], "resolved": False, "reason": apply.stderr}
        tests = sbx.exec(task["test_command"], cwd="/workspace/repo", timeout_ms=900_000)
        return {"id": task["id"], "resolved": tests.exit_code == 0, "timed_out": tests.timed_out}


with ThreadPoolExecutor(max_workers=100) as pool:
    results = list(pool.map(evaluate, tasks))
print(sum(r["resolved"] for r in results), "of", len(results))
```

- The sandbox stops when its block ends, even when a test crashes.
- A test that hangs returns `timedOut: true` with its output so far; it is a
  result, not an exception.
- With the internet off, a task cannot fetch its answer or leak the test set.
- Labels mark every sandbox with its run and task, so
  `runtime.sandboxes.list({ labels: { run: "eval-2026-09-25" } })` finds them.

## Build the task environments

A task image comes from any container image, public or private, a Dockerfile
with its build context, or a short recipe. Build each once; every attempt
starts from it without reinstalling:

```ts check
import { Runtime } from "withruntime";

const runtime = new Runtime();
await runtime.images.build({
  name: "task-env",
  recipe: {
    apt: ["libpq-dev"],
    commands: [
      "git clone https://github.com/your-org/your-repo.git repo",
      "cd repo && git checkout 0123abc && pip install -e .",
    ],
  },
});
```

Building is free. A stored image is charged on its size at $0.08 per GB per
30-day month, and an image can be up to 20 GiB ([custom images](/docs/images)).
If your harness already ships a Docker image per task, pass it as `image` and
build it as it is.

## Many attempts at one task

For pass@k, let the agent work in one prepared sandbox, or start it from the
task image, then fork it. `fork` copies the machine as it is, files, memory
and running processes, into 1 to 10 running sandboxes:

```ts check
import { Sandbox } from "withruntime";

await using base = await Sandbox.create({ image: "task-env", network: { internet: false } });
const copies = await base.fork({ count: 5 });
// ... run one agent attempt in each copy, then score each ...
await Promise.all(copies.map((copy) => copy.stop()));
```

To start attempts later, or from other processes, keep the prepared machine as
a [snapshot](/glossary/sandbox-snapshot) and create sandboxes from it with
`snapshot: id`.

## What matters for evals

| Need                                | How Runtime covers it                                                                |
| ----------------------------------- | ------------------------------------------------------------------------------------ |
| Tasks that cannot affect each other | A Firecracker microVM with its own kernel and disk per attempt                       |
| The same environment every time     | Versioned images: `name@version` pins one build                                      |
| Many at once                        | 100 sandboxes, 200 vCPUs and 400 GiB across running ones, on a paid account to start |
| Bursts past the limit               | Creates wait for room, up to two minutes by default or as long as you set            |
| Hung tests                          | Per-command `timeoutMs`, up to 24 hours; host-side leases                            |
| No leaked answers                   | `network: { internet: false }`, enforced outside the guest                           |
| Retries that do not double-count    | Idempotency keys on every write                                                      |
| Several samples per task            | Forks of 1 to 10 running copies; snapshots kept 1 to 365 days                        |
| Cost you can predict                | Measured CPU plus reserved memory; `maxCostMicros` per create                        |

To run more than 100 at once, write to support with the numbers you need
([pricing](/docs/pricing#how-many-at-once)).

## What it costs

Take 500 attempts, each keeping a 2 vCPU, 4 GiB sandbox running for 10 minutes
and using 120 CPU-seconds of test work:

```
CPU:    500 × 120 s / 3,600 × $0.025            = $0.42
Memory: 500 × 600 s / 3,600 × 4 GiB × $0.0075   = $2.50
Total:                                            $2.92
```

Runtime charges $0.025 per vCPU-hour of measured CPU, with a floor of a
twentieth of a vCPU, and $0.0075 per GiB-hour of memory
([pricing](/docs/pricing)). A sandbox waiting on the agent's model costs
$0.03125 an hour at this size. Model calls are billed by your model provider,
not here. New accounts get 50 free sandbox hours, no card, with eight sandboxes
at once.

## Sources

- [SWE-bench documentation](https://www.swebench.com/SWE-bench/): what the
  benchmark asks and its containerized harness, read 25 September 2026.
- [SWE-bench datasets](https://www.swebench.com/SWE-bench/guides/datasets/):
  task counts and fields, read 25 September 2026.
- [SWE-bench evaluation](https://www.swebench.com/SWE-bench/guides/evaluation/)
  and [Docker setup](https://www.swebench.com/SWE-bench/guides/docker_setup/):
  how patches are scored, `--max_workers`, and resource requirements, read
  25 September 2026.

Related: [RL environments](/use-cases/rl-environments),
[coding agent sandbox](/use-cases/coding-agent-sandbox),
[sandbox forks](/glossary/sandbox-fork),
[sandbox cost calculator](/calculators/sandbox-cost-calculator).

Facts on this page were checked on 25 September 2026.