Every SWE-bench attempt needs its own machine with the repository at the right commit. On Runtime, that machine is a Firecracker microVM. It bills the CPU the tests use. A paid account runs 100 sandboxes at once. A task image built once gives every attempt a ready start (agent evals and SWE-bench).
Why it matters for AI agents
SWE-bench measures coding agents on real bugs from real projects. It does not use puzzles written for a test. In the project's words: "Given a codebase and an issue, a language model is tasked with generating a patch that resolves the described problem." A result is a count of issues resolved. A higher score means more real bugs fixed, with the project's own tests as the judge.
That makes it an infrastructure problem as much as a model problem. Each task has its own repository, version and dependencies. A fair run gives every attempt a clean copy of all three.
SWE-bench in facts
| Dataset | Size, as SWE-bench states it | What it is |
|---|---|---|
| SWE-bench | 2,294 instances | The full set of GitHub issues |
| SWE-bench Lite | 534 instances | "Smaller subset for quick evaluations" |
| SWE-bench Verified | 500 instances | "Expert-verified solvable problems" |
| SWE-bench Multimodal | 100 dev instances (500 test) | Problem statements with image assets |
| SWE-bench Multilingual | 300 instances | A multilingual set |
Each instance has a repository, a base_commit, the problem_statement, the
gold patch, a test_patch, and two test lists: FAIL_TO_PASS and
PASS_TO_PASS. SWE-bench reports an instance as resolved when "the patch made
the required tests pass". It was accepted to ICLR 2024 as an oral
presentation. It gained a fully containerized Docker harness on 27 June 2024.
It released Verified on 13 August 2024.
How a run works
- Build an environment for each task: the repository at
base_commit, with its dependencies installed. - Give the agent the problem statement, and let it produce a patch.
- Apply the patch, run the repository's tests, and record each attempt as resolved or unresolved.
The official harness runs step 3 in Docker. --max_workers sets how many
tasks run side by side. Its guide's examples use 8 for Lite and 12 for the
full set. On one machine, that number stops where its cores and disk do.
Running it on Runtime
- One microVM per attempt. Each has its own kernel and disk, so tests cannot disturb each other.
- Task images. Build them free from a Dockerfile, any registry image or a
recipe. Pin them with
name@version(custom images). - Internet off with
network: { internet: false }, so an attempt cannot fetch the fix. - Forks of 1 to 100 running copies, for pass@k, from one prepared machine (sandbox forks).
- Measured CPU: $0.025 per vCPU-hour used and $0.0075 per GiB-hour of memory, so time spent waiting on the model costs little (pricing).
The full loop, in TypeScript and Python, with a cost for 500 attempts, is in how to run SWE-bench-style agent evals.
Related
Sources
Read 25 October 2026.
- SWE-bench documentation: the definition, the harness and its milestones
- SWE-bench datasets: dataset sizes and instance fields
- SWE-bench evaluation: resolved and unresolved, and
--max_workers
Facts on this page were checked on 25 October 2026.