What is SWE-bench?
SWE-bench is a benchmark of real GitHub issues: a model gets a codebase and an issue, writes a patch, and the repository's tests judge it.
Every SWE-bench attempt needs its own machine with the repository at the right commit, and on Runtime that machine is a Firecracker microVM that bills the CPU the tests use. A paid account runs 100 sandboxes at once to start, and a task image built once starts every attempt ready (agent evals and SWE-bench).
Why it matters for AI agents
SWE-bench measures coding agents on real bugs from real projects rather than puzzles written for a test. In the project's words: "Given a codebase and an issue, a language model is tasked with generating a patch that resolves the described problem." A result is a count of issues resolved, so a higher score means more real bugs fixed with the project's own tests as the judge.
That makes it an infrastructure problem as much as a model problem. Each task has its own repository, version and dependencies, and a fair run gives every attempt a clean copy of all three.
SWE-bench in facts
| Dataset | Size, as SWE-bench states it | What it is |
|---|---|---|
| SWE-bench | 2,294 instances | The full set of GitHub issues |
| SWE-bench Lite | 534 instances | "Smaller subset for quick evaluations" |
| SWE-bench Verified | 500 instances | "Expert-verified solvable problems" |
| SWE-bench Multimodal | 100 dev instances (500 test) | Problem statements with image assets |
| SWE-bench Multilingual | 300 instances | A multilingual set |
Each instance carries a repository, a base_commit, the problem_statement,
the gold patch, a test_patch, and two test lists, FAIL_TO_PASS and
PASS_TO_PASS. SWE-bench reports an instance as resolved when "the patch made
the required tests pass". It was accepted to ICLR 2024 as an oral
presentation, gained a fully containerized Docker harness on 27 June 2024, and
released Verified on 13 August 2024.
How a run works
- Build an environment per task: the repository at
base_commit, with its dependencies installed. - Give the agent the problem statement and let it produce a patch.
- Apply the patch, run the repository's tests, and record resolved or unresolved.
The official harness runs step 3 in Docker, with --max_workers setting how
many tasks run side by side; its guide suggests 8 for Lite and 12 for the full
set. On one machine that number stops where its cores and disk do.
Running it on Runtime
- One microVM per attempt, each with its own kernel and disk, so tests cannot disturb each other.
- Task images, built free from a Dockerfile, any registry image or a recipe,
and pinned with
name@version(custom images). - Internet off with
network: { internet: false }, so an attempt cannot fetch the fix. - Forks of 1 to 10 running copies for pass@k from one prepared machine (sandbox forks).
- Measured CPU: $0.025 per vCPU-hour used and $0.0075 per GiB-hour of memory, so time spent waiting on the model costs little (pricing).
The complete loop, in TypeScript and Python, with a cost worked out for 500 attempts, is in how to run SWE-bench-style agent evals.
Related
Sources
Read 25 September 2026.
- SWE-bench documentation: the definition, the harness and its milestones
- SWE-bench datasets: dataset sizes and instance fields
- SWE-bench evaluation: resolved and unresolved, and
--max_workers
Facts on this page were checked on 25 September 2026.