# What is SWE-bench?

SWE-bench is a benchmark of real GitHub issues: a model gets a codebase and an issue, writes a patch, and the repository's tests judge it.

**Every SWE-bench attempt needs its own machine with the repository at the
right commit, and on Runtime that machine is a Firecracker microVM that bills
the CPU the tests use.** A paid account runs 100 sandboxes at once to start,
and a task image built once starts every attempt ready
([agent evals and SWE-bench](/use-cases/agent-evals-and-swe-bench)).

## Why it matters for AI agents

SWE-bench measures coding agents on real bugs from real projects rather than
puzzles written for a test.
In the project's words: "Given a codebase and an issue, a language model is
tasked with generating a patch that resolves the described problem." A result
is a count of issues resolved, so a higher score means more real bugs fixed
with the project's own tests as the judge.

That makes it an infrastructure problem as much as a model problem. Each task
has its own repository, version and dependencies, and a fair run gives every
attempt a clean copy of all three.

## SWE-bench in facts

| Dataset                | Size, as SWE-bench states it | What it is                             |
| ---------------------- | ---------------------------- | -------------------------------------- |
| SWE-bench              | 2,294 instances              | The full set of GitHub issues          |
| SWE-bench Lite         | 534 instances                | "Smaller subset for quick evaluations" |
| SWE-bench Verified     | 500 instances                | "Expert-verified solvable problems"    |
| SWE-bench Multimodal   | 100 dev instances (500 test) | Problem statements with image assets   |
| SWE-bench Multilingual | 300 instances                | A multilingual set                     |

Each instance carries a repository, a `base_commit`, the `problem_statement`,
the gold `patch`, a `test_patch`, and two test lists, `FAIL_TO_PASS` and
`PASS_TO_PASS`. SWE-bench reports an instance as resolved when "the patch made
the required tests pass". It was accepted to ICLR 2024 as an oral
presentation, gained a fully containerized Docker harness on 27 June 2024, and
released Verified on 13 August 2024.

## How a run works

1. Build an environment per task: the repository at `base_commit`, with its
   dependencies installed.
2. Give the agent the problem statement and let it produce a patch.
3. Apply the patch, run the repository's tests, and record resolved or
   unresolved.

The official harness runs step 3 in Docker, with `--max_workers` setting how
many tasks run side by side; its guide suggests 8 for Lite and 12 for the full
set. On one machine that number stops where its cores and disk do.

## Running it on Runtime

- **One microVM per attempt,** each with its own kernel and disk, so tests
  cannot disturb each other.
- **Task images,** built free from a Dockerfile, any registry image or a recipe,
  and pinned with `name@version` ([custom images](/docs/images)).
- **Internet off** with `network: { internet: false }`, so an attempt cannot
  fetch the fix.
- **Forks** of 1 to 10 running copies for pass@k from one prepared machine
  ([sandbox forks](/glossary/sandbox-fork)).
- **Measured CPU:** $0.025 per vCPU-hour used and $0.0075 per GiB-hour of
  memory, so time spent waiting on the model costs little
  ([pricing](/docs/pricing)).

The complete loop, in TypeScript and Python, with a cost worked out for 500
attempts, is in [how to run SWE-bench-style agent evals](/use-cases/agent-evals-and-swe-bench).

## Related

- [RL environments](/use-cases/rl-environments)
- [Coding agent sandbox](/use-cases/coding-agent-sandbox)
- [What is a sandbox snapshot?](/glossary/sandbox-snapshot)
- [What is an agent sandbox?](/glossary/agent-sandbox)

## Sources

Read 25 September 2026.

- [SWE-bench documentation](https://www.swebench.com/SWE-bench/): the definition, the harness and its milestones
- [SWE-bench datasets](https://www.swebench.com/SWE-bench/guides/datasets/): dataset sizes and instance fields
- [SWE-bench evaluation](https://www.swebench.com/SWE-bench/guides/evaluation/): resolved and unresolved, and `--max_workers`

Facts on this page were checked on 25 September 2026.