How to generate unit tests with an LLM and keep only the ones that pass

Run each model-written test in a sandbox three times and keep it only if it always passes and raises coverage.

On Runtime each candidate test runs on a fresh copy of a machine that already has the project installed, because a fork copies files, memory and running processes into 1 to 10 new sandboxes in one call. Runtime bills measured CPU at $0.025 per vCPU-hour, and a new sandbox answered its first Python command in 351 ms at the median on 24 September 2026 (speed).

Why generated tests need a gate

A model asked for tests produces files that look right. Some fail on the first run because they call a function wrongly. Some pass by asserting nothing useful. Some pass once and fail the next time, because they depend on time, order or randomness. Some cover code the suite already covered. Merging them unchecked adds noise to the suite and teaches people to ignore failures.

The gate is mechanical, and a sandbox is where it runs:

1. The test file must pass.
2. It must pass again, three runs in a row, to rule out flakiness.
3. Coverage must rise: the file has to execute lines no existing test reaches.
4. The rest of the suite must still pass with it added.

Running model-written tests on a laptop or CI runner means running model-written code there. In a sandbox it runs in a Firecracker microVM with its own kernel and, once dependencies are in, no internet.

Measure the baseline

coverage.py runs a test runner under measurement with coverage run -m, and coverage json writes the results to coverage.json (coverage.py's documentation, read 25 September 2026). Install the project once and record what is covered:

TypeScriptimport { Sandbox } from "withruntime";await using base = await Sandbox.create({ diskMiB: 8192, timeoutSeconds: 3600 });await base.exec("git clone --depth 1 https://github.com/your-org/your-lib.git repo", {  check: true,  timeoutMs: 300_000,});await base.exec("cd repo && pip install -e . pytest coverage", { check: true, timeoutMs: 600_000 });await base.exec(  "cd repo && coverage run -m pytest -q && coverage json -o /workspace/baseline.json",  {    timeoutMs: 900_000,  },);await base.network.set({ internet: false }); // nothing the model writes can reach outconst baseline = JSON.parse(await base.files.readText("/workspace/baseline.json"));console.log("baseline:", baseline.totals.percent_covered);

Try each candidate on its own copy

Fork the prepared machine, write one candidate into each copy, and apply the gate. Copies start with the project installed and the baseline on disk, and none of them can disturb another.

TypeScriptimport { Sandbox } from "withruntime";const candidates: string[] = []; // test files the model wrote, up to 10const base = await Sandbox.connect("your-prepared-sandbox-id");const copies = await base.fork({ count: Math.max(1, candidates.length) });const verdicts = await Promise.all(  copies.map(async (copy, i) => {    await copy.files.write("/workspace/repo/tests/test_generated.py", candidates[i] ?? "");    for (let run = 0; run < 3; run++) {      const r = await copy.exec("python3 -m pytest -q tests/test_generated.py", {        cwd: "/workspace/repo",        timeoutMs: 120_000,      });      if (r.exitCode !== 0) return { i, keep: false, why: r.stdout.slice(-1500) };    }    const full = await copy.exec(      "coverage run -m pytest -q && coverage json -o /workspace/after.json",      {        cwd: "/workspace/repo",        timeoutMs: 900_000,      },    );    if (full.exitCode !== 0) return { i, keep: false, why: "breaks the suite" };    const after = JSON.parse(await copy.files.readText("/workspace/after.json"));    const before = JSON.parse(await copy.files.readText("/workspace/baseline.json"));    return { i, keep: after.totals.covered_lines > before.totals.covered_lines };  }),);await Promise.all(copies.map((c) => c.stop()));console.log(verdicts);

Pythonimport jsonfrom withruntime import Sandboxcandidates = []  # test files the model wrote, up to 10base = Sandbox.connect("your-prepared-sandbox-id")copies = base.fork(count=max(1, len(candidates)))def judge(i, copy):    copy.files.write("/workspace/repo/tests/test_generated.py", candidates[i] if i < len(candidates) else "")    for _ in range(3):        r = copy.exec("python3 -m pytest -q tests/test_generated.py", cwd="/workspace/repo", timeout_ms=120_000)        if r.exit_code != 0:            return {"i": i, "keep": False, "why": r.stdout[-1500:]}    full = copy.exec("coverage run -m pytest -q && coverage json -o /workspace/after.json",                     cwd="/workspace/repo", timeout_ms=900_000)    if full.exit_code != 0:        return {"i": i, "keep": False, "why": "breaks the suite"}    after = json.loads(copy.files.read_text("/workspace/after.json"))    before = json.loads(copy.files.read_text("/workspace/baseline.json"))    return {"i": i, "keep": after["totals"]["covered_lines"] > before["totals"]["covered_lines"]}verdicts = [judge(i, copy) for i, copy in enumerate(copies)]for copy in copies:    copy.stop()print(verdicts)

A rejected candidate goes back to the model with its failure output: "this assertion failed", or "it passes but covers nothing new; here are the uncovered lines of the module". The missing_lines for each file in coverage.json are the most useful thing to put in that prompt.

Why a fork and not a fresh sandbox

Installing a real project takes minutes; a fork of a fresh sandbox takes about a second, during which the source pauses and then carries on. Each copy is billed as a sandbox of the same size would be, and the snapshot the fork makes for itself is deleted afterwards and not charged (snapshots and forks). For more than ten candidates, fork again, or keep a snapshot and create copies from it with snapshot: id.

What test generation needs

What it costs

Take 400 modules a month, each given a 12-minute session on a 2 vCPU, 4 GiB sandbox, whose test runs use 300 CPU-seconds in total:

TextCPU:    400 × 300 s / 3,600 × $0.025             = $0.83Memory: 400 × 720 s / 3,600 × 4 GiB × $0.0075    = $2.40Total:                                             $3.23

Forked copies are billed the same way, by their own running seconds and measured CPU, so the figure holds however the session splits its work between copies. Rates are on the pricing page. Fifty free hours come with every new account, and no card is asked for.

Sources

• coverage.py: run and json commands, read 25 September 2026. The totals.covered_lines, totals.percent_covered and per-file missing_lines fields were checked by running coverage.py 7.16.1 on 25 September 2026.

Related: parallel agent exploration, CI for agent pull requests, what a sandbox fork is, agent evals and SWE-bench.

Facts on this page were checked on 25 September 2026.