How to generate unit tests with an LLM and keep only the ones that pass

On Runtime, each candidate test runs on a fresh copy of a machine that already has the project installed. A fork copies files, memory and running processes into 1 to 100 new sandboxes in one call. Runtime bills measured CPU at $0.025 per vCPU-hour.

Why generated tests need a gate

A model asked for tests produces files that look right. Some fail on the first run because they call a function wrongly. Some pass by asserting nothing useful. Some pass once and fail the next time, because they depend on time, order or randomness. Some cover code that the suite already covered. Merging them unchecked adds noise to the suite and teaches people to ignore failures.

The gate is mechanical, and a sandbox is where it runs:

1. The test file must pass.
2. It must pass again, three runs in a row, to rule out flakiness.
3. Coverage must rise: the file has to execute lines that no existing test reaches.
4. The rest of the suite must still pass with the new file added.

Running model-written tests on a laptop or CI runner means running model-written code there. In a sandbox, the code runs in a Firecracker microVM with its own kernel. Once the dependencies are installed, the sandbox has no internet access.

Measure the baseline

coverage.py runs a test runner under measurement when you use coverage run -m. coverage json writes the results to coverage.json (coverage.py's documentation, read 26 October 2026). Install the project once, then record what is covered:

TypeScriptimport { Sandbox } from "withruntime";await using base = await Sandbox.create({ diskMiB: 8192, timeoutSeconds: 3600 });await base.exec("git clone --depth 1 https://github.com/your-org/your-lib.git repo", {  check: true,  timeoutMs: 300_000,});await base.exec("cd repo && pip install -e . pytest coverage", { check: true, timeoutMs: 600_000 });await base.exec(  "cd repo && coverage run -m pytest -q && coverage json -o /workspace/baseline.json",  {    timeoutMs: 900_000,  },);await base.network.set({ internet: false }); // nothing the model writes can reach outconst baseline = JSON.parse(await base.files.readText("/workspace/baseline.json"));console.log("baseline:", baseline.totals.percent_covered);

Try each candidate on its own copy

Fork the prepared machine. Write one candidate into each copy, and apply the gate. Each copy starts with the project installed and the baseline on disk. No copy can disturb another.

TypeScriptimport { Sandbox } from "withruntime";const candidates: string[] = []; // test files the model wrote, up to 10const base = await Sandbox.connect("your-prepared-sandbox-id");const copies = await base.fork({ count: Math.max(1, candidates.length) });const verdicts = await Promise.all(  copies.map(async (copy, i) => {    await copy.files.write("/workspace/repo/tests/test_generated.py", candidates[i] ?? "");    for (let run = 0; run < 3; run++) {      const r = await copy.exec("python3 -m pytest -q tests/test_generated.py", {        cwd: "/workspace/repo",        timeoutMs: 120_000,      });      if (r.exitCode !== 0) return { i, keep: false, why: r.stdout.slice(-1500) };    }    const full = await copy.exec(      "coverage run -m pytest -q && coverage json -o /workspace/after.json",      {        cwd: "/workspace/repo",        timeoutMs: 900_000,      },    );    if (full.exitCode !== 0) return { i, keep: false, why: "breaks the suite" };    const after = JSON.parse(await copy.files.readText("/workspace/after.json"));    const before = JSON.parse(await copy.files.readText("/workspace/baseline.json"));    return { i, keep: after.totals.covered_lines > before.totals.covered_lines };  }),);await Promise.all(copies.map((c) => c.stop()));console.log(verdicts);

Pythonimport jsonfrom withruntime import Sandboxcandidates = []  # test files the model wrote, up to 10base = Sandbox.connect("your-prepared-sandbox-id")copies = base.fork(count=max(1, len(candidates)))def judge(i, copy):    copy.files.write("/workspace/repo/tests/test_generated.py", candidates[i] if i < len(candidates) else "")    for _ in range(3):        r = copy.exec("python3 -m pytest -q tests/test_generated.py", cwd="/workspace/repo", timeout_ms=120_000)        if r.exit_code != 0:            return {"i": i, "keep": False, "why": r.stdout[-1500:]}    full = copy.exec("coverage run -m pytest -q && coverage json -o /workspace/after.json",                     cwd="/workspace/repo", timeout_ms=900_000)    if full.exit_code != 0:        return {"i": i, "keep": False, "why": "breaks the suite"}    after = json.loads(copy.files.read_text("/workspace/after.json"))    before = json.loads(copy.files.read_text("/workspace/baseline.json"))    return {"i": i, "keep": after["totals"]["covered_lines"] > before["totals"]["covered_lines"]}verdicts = [judge(i, copy) for i, copy in enumerate(copies)]for copy in copies:    copy.stop()print(verdicts)

A rejected candidate goes back to the model with its failure output. The output says either "this assertion failed" or "it passes but covers nothing new". In the second case, include the uncovered lines of the module. The missing_lines for each file in coverage.json are the most useful input for that prompt.

Why a fork and not a fresh sandbox

Installing a real project takes minutes. A fork skips it. While the fork captures it, the source pauses, then carries on. Each copy is billed as a sandbox of the same size would be. The snapshot that the fork makes for itself is deleted afterward and is not charged (snapshots and forks). For more than 100 candidates, fork again, or keep a snapshot and create copies from it with snapshot: id.

What test generation needs

What it costs

Take 400 modules a month. Each gets a 12-minute session on a 2 vCPU, 4 GiB sandbox. The test runs use 300 CPU-seconds in total:

Forked copies are billed the same way, by their own running seconds and measured CPU. The figure holds however the session splits its work between copies. Rates are on the pricing page. Every account gets 100 free hours of a 1 GB machine, with no card required.

Sources

• coverage.py: run and json commands, read 26 October 2026. The totals.covered_lines, totals.percent_covered and per-file missing_lines fields were checked by running coverage.py 7.16.1 on 25 September 2026.

Related: parallel agent exploration, CI for agent pull requests, what a sandbox fork is, agent evals and SWE-bench.

Facts on this page were checked on 26 October 2026.