How to reward an RL policy by running its code in sandboxes

Run each sampled program in an offline sandbox, feed it every test input with a time limit, and score the share of outputs that match.

On Runtime scoring 200,000 sampled programs costs $7.78. Each program gets its own Firecracker microVM with the internet off, so a completion that deletes files, forks without end or tries to fetch the answer harms only its own machine. A new sandbox answered its first Python command 351 ms after the create request at the median on 24 September 2026, and a command in a sandbox already in use took 105 ms (speed).

Why execution makes a good reward

For code, the cheapest trustworthy reward is to run the program. A policy that writes a function is judged by what the function returns, not by how the text looks, and the score needs no second model. The difficulty is that the policy writes code nobody has read, thousands of times a minute, and early in training much of it is broken or hostile by accident. The grader has to be:

Isolated, so one completion cannot touch another or your trainer.
Offline, so no completion can look an answer up or send a test out.
Bounded in time, so an infinite loop scores zero instead of stalling a batch.
Hard to game, so the reward measures the program, not its printouts.

Keep the verdict outside the sandbox

A reward computed inside the sandbox can be forged by the code being scored: it can print "all tests passed", edit the harness or read the expected answers. Keep the expected outputs in your trainer. Send only the input of each test case into the sandbox as standard input, bring back standard output, and compare it in your own process:

TypeScriptimport { Runtime } from "withruntime";type Case = { stdin: string; expected: string };const runtime = new Runtime({ waitForCapacityMs: 600_000 });export async function reward(program: string, cases: Case[]): Promise<number> {  await using sbx = await runtime.sandboxes.create({    vcpu: 1,    memoryMiB: 2048,    network: { internet: false },    timeoutSeconds: 300,    onLeaseEnd: "stop",    labels: { run: "grpo-2026-09-25" },  });  await sbx.files.write("/workspace/solution.py", program);  let passed = 0;  for (const test of cases) {    const run = await sbx.exec(["python3", "/workspace/solution.py"], {      stdin: test.stdin,      timeoutMs: 5_000,    });    if (!run.timedOut && run.exitCode === 0 && run.stdout.trim() === test.expected.trim()) passed++;  }  return passed / cases.length;}// A batch of completions, scored side by side.export const rewards = (programs: string[], cases: Case[]) =>  Promise.all(programs.map((program) => reward(program, cases)));

Pythonimport asynciofrom withruntime import AsyncRuntimeasync def reward(runtime, program: str, cases: list[dict]) -> float:    async with await runtime.sandboxes.create(        vcpu=1,        memory_mib=2048,        network={"internet": False},        timeout_seconds=300,        on_lease_end="stop",        labels={"run": "grpo-2026-09-25"},    ) as sbx:        await sbx.files.write("/workspace/solution.py", program)        passed = 0        for case in cases:            run = await sbx.exec(["python3", "/workspace/solution.py"],                                 stdin=case["stdin"], timeout_ms=5_000)            if not run.timed_out and run.exit_code == 0 and run.stdout.strip() == case["expected"].strip():                passed += 1        return passed / len(cases)async def rewards(programs: list[str], cases: list[dict]) -> list[float]:    async with AsyncRuntime(wait_for_capacity=600) as runtime:        return await asyncio.gather(*(reward(runtime, p, cases) for p in programs))

The command is an array, so nothing in the program's file name or arguments reaches a shell. A test that runs past five seconds comes back with timedOut: true and scores nothing. The sandbox stops when the block ends, whether the program behaved or not.

Reward design choices

Choice	What it gives	How to do it here
Share of cases passed	A dense signal early in training	`passed / cases.length`, as above
All or nothing	A strict signal once the policy is competent	Return `passed === cases.length ? 1 : 0`
Penalty for crashes	Separates wrong answers from broken programs	Score `exitCode !== 0` below a wrong answer
Penalty for slow programs	Pushes toward efficient solutions	Lower `timeoutMs`, or score on elapsed time
Unit tests, not I/O	Rewards functions rather than whole programs	Run your test file after the program, as below

For function-level tasks, write the program and a test file, run the tests and read their exit code. The tests then live in the sandbox, so treat that score as the program's own claim and keep a held-out I/O check outside it for evaluation.

Warm start for heavier graders

When a task needs packages, such as NumPy or a course's own library, prepare one sandbox, keep it as a snapshot, and create each grader from it. The copy starts with everything installed and with the source's size:

TypeScriptimport { Runtime } from "withruntime";const runtime = new Runtime();const base = await runtime.sandboxes.create({  vcpu: 1,  memoryMiB: 2048,  network: { internet: true, allow: ["pypi.org", "*.pythonhosted.org"] },});await base.exec("pip install --quiet sympy", { check: true, timeoutMs: 180_000 });await base.network.set({ internet: false });const grader = await base.snapshot({ name: "grader-v1", retentionDays: 30 });await base.stop();await using sbx = await runtime.sandboxes.create({  snapshot: grader.id,  network: { internet: false },});console.log((await sbx.exec(["python3", "-c", "import sympy; print(sympy.__version__)"])).stdout);

The registry is open only while the source installs; each grader made from the snapshot starts with the internet off. A kept snapshot is storage at $0.08 per GB per 30-day month on the bytes it alone stores (pricing).

What a code-reward grader needs

Need	How Runtime covers it
One program cannot reach another	A Firecracker microVM with its own kernel per completion
No looking answers up	`network: { internet: false }`, enforced on the host, root included
Infinite loops	`timeoutMs` per test case; the result says it timed out
A flood of printed output	At most 64 KiB of stdout and 64 KiB of stderr per result, with flags
Large batches	100 sandboxes at once on a paid account to start; more on request
Bursts past the limit	Creates wait for room (`waitForCapacityMs`) instead of failing
Identical graders every step	Create from a snapshot or a versioned custom image
A training bill with a ceiling	A daily spending limit per key; `maxCostMicros` per create

What it costs

Take a training run that scores 200,000 completions. Each keeps a 1 vCPU, 2 GiB sandbox running for 6 seconds, start to stop, and its ten test cases use 2 CPU-seconds:

TextCPU:    200,000 × 2 s / 3,600 × $0.025          = $2.78Memory: 200,000 × 6 s / 3,600 × 2 GiB × $0.0075 = $5.00Total:                                            $7.78

2 CPU-seconds in 6 seconds is above the floor of a twentieth of a vCPU, so the floor adds nothing, and compute has no one-minute minimum (pricing). Model inference is billed by whoever serves your policy. New accounts get 50 free sandbox hours, no card.

Start

Terminalnpx withruntime sandbox run --trial -- python3 -c 'print(6 * 7)'

The first run prints a link to approve in your browser. Then use the JavaScript or Python SDK as above.

Facts on this page were checked on 25 September 2026.