# How to reward an RL policy by running its code in sandboxes Run each sampled program in an offline sandbox, feed it every test input with a time limit, and score the share of outputs that match. **On Runtime scoring 200,000 sampled programs costs $7.78.** Each program gets its own Firecracker microVM with the internet off, so a completion that deletes files, forks without end or tries to fetch the answer harms only its own machine. A new sandbox answered its first Python command 351 ms after the create request at the median on 24 September 2026, and a command in a sandbox already in use took 105 ms ([speed](/docs/speed)). ## Why execution makes a good reward For code, the cheapest trustworthy reward is to run the program. A policy that writes a function is judged by what the function returns, not by how the text looks, and the score needs no second model. The difficulty is that the policy writes code nobody has read, thousands of times a minute, and early in training much of it is broken or hostile by accident. The grader has to be: - **Isolated**, so one completion cannot touch another or your trainer. - **Offline**, so no completion can look an answer up or send a test out. - **Bounded in time**, so an infinite loop scores zero instead of stalling a batch. - **Hard to game**, so the reward measures the program, not its printouts. ## Keep the verdict outside the sandbox A reward computed inside the sandbox can be forged by the code being scored: it can print "all tests passed", edit the harness or read the expected answers. Keep the expected outputs in your trainer. Send only the input of each test case into the sandbox as standard input, bring back standard output, and compare it in your own process: ```ts check import { Runtime } from "withruntime"; type Case = { stdin: string; expected: string }; const runtime = new Runtime({ waitForCapacityMs: 600_000 }); export async function reward(program: string, cases: Case[]): Promise { await using sbx = await runtime.sandboxes.create({ vcpu: 1, memoryMiB: 2048, network: { internet: false }, timeoutSeconds: 300, onLeaseEnd: "stop", labels: { run: "grpo-2026-09-25" }, }); await sbx.files.write("/workspace/solution.py", program); let passed = 0; for (const test of cases) { const run = await sbx.exec(["python3", "/workspace/solution.py"], { stdin: test.stdin, timeoutMs: 5_000, }); if (!run.timedOut && run.exitCode === 0 && run.stdout.trim() === test.expected.trim()) passed++; } return passed / cases.length; } // A batch of completions, scored side by side. export const rewards = (programs: string[], cases: Case[]) => Promise.all(programs.map((program) => reward(program, cases))); ``` ```python check import asyncio from withruntime import AsyncRuntime async def reward(runtime, program: str, cases: list[dict]) -> float: async with await runtime.sandboxes.create( vcpu=1, memory_mib=2048, network={"internet": False}, timeout_seconds=300, on_lease_end="stop", labels={"run": "grpo-2026-09-25"}, ) as sbx: await sbx.files.write("/workspace/solution.py", program) passed = 0 for case in cases: run = await sbx.exec(["python3", "/workspace/solution.py"], stdin=case["stdin"], timeout_ms=5_000) if not run.timed_out and run.exit_code == 0 and run.stdout.strip() == case["expected"].strip(): passed += 1 return passed / len(cases) async def rewards(programs: list[str], cases: list[dict]) -> list[float]: async with AsyncRuntime(wait_for_capacity=600) as runtime: return await asyncio.gather(*(reward(runtime, p, cases) for p in programs)) ``` The command is an array, so nothing in the program's file name or arguments reaches a shell. A test that runs past five seconds comes back with `timedOut: true` and scores nothing. The sandbox stops when the block ends, whether the program behaved or not. ## Reward design choices | Choice | What it gives | How to do it here | | ------------------------- | -------------------------------------------- | ---------------------------------------------- | | Share of cases passed | A dense signal early in training | `passed / cases.length`, as above | | All or nothing | A strict signal once the policy is competent | Return `passed === cases.length ? 1 : 0` | | Penalty for crashes | Separates wrong answers from broken programs | Score `exitCode !== 0` below a wrong answer | | Penalty for slow programs | Pushes toward efficient solutions | Lower `timeoutMs`, or score on elapsed time | | Unit tests, not I/O | Rewards functions rather than whole programs | Run your test file after the program, as below | For function-level tasks, write the program and a test file, run the tests and read their exit code. The tests then live in the sandbox, so treat that score as the program's own claim and keep a held-out I/O check outside it for evaluation. ## Warm start for heavier graders When a task needs packages, such as NumPy or a course's own library, prepare one sandbox, keep it as a snapshot, and create each grader from it. The copy starts with everything installed and with the source's size: ```ts check import { Runtime } from "withruntime"; const runtime = new Runtime(); const base = await runtime.sandboxes.create({ vcpu: 1, memoryMiB: 2048, network: { internet: true, allow: ["pypi.org", "*.pythonhosted.org"] }, }); await base.exec("pip install --quiet sympy", { check: true, timeoutMs: 180_000 }); await base.network.set({ internet: false }); const grader = await base.snapshot({ name: "grader-v1", retentionDays: 30 }); await base.stop(); await using sbx = await runtime.sandboxes.create({ snapshot: grader.id, network: { internet: false }, }); console.log((await sbx.exec(["python3", "-c", "import sympy; print(sympy.__version__)"])).stdout); ``` The registry is open only while the source installs; each grader made from the snapshot starts with the internet off. A kept snapshot is storage at $0.08 per GB per 30-day month on the bytes it alone stores ([pricing](/docs/pricing#snapshots-images-and-volumes)). ## What a code-reward grader needs | Need | How Runtime covers it | | -------------------------------- | -------------------------------------------------------------------- | | One program cannot reach another | A Firecracker microVM with its own kernel per completion | | No looking answers up | `network: { internet: false }`, enforced on the host, root included | | Infinite loops | `timeoutMs` per test case; the result says it timed out | | A flood of printed output | At most 64 KiB of stdout and 64 KiB of stderr per result, with flags | | Large batches | 100 sandboxes at once on a paid account to start; more on request | | Bursts past the limit | Creates wait for room (`waitForCapacityMs`) instead of failing | | Identical graders every step | Create from a snapshot or a versioned [custom image](/docs/images) | | A training bill with a ceiling | A daily spending limit per key; `maxCostMicros` per create | ## What it costs Take a training run that scores 200,000 completions. Each keeps a 1 vCPU, 2 GiB sandbox running for 6 seconds, start to stop, and its ten test cases use 2 CPU-seconds: ``` CPU: 200,000 × 2 s / 3,600 × $0.025 = $2.78 Memory: 200,000 × 6 s / 3,600 × 2 GiB × $0.0075 = $5.00 Total: $7.78 ``` 2 CPU-seconds in 6 seconds is above the floor of a twentieth of a vCPU, so the floor adds nothing, and compute has no one-minute minimum ([pricing](/docs/pricing)). Model inference is billed by whoever serves your policy. New accounts get 50 free sandbox hours, no card. ## Start ```bash no-run npx withruntime sandbox run --trial -- python3 -c 'print(6 * 7)' ``` The first run prints a link to approve in your browser. Then use the [JavaScript](/docs/javascript#run-commands) or [Python](/docs/python) SDK as above. Related: [RL environments for agents](/use-cases/rl-environments), [grade student code safely](/use-cases/grade-student-code), [agent evals and SWE-bench](/use-cases/agent-evals-and-swe-bench), [turn off sandbox internet](/how-to/turn-off-sandbox-internet), [sandbox cost calculator](/calculators/sandbox-cost-calculator). Facts on this page were checked on 25 September 2026.