Sandboxes for RL environments run agent rollouts in parallel

On Runtime, every episode starts from the same machine, running processes included, and pays only for the CPU it uses. A snapshot keeps a sandbox's files, memory and running processes, so an environment's server is already up when an episode begins. Runtime bills measured CPU at $0.025 per vCPU-hour and memory at $0.0075 per GiB-hour. For example, 10,000 ninety-second episodes on 1 vCPU and 2 GiB cost $4.79. The working is below.

What an RL environment for agents needs

Reinforcement learning on agents runs the same loop many thousands of times. Reset the environment, let the policy take actions, observe what happened, and compute a reward. When the actions are shell commands, file edits or code, the environment has to be a real computer. It has to be:

• Identical at every reset, or rewards are noise.
• Isolated, so one episode's rm -rf or fork bomb reaches nothing else.
• Parallel, because training waits on the slowest batch of rollouts.
• Cheap while waiting, because most of an episode is the policy's thinking time.

The core loop

Once, build the environment, start what it needs, and keep it as a snapshot. Then make each episode a fresh sandbox from that snapshot:

TypeScriptimport { Runtime } from "withruntime";const runtime = new Runtime({ waitForCapacityMs: 600_000 });// Once: prepare the environment and keep it, server running.const base = await runtime.sandboxes.create({  vcpu: 1,  memoryMiB: 2048,  network: { internet: false },});await base.files.upload("./env", "/workspace/env");await base.spawn("python3 env/server.py");const snapshot = await base.snapshot({ name: "env-v1", retentionDays: 30 });await base.stop();// A policy returns the next shell command, or null to end the episode.type Policy = (observation: string) => Promise<string | null>;export async function rollout(policy: Policy, maxSteps = 20) {  await using sbx = await runtime.sandboxes.create({    snapshot: snapshot.id,    labels: { run: "rl-2026-09-25" },  });  let observation = "";  for (let step = 0; step < maxSteps; step++) {    const action = await policy(observation);    if (action === null) break;    const result = await sbx.exec(action, { timeoutMs: 30_000 });    observation = `${result.exitCode}\n${result.stdout}${result.stderr}`;  }  const reward = await sbx.exec(["python3", "/workspace/env/reward.py"], { timeoutMs: 60_000 });  return Number(reward.stdout.trim());}

Pythonfrom concurrent.futures import ThreadPoolExecutorfrom withruntime import Runtimeruntime = Runtime(wait_for_capacity=600)base = runtime.sandboxes.create(vcpu=1, memory_mib=2048, network={"internet": False})base.files.upload("./env", "/workspace/env")base.spawn("python3 env/server.py")snapshot = base.snapshot(name="env-v1", retention_days=30)base.stop()def rollout(policy, max_steps=20):    """policy(observation) returns the next shell command, or None to end."""    with runtime.sandboxes.create(snapshot=snapshot["id"], labels={"run": "rl-2026-09-25"}) as sbx:        observation = ""        for _ in range(max_steps):            action = policy(observation)            if action is None:                break            result = sbx.exec(action, timeout_ms=30_000)            observation = f"{result.exit_code}\n{result.stdout}{result.stderr}"        reward = sbx.exec(["python3", "/workspace/env/reward.py"], timeout_ms=60_000)        return float(reward.stdout.strip())def batch(policies):    with ThreadPoolExecutor(max_workers=100) as pool:        return list(pool.map(rollout, policies))

• The copy gets the source's vCPUs, memory and disk. Its server is already running, so there is nothing to install or boot per episode.
• An action that hangs comes back with timedOut: true after timeoutMs, which your reward function can score.
• The copy is deleted when the block ends, whatever the policy did.

Branch from the middle of an episode

Some methods sample several continuations from one state. fork copies a running sandbox as it is, with its files, memory and processes. It returns 1 to 100 new running sandboxes together:

TypeScriptimport { Sandbox } from "withruntime";await using sbx = await Sandbox.create({ network: { internet: false } });await sbx.exec("mkdir -p work && echo step-1 > work/state.txt");const branches = await sbx.fork({ count: 4 });const rewards = await Promise.all(  branches.map(async (branch, i) => {    await branch.exec(`echo branch-${i} >> work/state.txt`);    return (await branch.files.readText("/workspace/work/state.txt")).length;  }),);console.log(rewards);await Promise.all(branches.map((branch) => branch.stop()));

Pythonfrom withruntime import Sandboxwith Sandbox.create(network={"internet": False}) as sbx:    sbx.exec("mkdir -p work && echo step-1 > work/state.txt")    branches = sbx.fork(count=4)    for i, branch in enumerate(branches):        branch.exec(f"echo branch-{i} >> work/state.txt")        print(len(branch.files.read_text("/workspace/work/state.txt")))        branch.stop()

The source pauses while the fork captures it. It runs again when the call returns, then carries on. The snapshot that a fork takes for itself is deleted when the fork ends, and it is not billed (sandbox forks).

How Runtime covers an RL workload

To run more than 100 sandboxes at the same time, write to support with the numbers you need (pricing).

What it costs

Take 10,000 episodes. Each keeps a 1 vCPU, 2 GiB sandbox running for 90 seconds. Actions and the reward check use 15 CPU-seconds:

15 CPU-seconds over 90 seconds is above the floor of a twentieth of a vCPU, so the floor adds nothing. The snapshot is billed as storage at $0.08 per GB per 30-day month, on the bytes it alone stores. For example, a 1 GB snapshot kept for a month adds $0.08 (pricing). Whoever serves your policy bills the model inference. Every account gets 100 free hours of a 1 GB machine in its first 30 days, with no card.

Start

Terminalprintf '%s\n' 'print(sum([125, 250, 375]))' > invoice.pynpx withruntime sandboxes run -- python3 - < invoice.py

The first run prints a link to approve in your browser. Then use the JavaScript or Python SDK as above.

Related: agent evals and SWE-bench, sandbox snapshots, run untrusted LLM code safely, sandbox cost calculator.

Facts on this page were checked on 26 October 2026.