Sandboxes for RL environments: run agent rollouts in parallel

Prepare the environment once as a snapshot, start a fresh copy for each episode, let the policy act, score the end state, and stop the copy.

On Runtime every episode starts from the same machine, running processes included, and pays only for the CPU it uses. A snapshot keeps a sandbox's files, memory and running processes, so an environment's server is already up when an episode begins. Runtime bills measured CPU at $0.025 per vCPU-hour and memory at $0.0075 per GiB-hour: 10,000 ninety-second episodes on 1 vCPU and 2 GiB cost $4.79, worked out below.

What an RL environment for agents needs

Reinforcement learning on agents runs the same loop many thousands of times: reset the environment, let the policy take actions, observe what happened, and compute a reward. When the actions are shell commands, file edits or code, the environment has to be a real computer, and it has to be:

Identical at every reset, or rewards are noise.
Isolated, so one episode's rm -rf or fork bomb reaches nothing else.
Parallel, because training waits on the slowest batch of rollouts.
Cheap while waiting, because most of an episode is the policy thinking.

The core loop

Once: build the environment, start what it needs, and keep it as a snapshot. Then each episode is a fresh sandbox made from that snapshot:

TypeScriptimport { Runtime } from "withruntime";const runtime = new Runtime({ waitForCapacityMs: 600_000 });// Once: prepare the environment and keep it, server running.const base = await runtime.sandboxes.create({  vcpu: 1,  memoryMiB: 2048,  network: { internet: false },});await base.files.upload("./env", "/workspace/env");await base.spawn("python3 env/server.py");const snapshot = await base.snapshot({ name: "env-v1", retentionDays: 30 });await base.stop();// A policy returns the next shell command, or null to end the episode.type Policy = (observation: string) => Promise<string | null>;export async function rollout(policy: Policy, maxSteps = 20) {  await using sbx = await runtime.sandboxes.create({    snapshot: snapshot.id,    labels: { run: "rl-2026-09-25" },  });  let observation = "";  for (let step = 0; step < maxSteps; step++) {    const action = await policy(observation);    if (action === null) break;    const result = await sbx.exec(action, { timeoutMs: 30_000 });    observation = `${result.exitCode}\n${result.stdout}${result.stderr}`;  }  const reward = await sbx.exec(["python3", "/workspace/env/reward.py"], { timeoutMs: 60_000 });  return Number(reward.stdout.trim());}

Pythonfrom concurrent.futures import ThreadPoolExecutorfrom withruntime import Runtimeruntime = Runtime(wait_for_capacity=600)base = runtime.sandboxes.create(vcpu=1, memory_mib=2048, network={"internet": False})base.files.upload("./env", "/workspace/env")base.spawn("python3 env/server.py")snapshot = base.snapshot(name="env-v1", retention_days=30)base.stop()def rollout(policy, max_steps=20):    """policy(observation) returns the next shell command, or None to end."""    with runtime.sandboxes.create(snapshot=snapshot["id"], labels={"run": "rl-2026-09-25"}) as sbx:        observation = ""        for _ in range(max_steps):            action = policy(observation)            if action is None:                break            result = sbx.exec(action, timeout_ms=30_000)            observation = f"{result.exit_code}\n{result.stdout}{result.stderr}"        reward = sbx.exec(["python3", "/workspace/env/reward.py"], timeout_ms=60_000)        return float(reward.stdout.strip())def batch(policies):    with ThreadPoolExecutor(max_workers=100) as pool:        return list(pool.map(rollout, policies))

The copy gets the source's vCPUs, memory and disk, and its server is already running: nothing to install or boot per episode.
An action that hangs comes back with timedOut: true after timeoutMs, which your reward function can score.
The copy stops when the block ends, whatever the policy did.

Branch from the middle of an episode

Some methods sample several continuations from one state. fork copies a running sandbox as it is, files, memory and processes, into 1 to 10 new running sandboxes, answered together:

TypeScriptimport { Sandbox } from "withruntime";await using sbx = await Sandbox.create({ network: { internet: false } });await sbx.exec("mkdir -p work && echo step-1 > work/state.txt");const branches = await sbx.fork({ count: 4 });const rewards = await Promise.all(  branches.map(async (branch, i) => {    await branch.exec(`echo branch-${i} >> work/state.txt`);    return (await branch.files.readText("/workspace/work/state.txt")).length;  }),);console.log(rewards);await Promise.all(branches.map((branch) => branch.stop()));

Pythonfrom withruntime import Sandboxwith Sandbox.create(network={"internet": False}) as sbx:    sbx.exec("mkdir -p work && echo step-1 > work/state.txt")    branches = sbx.fork(count=4)    for i, branch in enumerate(branches):        branch.exec(f"echo branch-{i} >> work/state.txt")        print(len(branch.files.read_text("/workspace/work/state.txt")))        branch.stop()

The source pauses for the moment the fork takes, about a second for a fresh sandbox, and then carries on. The snapshot a fork takes for itself is deleted when the fork ends and is not billed (sandbox forks).

How Runtime covers an RL workload

Need	How Runtime covers it
The same start state every episode	Create from a snapshot: files, memory and running processes as they were
Branching from a mid-episode state	`fork({ count })`, 1 to 10 running copies
Isolation between episodes	A Firecracker microVM with its own kernel for every copy
Environments with their own services	`spawn` for servers; `sudo enable-docker` for containers
Environments that need no internet	`network: { internet: false }`, which root in the guest cannot change
Runaway actions	Per-command `timeoutMs`; leases bound every sandbox on the host
Thousands of rollouts	100 at once on a paid account to start; creates past the limit wait for room
A reproducible environment definition	Custom images, versioned, from a recipe, an image or a Dockerfile
Spend a training job cannot run past	A daily spending limit per key; `maxCostMicros` per create

To run more than 100 at once, write to support with the numbers you need (pricing).

What it costs

Take 10,000 episodes, each keeping a 1 vCPU, 2 GiB sandbox running for 90 seconds, with actions and the reward check using 15 CPU-seconds:

TextCPU:    10,000 × 15 s / 3,600 × $0.025          = $1.04Memory: 10,000 × 90 s / 3,600 × 2 GiB × $0.0075 = $3.75Total:                                            $4.79

15 CPU-seconds over 90 seconds is above the floor of a twentieth of a vCPU, so the floor adds nothing. The snapshot is storage at $0.08 per GB per 30-day month on the bytes it alone stores: a 1 GB snapshot kept a month adds $0.08 (pricing). Model inference is billed by whoever serves your policy. New accounts get 50 free sandbox hours, no card.

Start

Terminalnpx withruntime sandbox run --trial -- python3 -c 'print(6 * 7)'

The first run prints a link to approve in your browser. Then use the JavaScript or Python SDK as above.

Facts on this page were checked on 25 September 2026.