# Sandboxes for RL environments: run agent rollouts in parallel

Prepare the environment once as a snapshot, start a fresh copy for each episode, let the policy act, score the end state, and stop the copy.

**On Runtime every episode starts from the same machine, running processes
included, and pays only for the CPU it uses.** A snapshot keeps a sandbox's
files, memory and running processes, so an environment's server is already up
when an episode begins. Runtime bills measured CPU at $0.025 per vCPU-hour and
memory at $0.0075 per GiB-hour: 10,000 ninety-second episodes on 1 vCPU and
2 GiB cost $4.79, worked out below.

## What an RL environment for agents needs

Reinforcement learning on agents runs the same loop many thousands of times:
reset the environment, let the policy take actions, observe what happened, and
compute a reward. When the actions are shell commands, file edits or code, the
environment has to be a real computer, and it has to be:

- **Identical at every reset**, or rewards are noise.
- **Isolated**, so one episode's `rm -rf` or fork bomb reaches nothing else.
- **Parallel**, because training waits on the slowest batch of rollouts.
- **Cheap while waiting**, because most of an episode is the policy thinking.

## The core loop

Once: build the environment, start what it needs, and keep it as a snapshot.
Then each episode is a fresh sandbox made from that snapshot:

```ts check
import { Runtime } from "withruntime";

const runtime = new Runtime({ waitForCapacityMs: 600_000 });

// Once: prepare the environment and keep it, server running.
const base = await runtime.sandboxes.create({
  vcpu: 1,
  memoryMiB: 2048,
  network: { internet: false },
});
await base.files.upload("./env", "/workspace/env");
await base.spawn("python3 env/server.py");
const snapshot = await base.snapshot({ name: "env-v1", retentionDays: 30 });
await base.stop();

// A policy returns the next shell command, or null to end the episode.
type Policy = (observation: string) => Promise<string | null>;

export async function rollout(policy: Policy, maxSteps = 20) {
  await using sbx = await runtime.sandboxes.create({
    snapshot: snapshot.id,
    labels: { run: "rl-2026-09-25" },
  });
  let observation = "";
  for (let step = 0; step < maxSteps; step++) {
    const action = await policy(observation);
    if (action === null) break;
    const result = await sbx.exec(action, { timeoutMs: 30_000 });
    observation = `${result.exitCode}\n${result.stdout}${result.stderr}`;
  }
  const reward = await sbx.exec(["python3", "/workspace/env/reward.py"], { timeoutMs: 60_000 });
  return Number(reward.stdout.trim());
}
```

```python check
from concurrent.futures import ThreadPoolExecutor
from withruntime import Runtime

runtime = Runtime(wait_for_capacity=600)

base = runtime.sandboxes.create(vcpu=1, memory_mib=2048, network={"internet": False})
base.files.upload("./env", "/workspace/env")
base.spawn("python3 env/server.py")
snapshot = base.snapshot(name="env-v1", retention_days=30)
base.stop()


def rollout(policy, max_steps=20):
    """policy(observation) returns the next shell command, or None to end."""
    with runtime.sandboxes.create(snapshot=snapshot["id"], labels={"run": "rl-2026-09-25"}) as sbx:
        observation = ""
        for _ in range(max_steps):
            action = policy(observation)
            if action is None:
                break
            result = sbx.exec(action, timeout_ms=30_000)
            observation = f"{result.exit_code}\n{result.stdout}{result.stderr}"
        reward = sbx.exec(["python3", "/workspace/env/reward.py"], timeout_ms=60_000)
        return float(reward.stdout.strip())


def batch(policies):
    with ThreadPoolExecutor(max_workers=100) as pool:
        return list(pool.map(rollout, policies))
```

- The copy gets the source's vCPUs, memory and disk, and its server is
  already running: nothing to install or boot per episode.
- An action that hangs comes back with `timedOut: true` after `timeoutMs`,
  which your reward function can score.
- The copy stops when the block ends, whatever the policy did.

## Branch from the middle of an episode

Some methods sample several continuations from one state. `fork` copies a
running sandbox as it is, files, memory and processes, into 1 to 10 new running
sandboxes, answered together:

```ts check
import { Sandbox } from "withruntime";

await using sbx = await Sandbox.create({ network: { internet: false } });
await sbx.exec("mkdir -p work && echo step-1 > work/state.txt");
const branches = await sbx.fork({ count: 4 });
const rewards = await Promise.all(
  branches.map(async (branch, i) => {
    await branch.exec(`echo branch-${i} >> work/state.txt`);
    return (await branch.files.readText("/workspace/work/state.txt")).length;
  }),
);
console.log(rewards);
await Promise.all(branches.map((branch) => branch.stop()));
```

```python check
from withruntime import Sandbox

with Sandbox.create(network={"internet": False}) as sbx:
    sbx.exec("mkdir -p work && echo step-1 > work/state.txt")
    branches = sbx.fork(count=4)
    for i, branch in enumerate(branches):
        branch.exec(f"echo branch-{i} >> work/state.txt")
        print(len(branch.files.read_text("/workspace/work/state.txt")))
        branch.stop()
```

The source pauses for the moment the fork takes, about a second for a fresh
sandbox, and then carries on. The snapshot a fork takes for itself is deleted
when the fork ends and is not billed ([sandbox forks](/glossary/sandbox-fork)).

## How Runtime covers an RL workload

| Need                                  | How Runtime covers it                                                             |
| ------------------------------------- | --------------------------------------------------------------------------------- |
| The same start state every episode    | Create from a snapshot: files, memory and running processes as they were          |
| Branching from a mid-episode state    | `fork({ count })`, 1 to 10 running copies                                         |
| Isolation between episodes            | A Firecracker microVM with its own kernel for every copy                          |
| Environments with their own services  | `spawn` for servers; `sudo enable-docker` for containers                          |
| Environments that need no internet    | `network: { internet: false }`, which root in the guest cannot change             |
| Runaway actions                       | Per-command `timeoutMs`; leases bound every sandbox on the host                   |
| Thousands of rollouts                 | 100 at once on a paid account to start; creates past the limit wait for room      |
| A reproducible environment definition | [Custom images](/docs/images), versioned, from a recipe, an image or a Dockerfile |
| Spend a training job cannot run past  | A daily spending limit per key; `maxCostMicros` per create                        |

To run more than 100 at once, write to support with the numbers you need
([pricing](/docs/pricing#how-many-at-once)).

## What it costs

Take 10,000 episodes, each keeping a 1 vCPU, 2 GiB sandbox running for 90
seconds, with actions and the reward check using 15 CPU-seconds:

```
CPU:    10,000 × 15 s / 3,600 × $0.025          = $1.04
Memory: 10,000 × 90 s / 3,600 × 2 GiB × $0.0075 = $3.75
Total:                                            $4.79
```

15 CPU-seconds over 90 seconds is above the floor of a twentieth of a vCPU, so
the floor adds nothing. The snapshot is storage at $0.08 per GB per 30-day
month on the bytes it alone stores: a 1 GB snapshot kept a month adds $0.08
([pricing](/docs/pricing#snapshots-images-and-volumes)). Model inference is
billed by whoever serves your policy. New accounts get 50 free sandbox hours,
no card.

## Start

```bash no-run
npx withruntime sandbox run --trial -- python3 -c 'print(6 * 7)'
```

The first run prints a link to approve in your browser. Then use the
[JavaScript](/docs/javascript#snapshots-and-forks) or
[Python](/docs/python#images-volumes-and-snapshots) SDK as above.

Related: [agent evals and SWE-bench](/use-cases/agent-evals-and-swe-bench),
[sandbox snapshots](/glossary/sandbox-snapshot),
[run untrusted LLM code safely](/use-cases/run-untrusted-llm-code),
[sandbox cost calculator](/calculators/sandbox-cost-calculator).

Facts on this page were checked on 25 September 2026.