Sandbox cold start benchmarks what the published numbers leave out

On Runtime (withruntime.com) a new sandbox is running 102 ms after the create request reaches the API, and has run its first Python command 221 ms after it, at the median of 100 starts timed on our servers; every sample and the script that made them are public. Every sandbox provider publishes a start time, and they rarely measure the same thing. One quotes a fresh microVM, another resuming a paused one; one stops the clock when the machine boots, another when your code runs. This post takes one real run of ours, 1,800 requests timed on the server and on the laptop that sent them, and shows how far apart honest numbers for the same start can be. Then it gives you a script to time any provider the same way.

Why can't you compare published start times directly?

Because each one answers a different question, and the page rarely says which. Here is what some providers publish for starting a sandbox, in their own words (speed links each source):

Read on 28 September 2026. Most of these pages do not say where the figure was timed from, whether the network is in it, which machine size was started, or whether it is a median or a best case. Each of those choices can move a figure more than the gap between two providers. The four sections below take them one at a time, with our own numbers from the run of 28 September 2026.

Where does the clock run: on the server or on your machine?

On the server, a create is counted from the moment the request reaches the API until the API answers. Your code counts from before the request leaves until the answer is back, so it adds DNS, TLS and one round trip each way. The same 100 creates, timed both ways:

The laptop sat in the US Mountain time zone, one round trip from Virginia. From Europe or Asia the second column grows by your own round trip on every call. A figure under one round trip across the country can only have been timed on the provider's side; set it beside a server column, never beside what your own code sees.

Where does the clock stop: running, or your code's first result?

A sandbox is "running" when its machine is up. It is useful when your first command has come back. Between the two sit a second request, starting the process inside the machine, and the language runtime loading. On our servers that gap is about 119 ms at the median: 102 ms to running, 221 ms to the first python3 -c pass result. A provider that stops at "running" or "boot" is quoting the shorter number. For an agent, the second one is what the user waits for.

Which shape was started?

Many providers keep a few machines of their most common size started ahead of time (a warm pool) and hand one over when a create asks for exactly that shape. Ask for anything else and the machine is started on demand. On Runtime the default sandbox, 2 vCPU and 4 GiB, came from the warm pool in all 100 creates: 102 ms at the median. The same 2 vCPU with 2 GiB has no warm spare and is restored from a saved template instead: 398 ms, about 3.9 times as long. Both are true start times for the same product. A benchmark that only uses the default shape tells you nothing about the one your agent actually asks for, and a custom image is a third path again.

Which percentile is shown?

Medians hide the slow starts that users remember. In the same run:

The p95 is two to three times the median for each of these. "Under one second" with no percentile could be either column; a benchmark with fewer than about 50 samples cannot tell you the p99 at all (what p95 latency means).

What else moves the number?

• First time or repeat. The first snapshot of a sandbox writes its whole memory, and later ones are quicker. Say which one you timed.
• One at a time or many at once. Our run was sequential. A hundred creates in the same second meet queueing that one create never sees; time the concurrency you will really run.
• The date. Start times move with every release. On 15 October a change in how snapshots are taken cut a repeat snapshot of a 4 GiB sandbox from 1.96 s to 1.19 s at the median (changelog). A benchmark without a date and a version is a rumour.
• Errors. A run that drops its failed starts reports a better median than it earned. Count every attempt.

How do you benchmark sandbox start times yourself?

Time the step your agent waits for, from where your agent runs, for the shape it asks for, and report the tail. This script does that for Runtime: 30 sequential starts of the default shape, then 30 of a shape with no warm spare, each through its first Python result, with the median, p95 and p99. Swap the two calls inside time for another provider's create and first command to compare like with like.

TypeScriptimport { Sandbox } from "withruntime";const at = (sorted: number[], q: number) =>  Math.round(sorted[Math.min(sorted.length - 1, Math.ceil(q * sorted.length) - 1)]!);async function time(label: string, shape: { vcpu: number; memoryMiB: number }, runs = 30) {  const running: number[] = [];  const firstResult: number[] = [];  let failed = 0;  for (let i = 0; i < runs; i++) {    const started = performance.now();    try {      await using sbx = await Sandbox.create(shape);      running.push(performance.now() - started);      await sbx.exec(["python3", "-c", "pass"], { check: true });      firstResult.push(performance.now() - started);    } catch {      failed++;    }  }  for (const [step, xs] of [    ["running", running],    ["first result", firstResult],  ] as const) {    const s = [...xs].sort((a, b) => a - b);    console.log(      `${label}, ${step}: p50 ${at(s, 0.5)} ms, p95 ${at(s, 0.95)} ms, p99 ${at(s, 0.99)} ms (${s.length} ok, ${failed} failed)`,    );  }}await time("default shape", { vcpu: 2, memoryMiB: 4096 });await time("no warm spare", { vcpu: 2, memoryMiB: 2048 });

Pythonimport mathimport timefrom withruntime import Sandboxdef at(xs: list[float], q: float) -> int:    s = sorted(xs)    return round(s[min(len(s) - 1, math.ceil(q * len(s)) - 1)])def bench(label: str, vcpu: int, memory_mib: int, runs: int = 30) -> None:    running: list[float] = []    first_result: list[float] = []    failed = 0    for _ in range(runs):        started = time.perf_counter()        try:            with Sandbox.create(vcpu=vcpu, memory_mib=memory_mib) as sbx:                running.append((time.perf_counter() - started) * 1000)                sbx.exec(["python3", "-c", "pass"], check=True)                first_result.append((time.perf_counter() - started) * 1000)        except Exception:            failed += 1    for step, xs in (("running", running), ("first result", first_result)):        print(            f"{label}, {step}: p50 {at(xs, 0.5)} ms, p95 {at(xs, 0.95)} ms, "            f"p99 {at(xs, 0.99)} ms ({len(xs)} ok, {failed} failed)"        )bench("default shape", 2, 4096)bench("no warm spare", 2, 2048)

Percentiles here are nearest-rank, as in our own published runs. Run it from the server your agent loop runs on, not from a laptop, unless your users' laptops are where the loop runs. Thirty runs give a fair median and p95; for a p99 you need a few hundred.

Where are Runtime's full results?

Every step another provider quotes a time for, from create to fork, timed on the server and from a laptop, is in the speed guide, which links all 1,800 recorded samples. The script that made them is packages/cloud-guide/benchmarks/like-for-like.mts in the Runtime repository; it runs anywhere the SDK does:

Terminalnpm install withruntime && WHERE=laptop OUT=run.jsonl PRIVATE=ids.jsonl node like-for-like.mts

Runtime gives your agents Linux sandboxes that start in about a tenth of a second on the server, wake from pause as fast, and bill only measured use; try it at withruntime.com and time it yourself with the script above. For what start times add up to inside an agent loop, read cold starts in agent loops; for the other questions to ask a provider, the provider checklist.