How to control a Linux desktop in a cloud sandbox

Call sbx.desktop.start(), then drive it with open, click, type and press, and call screenshot() to see what changed.

On Runtime every sandbox can run a desktop, with no separate product or image to choose. One call installs and starts a Linux desktop inside the sandbox's own Firecracker microVM and returns a private link to watch it live. Your code, or a model, then moves the mouse, types, presses keys and reads the screen back as PNG. The first start in a sandbox took about 90 seconds and 1 GB of its disk on 23 September 2026, once; later starts in that sandbox skip the install (CLI).

Start it and drive it

TypeScriptimport { writeFile } from "node:fs/promises";import { Sandbox } from "withruntime";await using sbx = await Sandbox.create({ diskMiB: 8192, timeoutSeconds: 1800 });const { streamUrl } = await sbx.desktop.start({ width: 1280, height: 800 });console.log("watch it:", streamUrl); // a private link with a one-time tokenawait sbx.desktop.open("https://example.com"); // Firefox; waits if still installingawait sbx.desktop.press("ctrl+l");await sbx.desktop.type("https://www.wikipedia.org");await sbx.desktop.press("Return");await sbx.desktop.click(640, 400);await sbx.desktop.scroll(5);await writeFile("after.png", await sbx.desktop.screenshot());

Pythonfrom withruntime import Sandboxwith Sandbox.create(disk_mib=8192, timeout_seconds=1800) as sbx:    started = sbx.desktop.start(width=1280, height=800)    print("watch it:", started["streamUrl"])    sbx.desktop.open("https://example.com")    sbx.desktop.press("ctrl+l")    sbx.desktop.type("https://www.wikipedia.org")    sbx.desktop.press("Return")    sbx.desktop.click(640, 400)    sbx.desktop.scroll(5)    with open("after.png", "wb") as file:        file.write(sbx.desktop.screenshot())

Terminalruntime sandbox desktop "${id}" start          # prints a link to watch itruntime sandbox desktop "${id}" open https://example.comruntime sandbox desktop "${id}" screenshot screen.png

Coordinates are pixels from the top left of the screen. Key names are xdotool's, space separated: ctrl+l, Return, alt+Tab.

The actions

Call (TypeScript)	Python	What it does
`start({ width, height })`	`start(width=, height=)`	Installs on first use, starts, returns `streamUrl`
`open(url)`	`open(url)`	Opens the page in Firefox
`click(x, y)`, `doubleClick`, `rightClick`	`click`, `double_click`, `right_click`	Mouse buttons at a point
`move(x, y)`, `drag([x1, y1], [x2, y2])`	`move`, `drag((x1, y1), (x2, y2))`	Move the pointer, or press, move and release
`scroll(dy, { dx })`	`scroll(dy, dx=)`	Wheel clicks; positive `dy` scrolls down
`type(text)`	`type(text)`	Types a string
`press(keys)`	`press(keys)`	Key presses and chords
`windows()`, `focus(id)`	`windows()`, `focus(id)`	Lists windows; brings one to the front
`launch(argv)`	`launch(argv)`	Starts a program on the desktop, detached
`screenshot()`	`screenshot()`	The screen as PNG bytes
`recordings.start()`, `stop(id)`	`recordings.start()`, `stop(id)`	Records the screen to MP4

Let a model drive it

A computer-use loop is three steps repeated: take a screenshot, send it to the model with the goal, and perform the action it returns.

TypeScriptimport { Sandbox } from "withruntime";type Action =  { kind: "click"; x: number; y: number } | { kind: "type"; text: string } | { kind: "done" };declare function nextAction(png: Uint8Array, goal: string): Promise<Action>; // your model callawait using sbx = await Sandbox.create({ diskMiB: 8192, timeoutSeconds: 1800 });await sbx.desktop.start();await sbx.desktop.open("https://example.com");for (let step = 0; step < 20; step++) {  const action = await nextAction(await sbx.desktop.screenshot(), "Find the contact page");  if (action.kind === "done") break;  if (action.kind === "click") await sbx.desktop.click(action.x, action.y);  if (action.kind === "type") await sbx.desktop.type(action.text);}

An agent connected to Runtime's MCP server does the same with runtime_sandbox_desktop_act and runtime_sandbox_desktop_screenshot, and records with runtime_sandbox_desktop_record (MCP).

Keep a video of what happened

TypeScriptimport { writeFile } from "node:fs/promises";import { Sandbox } from "withruntime";await using sbx = await Sandbox.create({ diskMiB: 8192 });await sbx.desktop.start();const rec = await sbx.desktop.recordings.start({ fps: 10, maxMiB: 256 });await sbx.desktop.open("https://example.com");await sbx.desktop.recordings.stop(rec.id);await writeFile("session.mp4", await sbx.desktop.recordings.download(rec.id));

A recording uses one encoder thread at low priority and at most 4 Mbit/s, so the desktop comes first. It never grows past maxMiB, stops before the disk fills, and a sandbox keeps at most 8 GiB of recordings, counted as its own disk (a desktop).

Mistakes and how Runtime handles them

Acting before the browser exists. Firefox installs in the background after the first start (about a minute and 0.5 GB more). open waits for it rather than failing.
Sharing the live link. streamUrl is a private preview that carries its own one-time token. Anyone holding it can watch, so keep it to yourself.
Running out of disk. The desktop and browser take about 1.5 GB of a default 4 GiB sandbox, which had about 2.5 GiB free when measured on 24 September 2026. The samples ask for 8 GiB.
Starting a GUI program with exec. A process that exec starts ends with its command. launch starts a program on the desktop, detached.
A sandbox left idle. A paused sandbox keeps the desktop's memory and wakes by itself on the next desktop call, so a slow model does not lose its place (pause and resume).

Where this helps

A browser automation agent that clicks through sites a script cannot.
Testing a desktop or web app the way a person uses it, with a video of the run.
For a browser run from a script rather than by sight, Playwright works in the same sandbox; to capture only images, see take a screenshot.

Start

Terminalnpx withruntime sandbox run --trial --keep -- echo ready

New accounts get 50 free sandbox hours, no card. The first run prints a link to approve in your browser.

Facts on this page were checked on 25 September 2026.