How to run visual regression tests with stable screenshots

On Runtime, the machine that made a baseline can be rebuilt exactly, by version number. Each build of a custom image creates a new numbered version. A sandbox created from visual-tests@3 gets the files of that version and nothing newer. This holds on every run until you delete that version. Every run is a Firecracker microVM created from the image, on the same Ubuntu 24.04 base. It is billed on the CPU it uses. A 2 vCPU, 4 GiB sandbox costs $0.08 an hour when fully busy (pricing, 26 October 2026).

Why screenshot tests fail on the wrong machine

Playwright's toHaveScreenshot() saves reference screenshots on the first run and compares later runs against them. Its documentation warns that "browser rendering can vary based on the host OS, version, settings, hardware, power source (battery vs. power adapter), headless mode, and other factors". It advises running the tests "in the same environment where the baseline screenshots were generated" (Playwright visual comparisons).

A baseline made on a developer's laptop and compared on a CI runner fails on anti-aliasing and font hinting, not on real changes. The fix is one environment for both jobs. Make baselines in the sandbox, and compare in the sandbox.

Build the image once

Put the fonts and the browser in an image. Use the Playwright version that your package.json pins, so the browser build matches the test runner:

TypeScriptimport { Runtime } from "withruntime";const runtime = new Runtime();const image = await runtime.images.build({  name: "visual-tests",  recipe: {    apt: ["fonts-noto-core", "fonts-noto-color-emoji"],    env: { PLAYWRIGHT_BROWSERS_PATH: "/opt/ms-playwright" },    commands: [      "npm init -y && npm install --no-fund --no-audit playwright@1.64.1",      "npx playwright install --with-deps chromium",    ],  },});console.log(`pin this: visual-tests@${image.version}`);

Pythonfrom withruntime import Runtimeruntime = Runtime()image = runtime.images.build(    name="visual-tests",    recipe={        "apt": ["fonts-noto-core", "fonts-noto-color-emoji"],        "env": {"PLAYWRIGHT_BROWSERS_PATH": "/opt/ms-playwright"},        "commands": [            "npm init -y && npm install --no-fund --no-audit playwright@1.64.1",            "npx playwright install --with-deps chromium",        ],    },)print(f"pin this: visual-tests@{image['version']}")

Recipe commands run as root. PLAYWRIGHT_BROWSERS_PATH puts the browser where the sandbox's own user can find it. Playwright 1.64.1 was the current release on 25 October 2026. Record the version that the build prints in your CI config.

Compare, and bring the diffs back

Each run uploads the project and runs the suite in the pinned image. On a failure, it downloads Playwright's test-results folder, which holds the expected, actual and diff images:

TypeScriptimport { Runtime } from "withruntime";const runtime = new Runtime();const IMAGE = "visual-tests@3"; // the version your baselines were made withexport async function compareScreens(update = false) {  await using sbx = await runtime.sandboxes.create({    image: IMAGE,    diskMiB: 8192,    timeoutSeconds: 1800,  });  await sbx.files.upload("./", "/workspace/site");  await sbx.exec("npm ci", { cwd: "/workspace/site", check: true, timeoutMs: 600_000 });  const run = await sbx.exec(`npx playwright test${update ? " --update-snapshots" : ""}`, {    cwd: "/workspace/site",    timeoutMs: 1_200_000,    onStdout: (text) => process.stdout.write(text),  });  if (update)    await sbx.files.download("/workspace/site/tests", "./tests"); // the new baselines  else if (run.exitCode !== 0)    await sbx.files.download("/workspace/site/test-results", "./test-results");  return run.exitCode === 0;}

Pythonfrom withruntime import Runtimeruntime = Runtime()IMAGE = "visual-tests@3"  # the version your baselines were made withdef compare_screens(update: bool = False) -> bool:    with runtime.sandboxes.create(image=IMAGE, disk_mib=8192, timeout_seconds=1800) as sbx:        sbx.files.upload(".", "/workspace/site")        sbx.exec("npm ci", cwd="/workspace/site", check=True, timeout_ms=600_000)        flag = " --update-snapshots" if update else ""        run = sbx.exec(f"npx playwright test{flag}", cwd="/workspace/site", timeout_ms=1_200_000)        if update:            sbx.files.download("/workspace/site/tests", "./tests")  # the new baselines        elif run.exit_code != 0:            sbx.files.download("/workspace/site/test-results", "./test-results")        return run.exit_code == 0

Run it with update once to make the baselines. Commit them. Then run it without update on every pull request. Leave node_modules out of the upload with your own ignore step if it is large. The sandbox installs its own copy. Playwright's maxDiffPixels option, in the config or per assertion, sets how many pixels may differ before a test fails.

Let a reviewer see the page itself

A diff image shows what changed. A reviewer often wants to click around the page too. Start the site in the same sandbox, and share its port as a private HTTPS preview:

TypeScriptimport { Sandbox } from "withruntime";await using sbx = await Sandbox.create({ image: "visual-tests@3", diskMiB: 8192 });await sbx.files.upload("./", "/workspace/site");await sbx.spawn("cd site && npm ci && npm run dev -- --port 3000");const preview = await sbx.previews.create(3000);console.log(preview.urlWithToken); // a link that carries its token, for the reviewer

The preview is private by default, under runtimehost.com. Calling previews.rotate(3000) refuses every token issued so far.

Screens that are not web pages

For an app that draws its own window, or a page Playwright cannot drive, the sandbox's desktop takes screenshots of the whole screen at a size you set:

TypeScriptimport { writeFile } from "node:fs/promises";import { Sandbox } from "withruntime";await using sbx = await Sandbox.create({ diskMiB: 8192 });await sbx.desktop.start({ width: 1280, height: 800 });await sbx.desktop.open("https://example.com");await writeFile("screen.png", await sbx.desktop.screenshot());

Compare the PNG against its baseline with the diff tool you already use. The desktop is ready on every server. The first desktop start in a sandbox installs nothing and takes about a second.

What stable screenshots need

What it costs

Take 400 visual test runs a month. Each run keeps a 2 vCPU, 4 GiB sandbox running for 5 minutes, install included. The browser uses 1 vCPU on average. The image is stored at 2 GB:

A stored image is billed on its accounted storage at $0.08 per GB per 30-day month. Building one is free. An account without credit stores its first three images free (pricing). Every account gets 100 free hours of a 1 GB machine in its first 30 days, with no card required.

Start

Terminalnpx withruntime sandboxes run --keep -- node --version

The first run prints a link to approve in your browser. Then runtime images build and runtime images versions visual-tests manage the image from a terminal (custom images).

Related: Playwright in a cloud sandbox, end-to-end testing in sandboxes, preview agent-built apps, a browser automation agent.

Sources

Checked 26 October 2026.

• Playwright, visual comparisons

A stored image is charged on its whole file (image storage pricing).

Facts on this page were checked on 26 October 2026.