How to run visual regression tests with stable screenshots

Make and compare screenshot baselines in the same pinned sandbox image, so the OS, fonts and browser never change between runs.

On Runtime the machine that made a baseline can be rebuilt exactly, by version number. Each build of a custom image is a new numbered version, and a sandbox created from visual-tests@3 gets that version's files and nothing newer, on every run until you delete that version. Every run is a Firecracker microVM from that image, on the same Ubuntu 24.04 base, billed on the CPU it uses: a 2 vCPU, 4 GiB sandbox costs $0.08 an hour fully busy (pricing, 25 September 2026).

Why screenshot tests fail on the wrong machine

Playwright's toHaveScreenshot() saves reference screenshots on the first run and compares later runs against them. Its documentation warns that "browser rendering can vary based on the host OS, version, settings, hardware, power source (battery vs. power adapter), headless mode, and other factors", and advises running the tests "in the same environment where the baseline screenshots were generated" (Playwright visual comparisons).

A baseline made on a developer's laptop and compared on a CI runner fails on anti-aliasing and font hinting, not on real changes. The cure is one environment for both jobs: make baselines in the sandbox and compare in the sandbox.

Build the image once

Put the fonts and the browser in an image. Use the Playwright version your package.json pins, so the browser build matches the test runner:

TypeScriptimport { Runtime } from "withruntime";const runtime = new Runtime();const image = await runtime.images.build({  name: "visual-tests",  recipe: {    apt: ["fonts-noto-core", "fonts-noto-color-emoji"],    env: { PLAYWRIGHT_BROWSERS_PATH: "/opt/ms-playwright" },    commands: [      "npm init -y && npm install --no-fund --no-audit playwright@1.63.0",      "npx playwright install --with-deps chromium",    ],  },});console.log(`pin this: visual-tests@${image.version}`);

Pythonfrom withruntime import Runtimeruntime = Runtime()image = runtime.images.build(    name="visual-tests",    recipe={        "apt": ["fonts-noto-core", "fonts-noto-color-emoji"],        "env": {"PLAYWRIGHT_BROWSERS_PATH": "/opt/ms-playwright"},        "commands": [            "npm init -y && npm install --no-fund --no-audit playwright@1.63.0",            "npx playwright install --with-deps chromium",        ],    },)print(f"pin this: visual-tests@{image['version']}")

Recipe commands run as root; PLAYWRIGHT_BROWSERS_PATH puts the browser where the sandbox's own user finds it. Playwright 1.63.0 was the current release on 24 September 2026. Record the version the build prints in your CI config.

Compare, and bring the diffs back

Each run uploads the project, runs the suite in the pinned image and, on a failure, downloads Playwright's test-results folder, which holds the expected, actual and diff images:

TypeScriptimport { Runtime } from "withruntime";const runtime = new Runtime();const IMAGE = "visual-tests@3"; // the version your baselines were made withexport async function compareScreens(update = false) {  await using sbx = await runtime.sandboxes.create({    image: IMAGE,    diskMiB: 8192,    timeoutSeconds: 1800,  });  await sbx.files.upload("./", "/workspace/site");  await sbx.exec("npm ci", { cwd: "/workspace/site", check: true, timeoutMs: 600_000 });  const run = await sbx.exec(`npx playwright test${update ? " --update-snapshots" : ""}`, {    cwd: "/workspace/site",    timeoutMs: 1_200_000,    onStdout: (text) => process.stdout.write(text),  });  if (update)    await sbx.files.download("/workspace/site/tests", "./tests"); // the new baselines  else if (run.exitCode !== 0)    await sbx.files.download("/workspace/site/test-results", "./test-results");  return run.exitCode === 0;}

Pythonfrom withruntime import Runtimeruntime = Runtime()IMAGE = "visual-tests@3"  # the version your baselines were made withdef compare_screens(update: bool = False) -> bool:    with runtime.sandboxes.create(image=IMAGE, disk_mib=8192, timeout_seconds=1800) as sbx:        sbx.files.upload(".", "/workspace/site")        sbx.exec("npm ci", cwd="/workspace/site", check=True, timeout_ms=600_000)        flag = " --update-snapshots" if update else ""        run = sbx.exec(f"npx playwright test{flag}", cwd="/workspace/site", timeout_ms=1_200_000)        if update:            sbx.files.download("/workspace/site/tests", "./tests")  # the new baselines        elif run.exit_code != 0:            sbx.files.download("/workspace/site/test-results", "./test-results")        return run.exit_code == 0

Run it with update once to make the baselines, commit them, and run it without update on every pull request. Leave node_modules out of the upload with your own ignore step if it is large; the sandbox installs its own. Playwright's maxDiffPixels option, in the config or per assertion, sets how many pixels may differ before a test fails.

Let a reviewer see the page itself

A diff image says what changed; a reviewer often wants to click around. Start the site in the same sandbox and share its port as a private HTTPS preview:

TypeScriptimport { Sandbox } from "withruntime";await using sbx = await Sandbox.create({ image: "visual-tests@3", diskMiB: 8192 });await sbx.files.upload("./", "/workspace/site");await sbx.spawn("cd site && npm ci && npm run dev -- --port 3000");const preview = await sbx.previews.create(3000);console.log(preview.urlWithToken); // a one-time link for the reviewer's browser

The preview is private by default, under runtimehost.com, and previews.rotate(3000) refuses every token issued so far.

Screens that are not web pages

For an app that draws its own window, or a page Playwright cannot drive, the sandbox's desktop takes screenshots of the whole screen at a size you set:

TypeScriptimport { writeFile } from "node:fs/promises";import { Sandbox } from "withruntime";await using sbx = await Sandbox.create({ diskMiB: 8192 });await sbx.desktop.start({ width: 1280, height: 800 });await sbx.desktop.open("https://example.com");await writeFile("screen.png", await sbx.desktop.screenshot());

Compare the PNG against its baseline with the diff tool you already use. The first desktop start in a sandbox installs it, about 90 seconds and 1 GB of disk, once; keep a snapshot of a sandbox with it installed to skip that.

What stable screenshots need

Need	How Runtime covers it
One environment for baseline and test	A versioned image, pinned with `name@version`
The same fonts every time	`apt` fonts in the image recipe
The same browser build	Installed at build time, not at test time
Diffs to look at	`files.download` brings back `test-results`
A live look for reviewers	`previews.create(port)`, private with a token
Whole-screen captures	The desktop's `screenshot()` at a fixed width and height
Parallel runs	100 sandboxes at once on a paid account to start

What it costs

Take 400 visual test runs a month. Each keeps a 2 vCPU, 4 GiB sandbox running for 5 minutes, install included, with the browser using 1 vCPU on average. The image is stored at 2 GB:

TextCPU:    400 × 5 min / 60 × 1 vCPU × $0.025     = $0.83Memory: 400 × 5 min / 60 × 4 GiB × $0.0075     = $1.00Image:  2 GB × $0.08                           = $0.16Total:                                           $1.99

A stored image is charged on its whole file at $0.08 per GB per 30-day month; building one is free, and the free trial stores your first three images free (pricing). New accounts get 50 free sandbox hours, no card.

Start

Terminalnpx withruntime sandbox run --trial --keep -- node --version

The first run prints a link to approve in your browser. Then runtime image build and runtime image versions visual-tests manage the image from a terminal (custom images).

Sources

Checked 25 September 2026.

Playwright, visual comparisons

Facts on this page were checked on 25 September 2026.