# How to run visual regression tests with stable screenshots

Make and compare screenshot baselines in the same pinned sandbox image, so the OS, fonts and browser never change between runs.

**On Runtime the machine that made a baseline can be rebuilt exactly, by
version number.** Each build of a custom image is a new numbered version, and
a sandbox created from `visual-tests@3` gets that version's files and nothing
newer, on every run until you delete that version. Every run is a Firecracker
microVM from that image, on the same Ubuntu 24.04 base, billed on the CPU it
uses: a 2 vCPU, 4 GiB sandbox costs $0.08 an hour fully busy
([pricing](/docs/pricing), 25 September 2026).

## Why screenshot tests fail on the wrong machine

Playwright's `toHaveScreenshot()` saves reference screenshots on the first
run and compares later runs against them. Its documentation warns that
"browser rendering can vary based on the host OS, version, settings, hardware,
power source (battery vs. power adapter), headless mode, and other factors",
and advises running the tests "in the same environment where the baseline
screenshots were generated"
([Playwright visual comparisons](https://playwright.dev/docs/test-snapshots)).

A baseline made on a developer's laptop and compared on a CI runner fails on
anti-aliasing and font hinting, not on real changes. The cure is one
environment for both jobs: make baselines in the sandbox and compare in the
sandbox.

## Build the image once

Put the fonts and the browser in an image. Use the Playwright version your
`package.json` pins, so the browser build matches the test runner:

```ts check
import { Runtime } from "withruntime";

const runtime = new Runtime();
const image = await runtime.images.build({
  name: "visual-tests",
  recipe: {
    apt: ["fonts-noto-core", "fonts-noto-color-emoji"],
    env: { PLAYWRIGHT_BROWSERS_PATH: "/opt/ms-playwright" },
    commands: [
      "npm init -y && npm install --no-fund --no-audit playwright@1.63.0",
      "npx playwright install --with-deps chromium",
    ],
  },
});
console.log(`pin this: visual-tests@${image.version}`);
```

```python check
from withruntime import Runtime

runtime = Runtime()
image = runtime.images.build(
    name="visual-tests",
    recipe={
        "apt": ["fonts-noto-core", "fonts-noto-color-emoji"],
        "env": {"PLAYWRIGHT_BROWSERS_PATH": "/opt/ms-playwright"},
        "commands": [
            "npm init -y && npm install --no-fund --no-audit playwright@1.63.0",
            "npx playwright install --with-deps chromium",
        ],
    },
)
print(f"pin this: visual-tests@{image['version']}")
```

Recipe commands run as root; `PLAYWRIGHT_BROWSERS_PATH` puts the browser where
the sandbox's own user finds it. Playwright 1.63.0 was the current release on
24 September 2026. Record the version the build prints in your CI config.

## Compare, and bring the diffs back

Each run uploads the project, runs the suite in the pinned image and, on a
failure, downloads Playwright's `test-results` folder, which holds the
expected, actual and diff images:

```ts check
import { Runtime } from "withruntime";

const runtime = new Runtime();
const IMAGE = "visual-tests@3"; // the version your baselines were made with

export async function compareScreens(update = false) {
  await using sbx = await runtime.sandboxes.create({
    image: IMAGE,
    diskMiB: 8192,
    timeoutSeconds: 1800,
  });
  await sbx.files.upload("./", "/workspace/site");
  await sbx.exec("npm ci", { cwd: "/workspace/site", check: true, timeoutMs: 600_000 });
  const run = await sbx.exec(`npx playwright test${update ? " --update-snapshots" : ""}`, {
    cwd: "/workspace/site",
    timeoutMs: 1_200_000,
    onStdout: (text) => process.stdout.write(text),
  });
  if (update)
    await sbx.files.download("/workspace/site/tests", "./tests"); // the new baselines
  else if (run.exitCode !== 0)
    await sbx.files.download("/workspace/site/test-results", "./test-results");
  return run.exitCode === 0;
}
```

```python check
from withruntime import Runtime

runtime = Runtime()
IMAGE = "visual-tests@3"  # the version your baselines were made with


def compare_screens(update: bool = False) -> bool:
    with runtime.sandboxes.create(image=IMAGE, disk_mib=8192, timeout_seconds=1800) as sbx:
        sbx.files.upload(".", "/workspace/site")
        sbx.exec("npm ci", cwd="/workspace/site", check=True, timeout_ms=600_000)
        flag = " --update-snapshots" if update else ""
        run = sbx.exec(f"npx playwright test{flag}", cwd="/workspace/site", timeout_ms=1_200_000)
        if update:
            sbx.files.download("/workspace/site/tests", "./tests")  # the new baselines
        elif run.exit_code != 0:
            sbx.files.download("/workspace/site/test-results", "./test-results")
        return run.exit_code == 0
```

Run it with `update` once to make the baselines, commit them, and run it
without `update` on every pull request. Leave `node_modules` out of the
upload with your own ignore step if it is large; the sandbox installs its own.
Playwright's `maxDiffPixels` option, in the config or per assertion, sets how
many pixels may differ before a test fails.

## Let a reviewer see the page itself

A diff image says what changed; a reviewer often wants to click around. Start
the site in the same sandbox and share its port as a private HTTPS preview:

```ts check
import { Sandbox } from "withruntime";

await using sbx = await Sandbox.create({ image: "visual-tests@3", diskMiB: 8192 });
await sbx.files.upload("./", "/workspace/site");
await sbx.spawn("cd site && npm ci && npm run dev -- --port 3000");
const preview = await sbx.previews.create(3000);
console.log(preview.urlWithToken); // a one-time link for the reviewer's browser
```

The preview is private by default, under `runtimehost.com`, and
`previews.rotate(3000)` refuses every token issued so far.

## Screens that are not web pages

For an app that draws its own window, or a page Playwright cannot drive, the
sandbox's desktop takes screenshots of the whole screen at a size you set:

```ts check
import { writeFile } from "node:fs/promises";
import { Sandbox } from "withruntime";

await using sbx = await Sandbox.create({ diskMiB: 8192 });
await sbx.desktop.start({ width: 1280, height: 800 });
await sbx.desktop.open("https://example.com");
await writeFile("screen.png", await sbx.desktop.screenshot());
```

Compare the PNG against its baseline with the diff tool you already use. The
first desktop start in a sandbox installs it, about 90 seconds and 1 GB of
disk, once; keep a snapshot of a sandbox with it installed to skip that.

## What stable screenshots need

| Need                                  | How Runtime covers it                                    |
| ------------------------------------- | -------------------------------------------------------- |
| One environment for baseline and test | A versioned image, pinned with `name@version`            |
| The same fonts every time             | `apt` fonts in the image recipe                          |
| The same browser build                | Installed at build time, not at test time                |
| Diffs to look at                      | `files.download` brings back `test-results`              |
| A live look for reviewers             | `previews.create(port)`, private with a token            |
| Whole-screen captures                 | The desktop's `screenshot()` at a fixed width and height |
| Parallel runs                         | 100 sandboxes at once on a paid account to start         |

## What it costs

Take 400 visual test runs a month. Each keeps a 2 vCPU, 4 GiB sandbox running
for 5 minutes, install included, with the browser using 1 vCPU on average. The
image is stored at 2 GB:

```
CPU:    400 × 5 min / 60 × 1 vCPU × $0.025     = $0.83
Memory: 400 × 5 min / 60 × 4 GiB × $0.0075     = $1.00
Image:  2 GB × $0.08                           = $0.16
Total:                                           $1.99
```

A stored image is charged on its whole file at $0.08 per GB per 30-day month;
building one is free, and the free trial stores your first three images free
([pricing](/docs/pricing#snapshots-images-and-volumes)). New accounts get 50
free sandbox hours, no card.

## Start

```bash no-run
npx withruntime sandbox run --trial --keep -- node --version
```

The first run prints a link to approve in your browser. Then
`runtime image build` and `runtime image versions visual-tests` manage the
image from a terminal ([custom images](/docs/images)).

Related: [Playwright in a cloud sandbox](/integrations/playwright),
[end-to-end testing in sandboxes](/use-cases/end-to-end-testing),
[preview agent-built apps](/use-cases/preview-agent-built-apps),
[a browser automation agent](/use-cases/browser-automation-agent).

## Sources

Checked 25 September 2026.

- [Playwright, visual comparisons](https://playwright.dev/docs/test-snapshots)

Facts on this page were checked on 25 September 2026.