Lighthouse on an arm64 VPS with Playwright's Chromium

This site has to score at least 0.95 in every Lighthouse category (Lighthouse’s API scores from 0 to 1), on the home page and on a test post of a fixture build, checked on every test run rather than once by hand. The tests run on a small ARM server with Ubuntu 24.04 and no desktop browser, and Playwright’s own arm64 Chromium is already installed for the end-to-end tests. So Lighthouse runs inside a Playwright test, against that Chromium.

Why not the Lighthouse CLI’s own Chrome

The Lighthouse CLI starts its own browser through chrome-launcher. On this server it finds none:

The CHROME_PATH environment variable must be set to a
Chrome/Chromium executable no older than Chrome stable.

Pointing CHROME_PATH at Playwright’s Chromium gets further, then stops at “waiting for dynamic debugging port”. Chromium’s error log says why:

FATAL: No usable sandbox! If you are running on Ubuntu 23.10+
or another Linux distro that has disabled unprivileged user
namespaces with AppArmor, see …

Ubuntu 23.10 and later restrict unprivileged user namespaces through AppArmor, and Chromium’s sandbox needs them. Adding --no-sandbox to --chrome-flags, next to --headless=new since there’s no display, makes the CLI work:

CHROME_PATH=~/.cache/ms-playwright/chromium-1208/chrome-linux/chrome \
  lighthouse https://example.com --chrome-flags="--headless=new --no-sandbox"

Playwright already launches Chromium without the sandbox; its chromiumSandbox option defaults to false. Running Lighthouse from a test therefore needs no extra flag, uses the same browser build as the rest of the suite, and puts its reports in the same HTML report. Turning the sandbox off is fine for auditing your own pages; for untrusted ones, see Chromium’s AppArmor notes.

Port 0 and DevToolsActivePort

Lighthouse drives the browser over the Chrome DevTools Protocol, so it needs a debugging port. A fixed port such as 9222 can collide with anything else on the server. Port 0 asks Chromium to pick a free one and write it to a file named DevToolsActivePort in its profile directory.

That’s why the test uses a persistent context: its profile directory is one the test chose, so it knows where the file will appear. chromium.launch() uses a temporary profile whose path the test never sees.

profileDir = await mkdtemp(join(tmpdir(), 'hp-lighthouse-'));
context = await chromium.launchPersistentContext(profileDir, {
  channel: 'chromium',
  args: ['--remote-debugging-port=0'],
});
port = await readDevToolsPort(profileDir);

channel: 'chromium' selects the full Chromium build in its new headless mode, rather than the separate headless shell that Playwright uses by default.

The file can appear a moment after the launch returns, so readDevToolsPort() polls it every 100 ms for up to 10 seconds, and accepts only a first line that is all digits. Each run gets a fresh mkdtemp directory, so a stale file from an earlier run can never be read. The directory is deleted afterwards.

Lighthouse’s Node API then connects to that port:

const result = await lighthouse(url, {
  port,
  output: 'html',
  logLevel: 'error',
  onlyCategories: [...CATEGORIES],
});

No configuration is passed, so it audits with the default mobile settings: an emulated phone and simulated throttling.

The scores as test annotations

Each page is one test. It writes the HTML report to test-results/lighthouse/ and attaches it to the test, so the full report opens from the Playwright report. Each category’s score becomes an annotation, which shows the numbers next to the test without opening anything. Then each category gets a soft assertion:

for (const id of CATEGORIES) {
  expect.soft(categories[id].score, `${id} on ${path}`)
    .toBeGreaterThanOrEqual(MIN_SCORE);
}

expect.soft keeps checking after a failure, so one run lists every category below 0.95, not only the first. A runtimeError in the result fails the test outright, because a run that couldn’t load the page has no meaningful scores.

What it caught

The first run failed. A post scored 0.93 on performance, with a Cumulative Layout Shift of 0.158. The display face used for titles arrived after the first paint. A size-adjusted fallback font can’t match a condensed face’s width, so when it swapped in, the title rewrapped and pushed the article down. Preloading that face next to the body face brought the shift to 0 and performance to 1.0. Every page already downloaded the file, so the fix changed only when it arrives, not how much is sent.

Both pages now score 1.0 in all four categories, and the suite fails if either drops below 0.95.