[Bug]: The image created by toHaveScreenshot exceeds the element’s bounding box size

[cat-ninja]

Version

1.49.0

Steps to reproduce

1. Clone https://github.com/cat-ninja/pw-repro
2. Run cd toHaveScreenshot\ repro/
3. Run npm install
4. Run npx playwright test
5. Check the console output for the bounding box dimensions. I.e:

npx playwright test

Running 1 test using 1 worker
[Google Chrome] › repro.spec.ts:3:5 › Image size bug @desktop @visual
{ x: 288.328125, y: 381.3828125, width: 198.296875, height: 114 } <----
  1 passed (2.7s)

To open last HTML report run:

  npx playwright show-report

5. Check the size of the created snapshot. On macOS you can run : sips -g pixelWidth -g pixelHeight tests/repro.spec.ts-snapshots/Image-size-bug-1-Google-Chrome-darwin.png
6. Compare the sizes.

Expected behavior

Created snapshot has the same size as the actual element on the page.

Actual behavior

Created snapshot is bigger in size.

toHaveScreenshot repro/tests/repro.spec.ts-snapshots/Image-size-bug-1-Google-Chrome-darwin.png
  pixelWidth: 199
  pixelHeight: 115

Additional context

The problem has occurred while trying to upgrade from 1.48.1 to 1.49.0 The browser channel is chrome.

Not sure if it’s possible to reproduce with the example I linked, but in my work project, a line of additional pixels can be observed on either the left or right side of the image. I tried to make sure nothing is "moving" by adding a waitWorTimeout() call before taking the snapshot but this didn't help.

Environment

System:
    OS: macOS 15.1
    CPU: (10) arm64 Apple M1 Pro
    Memory: 96.94 MB / 16.00 GB
  Binaries:
    Node: 21.7.3 - ~/.nvm/versions/node/v21.7.3/bin/node
    npm: 10.5.0 - ~/.nvm/versions/node/v21.7.3/bin/npm
    bun: 1.1.3 - /opt/homebrew/bin/bun
  IDEs:
    VSCode: 1.95.3 - /usr/local/bin/code
  Languages:
    Bash: 3.2.57 - /bin/bash
  npmPackages:
    @playwright/test: ^1.49.0 => 1.49.0

[Skn0tt]

Hi there! The image dimensions look roughly correct to me. I see that there's a one pixel difference - could you outline the problems it's causing for you?

[cat-ninja]

Hi there! The image dimensions look roughly correct to me. I see that there's a one pixel difference - could you outline the problems it's causing for you?

Hi @Skn0tt! One of my tests has become flaky due to this issue. I updated the snapshot using --update-snapshots, and an additional line of pixels appeared on the left side. However, when the test ran on CI, it failed, and this time the line was on the right side. Each time I run --update-snapshots locally, the line shifts, which is not something I observed with the previous version of Playwright.

[Skn0tt]

Sorry for the late response! I had another look at your repo, and switched around different @playwright/test versions. It's definitely flaky on 1.49.0, but there's also some flakiness on 1.48.1: When I ran npx playwright test --repeat-each=100, 2 out of the 100 runs failed because the screenshot was flaky.

That's known behaviour, screenshot tests are flaky because browser rendering isn't deterministic. See https://playwright.dev/docs/test-snapshots#options for more info on that. I'd recommend you either replace your screenshot expectation with some other expectation - ARIA snapshots are a great alternative, we just shipped that in 1.49.0 - or use maxDiffPixelRatio: 0.05 to allow for some rendering differences in the screenshots.