[BUG] Same test, different runs causes different heights with a weird compression effect

[filipesmedeiros]

Context:

• Playwright Version: 1.27.1
• Operating System: MacOS 13.0
• Node.js version: 16.15.0
• Browser: All
• Extra: Also CI using MacOS 12 (Github Actions)

Describe the bug

Subsequent screenshots (on different test runs) of the same page are coming out with different heights. This is mainly due to a weird "compression" (like smashing) effect on one of the screenshots.

These two example come from the same code, same server (NextJS locally) running, just yarn test two times in a row:

I'm sorry for the long image, but that may be part of the problem (although short pages sometimes have this problem too).

If you compare the images using something like https://www.diffchecker.com/image-diff/ you can see the weird effect.

Is this a known bug? The differences are even greater once we run the tests on CI (more screenshots are different), but it's flaky, so sometimes it happens and sometimes everything is good.

EDIT: Just to clarify, by compression I mean that the pixels themselves are smashed, so everything is shorter, which in turn causes the heights of the screenshots to differ, like on of them was "smashed", which is wrong, e.g. causes the square logo to turn into a rectangle, etc

Thanks in advance!

[pavelfeldman]

This looks like a bug to me. My only idea would be that the actual pixel data and the size of the page are captured in a racy manner and then the image is scaled based on the wrong size. Could you try waiting for layout to settle before you capture an image a little? Also, await expect.toHaveScreenshot should wait for the image to stabilize, are you using it?

[filipesmedeiros]

@pavelfeldman I used shouldMatchSnapshot on all runs, including the first. Is that bad? I know the docs say to use toHaveScreenshot but since shouldMatch also writes if not present, I assumed it was the same.

I can test with that, though! Thanks for the reply!

[filipesmedeiros]

Also, it's supposed to be expect(await page.screenshot).toMatchScreenshot right? We don't have to await expect like you said, I think (just checking)

[pavelfeldman]

We do, that's the point of resilient screenshotting, should be await expect(page).toHaveScreenshot().

[filipesmedeiros]

@pavelfeldman But expect doesn't return a Promise 😕

[filipesmedeiros]

Wait my bad, I'm not awaiting expect, but toHaveScreenshot. But toMatchSnapshot does not return a Promise

[filipesmedeiros]

Ok I see what's happening: I'm doing expect(await page.screenshot).toMatchSnapshot (like documented here https://playwright.dev/docs/test-assertions#screenshot-assertions-to-match-snapshot-1) but maybe that's wrong and I should be doing await expect(page).toHaveScreenshot. Let me try and I'll get back to you!

[filipesmedeiros]

Same thing happens 😞 e.g. in the above image you can see a 1 pixel height different between screenshots

[pavelfeldman]

That is something we might be able to fix!

[filipesmedeiros]

Ok I don't know if it helps, but running it inside Docker Linux (and CI on Linux) fixed it (or at least all the tests passed).

So the issue seems to be specific for Mac!

EDIT: nevermind, I think it was a flaky test that passed :(

[thraizz]

Wanted to report in that this happens on my team's MacBooks as well. We run both M1 and Intel chips. Our CI runs the official image. Locally generated, fullPage screenshots differ in 1px height from the CI screenshots.

We can mitigate this by generating screenshots in a docker image, but a fix for this would be awesome. If I can help anyhow, let me know :)

Playwright Version: 1.29.2 Playwright config: https://gist.github.com/thraizz/0b2ab81f31925265a467e75fec58935f macOS 13.1

[Dwlad90]

Any update on this issue?

[kr-anastasiia]

We are also experiencing the same problem. When we run the test using 'run test button' and save screenshots we don't get any problems. But when we try to run test using command line we get the error. This only applies to Google Chrome. Mac OS 13.2.1, M1 Pro

[maartenbreddels]

We had this issue as well (in https://github.com/widgetti/solara/pull/67), we had a screenshot that had a different width sometimes. Based on https://github.com/microsoft/playwright/issues/20015#issuecomment-1427285592, we had the idea of printing out the bounding box.

• On failure we got: {'x': 416.765625, 'y': 611, 'width': 134.5, 'height': 36}, and our resulting image was 136x36
• On success we got: {'x': 398, 'y': 611, 'width': 134.5, 'height': 36}, and our resulting image was 135x36

So I think it is a combination of something else on the page pushing an element to a non-integer pixel position (but not stable every time) that can cause an image to change by 1-pixel width or height.

Hopefully, this is useful for others and also the cause of this issue.

[dabbottQA]

Unfortunately this issue is limiting our ability to write and run concise tests. Our Bamboo test runs fail with this 1px issue. We have been able to work around the issue by specifying locators that capture a broader area which is not always ideal.

Error: Screenshot comparison failed:

  Expected an image 816px by 96px, received 817px by 96px. 

[maartenbreddels]

Did you check if your bounding box has a non integer x coordinate?

[adelara]

So I think it is a combination of something else on the page pushing an element to a non-integer pixel position (but not stable every time) that can cause an image to change by 1-pixel width or height.

You know, this is exactly the thought I have... and it's the same system we use over and over to run the tests. We run the stuff on a CI/CD pipeline, inside a Linux container and develop on Windows. We don't save the Windows generated images. When the images are captured on the container, it is then sent to a separate server where we keep the benchmarked images, and the other server does the comparison. When it works, great but more than 70% of time, we get a major pain in the arse as the images are different, sometimes by a couple pixels here and there, or by generating images with different sizes, which invalidates the image comparison. Another annoyance is that text renders in slightly different positions or the spacing between lines is different (this also causes the image to have different size).

So, yes, anything to help, is appreciated :-)

[mumair-pc]

Any update on this?

[pascalknupper]

Is there any progress on this issue or do you know some workaround for this issue:

Expected an image 768px by 1465px, received 768px by 1463px.

[afonte15]

Is there any updates on this issue or a workaround?

• Expected an image 201px by 35px, received 201px by 36px.

[maartenbreddels]

I'm not sure it's a playwright issue, I think it's a combination of something unstable left or above an elements which causes the bounding box to be at floating point coordinates (see https://github.com/microsoft/playwright/issues/18827#issuecomment-1513345600) which when rounded to integer pixels positions will sometimes jump by one pixel.

[unikitty37]

I think I'm seeing something similar, except it's the width, not the height.

If I generate the screenshot using a headless Chromium (either from pnpm run playwright test or using the VSCode extension with "Reuse Browser" set to false) it generates a different width screenshot from the one generated by using the VSCode extension with "Reuse Browser" set to true:

Expected an image 1280px by 1108px, received 1265px by 1108px.

Additionally, if I use the mask option, the mask is placed in the wrong place if generating the screenshot with "Reuse Browser" set to true. In these examples, the second year (2023) should have been masked.

Reuse Browser is false (correct positioning)

Reuse Browser is true (offset positioning)

(I was going to raise a separate issue for this, but it feels like it might be related to this one. Let me know if you want it raised separately!)

This is with Playwright 1.37.1, Playwright Test for VSCode version v1.0.15, macOS macOS 13.5 (22G74) on Apple M1 Max, and whatever Playwright browsers were current at 09:00 (Europe/London) on 2023-08-30 :)

[nikolay-yavorovskiy]

We have this issue too, the fix would be very helpful, this issue is causing a lot of flaky tests...

[JBill85]

This is happening for us as well... It's driving me crazy. Would be nice to be able to specify some tolerance for this

Expected an image 161px by 161px, received 161px by 160px

[PeterHewat]

Hello, we are also having this issue. Tests are always running inside the same environement (GitHub Action Linux Chrome headless)

Expected an image 344px by 335px, received 344px by 333px.
Expected an image 704px by 580px, received 702px by 580px.
Expected an image 1058px by 330px, received 1058px by 329px.
...

Using the latest playwright 1.38.1

[JBill85]

Hello, I'm curious if there are any plans to look at this issue. I posted last month and never heard from anyone.

[maartenbreddels]

I really think the issue is not with playwright: https://github.com/microsoft/playwright/issues/18827#issuecomment-1513345600 but having a bounding box with non-integer values (e.g 3.5) in combination with a non-stable page (e.g. fonts still loading).

[JBill85]

I really think the issue is not with playwright: #18827 (comment) but having a bounding box with non-integer values (e.g 3.5) in combination with a non-stable page (e.g. fonts still loading).

I mean they could give us the option to turn this check off or something. I'm using the image checks for egregious failures like the picture is missing. Not really interested in the way the failure works right now.

[maartenbreddels]

If the failure is due to what I describe, I don't think there is simple solution. It's something else in the page that is not stable (playwright cannot be blamed for that) that causes the rounded off width and height to differ by 1 or 2 pixels. Maybe playwright could warn/error when taking screenshots of non-integer bounding boxes.

[paulsmithkc]

@maartenbreddels PR submitted to a add configurable tolerance on the size of the image. https://github.com/microsoft/playwright/pull/27513

Due to floating point error in both CSS and JS, it isn't practical/possible to ensure exact pixel sizing in the vast ecosystem of tools and frameworks.