False positive on images

[AdeZwart]

Often we see false positives on images used, for example:

To me it feels/looks like there's some compression on the baseline, which is not applied when comparing the screenshots.

[sebinsua]

What applies the compression? I think it is Cypress#screenshot itself which saves the screenshot in png and this is meant to be lossless.

[cbrunnkvist]

I have exactly this issue too. My particular pixel diff lands at around 10% which is too high of a threshold to be used as a viable workaround...

I typically run the CI on linux hosts and development & "update snapshots" on MacOS, but they both use Electron to do so headlessly so I don't understand what could make any difference there. If it has something to do with compression, that would make more sense.

[cbrunnkvist]

I can confirm that it is failing on color - one large plain colored UI element in my example RGB #47BFBB in the reference snapshot but #40BFBC in the right-hand-side part of the diff. Maybe somewhere the image is processed/compressed twice or something. But it still feels random because the diff I get from the linux CI host is different from the diff I can reproduce on my Mac.

[jameshoward]

I think it's a colour profile issue. On OSX I have changed mine to different settings and get different colours in the screenshots.

It could be that your snapshots were generated with one colour profile and then your run is creating (and comparing) screenshots in another. This happens for us between locally on OSX and CI.

I don't think there's a way to force the colour profile that Electron uses, only Chrome, so we are seeing differences between the two as well.

[smrr723]

I too am getting false positives. In the images above, I've zoomed in to show there is actually a slight pixel difference in some of the lettering on my page. It's not limited to text only, though. Other elements on the page are showing diffs without code changes.

I'm running this in a Bitbucket pipeline after generating images locally - both the pipeline and local installations are running the same (latest) version of Cypress & Electron. Are we any closer to an answer on what is causing this?

[smrr723]

I think my issue above can be disregarded - it's a separate issue. On closer inspection, it looks like it's CSS / Operating System related, i.e the scrollbar is rounded in one image and square in the other, which I think is caused by my Mac.

Edit: my workaround for, for anyone else who encounters it, was to use docker both locally when creating my screenshots and in my pipeline (bitbucket), and used the same docker image on both. I followed the docker setup in the cypress docs:-

• https://github.com/bahmutov/cypress-open-from-docker-compose/blob/master/e2e/docker-compose.yml
• https://github.com/cypress-io/cypress-docker-images

[0xIslamTaha]

I have the same issue and as a workaround, I manage to use a docker image to generate the base screenshots and use the same docker fie with my CI to do the visual testing

[billsaysthis]

We're experiencing a similar issue. We do use Docker images (running the tests on AWS). Out issue is that text is showing in slightly different positions (letters are a few pixels to the right or left compared to the baseline images). There are no changes to the files being tested, that is no CSS or HTML changes. This only happens on some runs of some tests but enough to block us from letting the tests act as blockers for allowing a PR to be merged.

An example image is attached.

[mcintyret]

Could someone who has come up with a workaround kindly post more details of their approach? Would really appreciate it!

[mcintyret]

Actually I may have found one - adding ELECTRON_EXTRA_LAUNCH_ARGS=--disable-color-correct-rendering as an environment variable both locally and in CI seems to fix it (diff % went from ~25% to 0.1% for my case)

[acotapr]

Actually I may have found one - adding ELECTRON_EXTRA_LAUNCH_ARGS=--disable-color-correct-rendering as an environment variable both locally and in CI seems to fix it (diff % went from ~25% to 0.1% for my case)

Hi, we have the exact issue, @mcintyret tried your suggestion setting this flag but the new images are the same, nothing changes. How do you set the env var maybe I'm doing something wrong...

Edit: I think I know why it wasn't working, we were using the chrome browser, I changed it to electron and then worked for colors but still getting the fonts issues

[MatiasArriola]

Same happening here. When I run the snapshots in CI, there is a difference in the colors. I am using electron, and tried the ELECTRON_EXTRA_LAUNCH_ARGS=--disable-color-correct-rendering env variable without success. I get a 30% difference on some images, so I cannot workaround it by changing the threshold.

[0xIslamTaha]

@MatiasArriola I had the same issue and I managed to solve it by running the test cases in docker containers in my local machine and in my CI machine too.

[SunStupic]

Same issue here, trying to use docker now.

[0xIslamTaha]

@sebinsua I encourage u to use the orchestrator as a runner in your local machine or your CICD https://github.com/0xIslamTaha/orchestrator

[archfz]

See https://github.com/electron/electron/issues/10732#issuecomment-1061163988

[tomgazitbc]

Seemed that the following code fixes the color difference issue, notice that launchOptions has to be returned:

setupNodeEvents(on, config) {
            addMatchImageSnapshotPlugin(on, config);
            // force color profile
            on('before:browser:launch', (browser = null, launchOptions) => {
                if (browser.family === 'chromium' && browser.name === 'chrome') {
                    launchOptions.args.push('--force-color-profile=srgb'); // Works only in chrome
                    return launchOptions;
                }
            });
            return config;
        }

[MrCoder]

I had to change my OS color profile to sRGB so the captured screenshot match the screenshot take in Github actions.