[Feedback]: Experimental ssim-cie94 comparator

[samschurter]

Version

1.45.3

Steps to reproduce

1. Use the test to take a screenshot of a page with an image locally using the headful browser using the experimental {_comparator: 'ssim-cie94'} configuration.
2. Run the same test in CI using the headless=new flag when launching the browser
3. Observe that the screenshot comparison fails
4. Adjust the threshold setting to be more permissive and run again
5. Observe that the comparison still fails with the same diff.
6. Swap the comparator for the default pixelmatch and run the test again
7. Observe that the diff is already much more lenient about small color changes.

Expected behavior

The ssim-cie94 should be more resilient to differences between rendering engines, even when not comparing text, or should have configuration options for extra lenience when required.

Actual behavior

ssim-cie94 results in worse matches than pixelmatch in some circumstances.

Additional context

This isn't really a bug report as I don't think there is any actual bug, just feedback on the ssim-cie94 comparator as requested in a couple issues. Reproduction: https://github.com/samschurter/playwright-threshold-bug

Environment

System:
    OS: macOS 14.5
    CPU: (12) arm64 Apple M3 Pro
    Memory: 7.15 GB / 36.00 GB
  Binaries:
    Node: 22.5.1 - ~/.nvm/versions/node/v22.5.1/bin/node
    npm: 10.8.2 - ~/.nvm/versions/node/v22.5.1/bin/npm
  IDEs:
    VSCode: 1.92.0 - /Applications/Visual Studio Code.app/Contents/Resources/app/bin/code
  Languages:
    Bash: 3.2.57 - /bin/bash

GitHub Actions CI environment:
- Ubuntu 22.04
- 4 amd64 vCPUs
- Node 20

[yury-s]

Thanks for the feedback!

Related issue: https://github.com/microsoft/playwright/issues/24312

[ThibaultFy]

Hello 👋

We would be interested to try the ssim comparator as part of our project with toHaveScreenshot. Is there a way to access it already on playwright ?

Thanks a lot !