Internal: re-enable playwright visual tests

[christianvuerings]

Summary

What changed?

Re-enable visual integration tests with these changes:

• Easier to run: Remove Docker dependency & linux screenshots
• More consistent: Only run playwright tests on macOS
• Up-to-date: Update playwright, eslint-plugin-playwright and @axe-core/playwright
• Less flaky:
  • set maxDiffPixelRatio and maxDiffPixels options
  • set retries to 3
• Faster: We now shard the runs in CI

Why?

To avoid regressions moving forward - we also saw it guards against hydration issues - e.g. #2284

Notes

Visual integration tests were disabled in #2200

[netlify[bot]]

✅ Deploy Preview for gestalt ready!

Name	Link
🔨 Latest commit	9dd004830924fdc69963a8b840ed82c5d327db8b
🔍 Latest deploy log	https://app.netlify.com/sites/gestalt/deploys/6350017f90cb060008f0e465
😎 Deploy Preview	https://deploy-preview-2286--gestalt.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[AlbertCarreras]

@christianvuerings I wouldn't merge this until you're back. We disabled them cos they were causing many problems. Here's a list of issues I found working with them.

- the error messing is not intuitive, very opaque
- vis diffs fail… God knows why, the errors seem inconsistent
- Docker requires syncing with local updates otherwise the linux test fail
        How to Configure Your Container to Sync Your Code Automatically
        1. Find the folder in your Docker container that has your code. ...
        2. Find the path to the folder on your laptop that has the same code.
        3. Add a host volume to your Docker-compose file. ...
        4. Switch from using node.js to nodemon. ...
        5. Run Docker Compose or Blimp.

- There’s no interface, says it couldn’t render the image
    - Datepicker not picking date value in linux
    - cannot run individual test in linux, but all and they all get updated
    - Playwright images don’t display well on Github
    - Another component snapshot shows an error message in  the snapshot
    
    - Fonts differ
    - ComboBox positioning is different between linux/build
- Different  behavior between local/CI

it wasn't easy and caused a lot of frustration. I'd be careful merging again

[christianvuerings]

@AlbertCarreras @dangerismycat thank you for the feedback and list of issues you encountered. It was totally fine to disable them since they blocked the team.

Couple of notes below of how we could potentially address them:

Docker requires syncing with local updates otherwise the linux test fail

Lots of issue are due to Linux / Docker - instead, I wonder if we should use macos as a container everywhere. That way we wouldn't have to run docker anywhere.

There’s no interface, says it couldn’t render the image

Could we try setting headless to false? https://playwright.dev/docs/api/class-testoptions#test-options-headless

Another component snapshot shows an error message in the snapshot

Was this #2284 ? If so, it means we now guard against hydration issues

I don't have time right now to dive deep into Playwright and why the tests were so flaky and slow

• The flakiness can most likely be resolved with maxDiffPixelRatio and / or maxDiffPixels settings: https://playwright.dev/docs/test-assertions#locator-assertions-to-have-screenshot-2
• For the slowness, we should be able to use playwright sharding in CI: https://playwright.dev/docs/test-parallel#shard-tests-between-multiple-machines

[christianvuerings]

@AlbertCarreras @dangerismycat want to take another look at this diff?

I updated the PR description with all the changes to make the playwright tests faster, less flaky & easier to update.

[christianvuerings]

@dangerismycat @AlbertCarreras heads up that I also added a section to the docs about how to update the visual snapshots for a specific test:

Are there any other processes / things we should change?

[christianvuerings]

@AlbertCarreras @dangerismycat What do you recommend as next steps for this PR? Please let me know if something is missing.

It's totally fine if we would have to disable the visual tests again if an issue would pop up. For now though, the tests should be way faster, less flaky and easier to maintain.

[dangerismycat]

Hey @christianvuerings - Sorry for the radio silence on this one. We've been pretty busy / OOO, but I'll take a look at this PR again next week and possibly merge it. Thanks for this work!