PoC for component testing

[giamir]

This PoC introduces a potential setup to enable component testing for Stacks. Tests are executed via Web Test Runner in a browser context. Web Test Runner (WTR) is configured to launch playwright and run tests in 3 different browser contexts: chromium, firefox and webkit. WTR is also using rollup to transform transitive dependencies which are not shipped as native ESM and to handle css imports. Furthermore WTR is using mocha as the default test runner.

The tests are written leveraging the open-wc testing package which ships with chai-dom as assertion library and provide some useful testing helpers to render html fixtures in the DOM.

Finally to query the DOM and simulate user interaction, tests are using respectively @testing-library/dom and @testing-library/user-event.

Testing library was built initially to run in a nodejs context and still uses some transitive dependencies which do not ship as ESM. We need rollup and its commonjs plugin in order to make the library work in a browser context (see the web-test-runner.config.mjs file for more details). See this open issue to get a better understanding of @testing-library/dom limitations in browser contexts.

As an alternative approach we could explore using playwright directly (no WTR) in combination with these Testing Library bindings.

---

The sample tests I have added in this PR (see the tooltip.test.ts file) are supposed to showcase the value of testing for catching unexpected regressions. Not long ago we closed this bug without having to add any extra code because it was fixed as a side effects of our codebase evolving. In similar way the bug could represent itself with us being unaware: a test could help with that. For convenience I have added to the codebase a version of the tooltip where the bug was still reproducible. You can switch to that version of the code in the ts/controllers/index.ts file and observe the tests failing catching the regression.

---

Please comment about this approach and mention any alternative you would be interested on exploring for making Stacks more robust and reliable as it evolves.

[netlify[bot]]

✅ Deploy Preview for stacks ready!

Name	Link
🔨 Latest commit	414fb28f48d234194cca868e69ffd67c2d693462
🔍 Latest deploy log	https://app.netlify.com/sites/stacks/deploys/635a476bb372940008e753e2
😎 Deploy Preview	https://deploy-preview-1137--stacks.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[giamir]

FYI https://github.com/testing-library/playwright-testing-library/issues/558

@dancormier thanks for sharing this. It might be worth considering using exclusively playwright to cut down on dev dependencies. I am personally leaning on "WTR+Testing-Library" simply because it creates a familiar dev experience for me (selfishly speaking - having used jest+JSDOM and Mocha+Sinon+Chai setups to write JS unit and component tests in the past). This motivation certainly doesn't justify the tool.

Playwright APIs remind me of classic E2E testing framework like Puppeteer, Selenium, etc.... although they are trying to up their game in the components space too: https://playwright.dev/docs/test-components That stuff is still experimental but it goes into a similar direction to what WTR is doing - serving only the component(s) under test with on the fly source code transpilation and not the "whole page". Some people use the playwright API for E2E tests and WTR for unit testing (in a real browser context -> more confidence than JSDOM or similar). https://twitter.com/auniverseaway/status/1581320051684044800

By the way since we have been complaining a bit about Backstop as well in the past few days I thought I would add an extra commit showing how we could leverage the same exact WTR setup to run visual regression too. It's an option we could certainly evaluate: https://github.com/StackExchange/Stacks/pull/1137/commits/414fb28f48d234194cca868e69ffd67c2d693462 Here is some extra info: https://github.com/modernweb-dev/web/discussions/427

[dancormier]

FYI testing-library/playwright-testing-library#558

@dancormier thanks for sharing this. It might be worth considering using exclusively playwright to cut down on dev dependencies. I am personally leaning on "WTR+Testing-Library" simply because it creates a familiar dev experience for me (selfishly speaking - having used jest+JSDOM and Mocha+Sinon+Chai setups to write JS unit and component tests in the past). This motivation certainly doesn't justify the tool.

I'm into the "WTR+Testing-Library" approach for now. I think trying to squeeze our testing exclusively into playwright would be opting for simplicity at the cost of utility.

Playwright APIs remind me of classic E2E testing framework like Puppeteer, Selenium, etc.... although they are trying to up their game in the components space too: https://playwright.dev/docs/test-components That stuff is still experimental but it goes into a similar direction to what WTR is doing - serving only the component(s) under test with on the fly source code transpilation and not the "whole page". Some people use the playwright API for E2E tests and WTR for unit testing (in a real browser context -> more confidence than JSDOM or similar). https://twitter.com/auniverseaway/status/1581320051684044800

By the way since we have been complaining a bit about Backstop as well in the past few days I thought I would add an extra commit showing how we could leverage the same exact WTR setup to run visual regression too. It's an option we could certainly evaluate: 414fb28 Here is some extra info: modernweb-dev/web#427

Oh, I'm excited to check this out! Thanks for adding it.

[giamir]

Should we be committing visual regression testing screenshots to the repo?

Storing images in the repo is probably the most pragmatic and simpler approach. That said if you have already had bad experiences with that approach in the past we could look into the git-lfs addon which is supported by Github. It should be pretty straightforward to setup. https://docs.github.com/en/repositories/working-with-files/managing-large-files/configuring-git-large-file-storage

The goal should be to be able to run tests locally and in CI against the most updated baseline images. How we achieve that is secondary (there are also 3rd party commercial services which store the baseline images for you in their cloud).

The setup we currently have with backstop (not storing baseline images) is suboptimal and time expensive: contributors are not incentivised enough to run the test suite before they push their changes (which in turn diminish the value-cost ratio of our tests).

@b-kelly when you have some time (there is no rush) I'd like to hear also your opinion on the matter. 🙂

[giamir]

Archiving this PoC Testing PR in favor of this one: https://github.com/StackExchange/Stacks/pull/1194