Write instructions and examples for testing the website UI

[yk]

there are several frameworks that simulate browser-interactions and are able to quickly test whether the UI is behaving as expected.

• [x] research the pros and cons of different frameworks
• [ ] decide one a framework (playwright or nightwatch)
• [ ] implement a few example tests
• [ ] write instructions for people who want to write their own tests

[martinnormark]

Adding my two cents on a few end2end testing frameworks.

Options you could consider:

• Selenium
• Cypress
• Playwright
• Puppeteer

Selenium

web: https://www.selenium.dev/ docs: https://www.selenium.dev/documentation/

Pros

• Selenium is a widely-used and well-established tool for end-to-end testing.
• It supports a wide variety of languages and can be used to test applications on different web browsers as it is based on the WebDriver protocol.
• It has a large user community and good documentation, which makes it easy to find help and resources.

Cons

• Selenium tests can be slow to run, especially if you have a large test suite.
• Setting up Selenium can be complex, especially if you are using it with a language or framework that it does not natively support.

Cypress

web: https://www.cypress.io/ docs: https://docs.cypress.io/guides/overview/why-cypress

Pros

• Cypress is easy to use and maintain, with a clean, intuitive interface.
• It has a short learning curve and allows you to write tests in a natural, human-readable language.
• Cypress tests run quickly, as they are executed within the browser itself.

Cons

• ~~Cypress only supports testing applications that run in Chrome.~~ It does support other browsers than Chrome, key difference from e.g. Playwright is that the browser must be installed on the system (doesn't support headless).
• It does not use the WebDriver protocol.

Playwright

web: https://playwright.dev docs: https://playwright.dev/docs/intro

Pros

• Playwright is a relatively new tool, but it has gained popularity due to its performance and ease of use.
• It allows you to write tests in a variety of languages, including TypeScript, JavaScript, Python, and C#.
• Playwright supports multiple web browsers, including Chrome, Firefox, and Safari as it is built on top of the WebDriver protocol, which is a standard for controlling web browsers programmatically.

Cons

• Playwright is a relatively new tool, so it may not have as many features or as much documentation as some other end-to-end testing frameworks.
• Only experimental support for component testing

Puppeteer

web/docs: https://pptr.dev/

Pros

• Puppeteer is a powerful tool for automating tasks and testing web applications.
• It has a large user base and good documentation, which makes it easy to find help and resources.
• Puppeteer tests run quickly, as they are executed within the browser itself.

Cons

• Puppeteer only supports testing applications that run in Chrome or Chromium.
• It can be complex to set up and use, especially if you are not familiar with Node.js.

---

In my opinion the real contenders are Cypress and Playwright.

I would probably decide on Playwright as it supports more browsers, are based on a standard (WebDriver protocol) for controlling browsers, it supports writing tests in more languages.

Thoughts?

[yk]

thank you for the detailed explanations, very cool!

your suggestion of Playwright makes total sense 👍

[fozziethebeat]

Question, is there any framework that renders the ui component to generate a screen shot image? Within Google we had some UI framework that would let us write some fake data, render the UI element, and then check in the screen shot during review.

This made it both easy to automatically test when things changed and made it easy to review UI changes. Reviewers didn't have to pull the change and run it locally, the change was captured in an image.

If some library or framework lets us do that, that's the winning framework.

[martinnormark]

is there any framework that renders the ui component to generate a screen shot image?

Yes, to my knowledge all these end2end frameworks supports both screenshot- and video capture. Some tools can also make a diff on screenshots to identify unwanted UI changes. For example Playwright's visual comparison: https://playwright.dev/docs/test-snapshots

However, for component isolation and fake data you need to combine with a unit testing framework. End2end works on the "live" website, so to achieve what you describe you'd have to create a "test host" page and serve components one at a time somehow. It could be done with some query params, dynamically load test files for a specific component etc.

So instead you should look at unit testing with something like Jest (which has a large community) as the runner and find a snapshot tool that does the job at a component level.

I don't have hands-on experience with Enzyme but they have a JSON based snapshot tool where the DOM tree is serialised and used for comparison. This approach seem to be used in multiple testing frameworks.

Same technique is outlined here: https://circleci.com/blog/snapshot-testing-with-jest/

Best source for screenshot testing at component level I've found is react-screenshot-test but it hasn't been updated for over a year.

There's a blog post here that describes how to use it: https://bestofreactjs.com/repo/fwouts-react-screenshot-test-react-component-libraries

Another one is Viteshot, which aligns well with what you're describing. You create a bunch of .screenshot.tsx/ files, runs the command and it traverses through them all and produce the screenshots. Again, seems to be maintained a little less than what you'd hope for.

So long story short. All end2end tools supports it, for component level there's for sure a way to achieve the same.

[martinnormark]

Just stumbled onto Nightwatch JS from my Github starred repos. I do not have hands-on experience with it, but it checks the same boxes as Playwright but calls out in their readme support for component isolation.

Would be worth checking the two of them out with example tests to see how they fare.

UPDATE Playwright has experimental support for component testing.

[fozziethebeat]

@martinnormark would you be interested in making an initial PR setting up either Nightwatch or Playright? They seem worth trying out and seeing if there's some hidden incompatibility.

[martinnormark]

@fozziethebeat Yes, wanted to take a stab at it but had some troubles with docker compose on Mac with M1 chip since PR #119. Do you want me to comment there, or create a new issue?

[fozziethebeat]

I filed https://github.com/LAION-AI/Open-Assistant/issues/134. If you have a chance, can you copy paste the error you're seeing into the bug?

[fozziethebeat]

For now I'll assign this bug to you. No rush in addressing this issue now if the docker bug is blocking you.

[martinnormark]

👍 I found a workaround, described in https://github.com/LAION-AI/Open-Assistant/issues/134#issuecomment-1367005345.

I have Playwright end2end for email log in running in my fork here: https://github.com/martinnormark/Open-Assistant

[martinnormark]

Turning out to be a bit of a rabbit hole... Current state, I have Playwright and Cypress implemented with a basic e2e test and a basic component test, which uses visual diff for regressions. Nice to get a refresh on some of the details.

See code needed for the two here:

• Cypress
• Playwright

General experience with both is good, however Cypress does have a much better "control panel" for viewing and running tests. Playwright relies on command line or test explorer in the IDE.

Debugging experience with Cypress is also superior, as well as writing tests with live-reload. It is really fast and you can inspect elements and get a recommended Cypress selector.

Cypress

Playwright

Cypress also wins on community clout.

Since Cypress runs tests inside the browser, as opposed to controlling the browser from the outside, it must have the browsers you want to test with installed on local dev box, Docker, CI etc. This is where Playwright is easier, you just configure the browsers and it installs them as packages for you.

Visual regression tests via screenshots is supported by both, on Cypress however it is a plugin from the community which at the moment is maintained, but similar plugins for Cypress has not been updated for a year or more.

They both have their pros and cons in writing the actual tests. Cypress does things like awaiting navigation automatically where in Playwright you need to be more explicit:

Cypress

  it("passes", () => {
    cy.visit("/auth/signin");
    cy.get(".chakra-input").type(`[email protected]{enter}`);
    cy.url().should("contain", "/auth/verify");
  });

Playwright

test("Log in page redirects after input", async ({ page }) => {
  await page.goto("/auth/signin");

  const emailInput = page.getByPlaceholder("Email Address");
  await emailInput.fill("[email protected]");
  await emailInput.press("Enter");

  await page.waitForNavigation();

  expect(page.url()).toContain("/auth/verify");
});

Based on browser support being equal, and assuming writing tests in JavaScript/TypeScript is not a problem, I would pick Cypress primarily for ease-of-use (slightly easier) and community.

~~Still want to take a look at Nightwatch, though.~~ The onboarding experience of Nightwatch is way too cumbersome compared to both Cypress and Playwright.

I'm firming up some docs for Cypress, and submitting a PR.

[yk]

wow thank you that is amazing investigation!