ionic-framework icon indicating copy to clipboard operation
ionic-framework copied to clipboard

test(ci): run tests in docker container

Open liamdebeasi opened this issue 4 months ago • 0 comments

Issue number: Internal


What is the current behavior?

The team currently faces two challenges:

  1. Verifying visual changes locally is difficult because we cannot use the existing ground truths as they were generated in Linux environments and most of our team uses either macOS or Windows. While team members can generate ground truths in the correct environment, they need to remember to do that first before making changes.
  2. Updating visual diffs is time consuming and can only be done by team members. Our GitHub Action runs the entire test suite which can take ~10 even if only a handful of screenshots are generated. Additionally, this job can only be run by team members meaning community contributors cannot update/add screenshots. This limits them to non-visual tasks when contributing. In the event that they do want to make visual changes, the team needs to copy all their code into a branch and manually run screenshot diffs for them.

What is the new behavior?

  • This PR introduces the ability to run all Playwright tests inside of a Docker container using an image with Playwright dependencies. The container will have access to the local project, so developers can make changes and then run tests in the container after the changes are compiled. This enables anyone to propose new screenshot changes. However, the "update screenshot" job will still be available for folks who do not want/are unable to use docker.
  • There are some typeface differences between GH Actions and the Docker image which is why there are a handful of screenshots that needed to be updated.

One risk here is that the Playwright npm and Docker image versions must be kept in sync. As a result, I also migrated us to use Renovate to allow us to auto update the npm and Docker image versions at the same time. Example: https://github.com/liamdebeasi/framework-renovate-test/pull/40

Does this introduce a breaking change?

  • [ ] Yes
  • [x] No

Other information

liamdebeasi avatar Jan 26 '24 16:01 liamdebeasi