Summary

This issue is to track the migration of out vrt runner to playwright.

Tasks

[x] Refactor CI to accommodate running playwright on GH actions with deployed instance
[x] Copy old tests to future home under the new playwright system (#1573)
[x] Migrate all supporting code from puppeteer api to playwright
[x] Migrate testing code from puppeteer api to playwright
[x] Setup local playwright runner under same scripts as today (e.g. test:integration[:*] to test:e2e[:*])
[x] Refactor page.emulateTimezone usage with playwright api
[x] Fix preloading custom fonts
[x] Update vrt screenshots locally using new playwright workflow
[x] Test CI deploy action prevented from external users' prs.
[x] Upload test failure artifacts to buildkite
[x] Merge playwright html reports and deploy
[x] Migrate CI to buildkite

[ ] Create fallback flow if deployment fails to load archive e2e-server files. Could be possible by downloading e2e build artifacts.
[ ] Create scheduled CI cleanup tasks to tear down environments and firebase channels.
[ ] Improve separation of concerns between e2e-server and e2e test runner
[ ] Wait for better ts support from playwright. Currently the interdependence on charts and other external directories is not nicely supported. The only way to achieve this is duplicate imports or transpile ts into a separate directory (see this) then run tests from there. This is not nice because it messes up all the test output directories.
[ ] Improve e2e-server script behavior to be more reactive to changes. In alternative we can also replace it completely and use directly the component testing in playwright
[ ] Could improve testing errors and simplicity using fixtures and custom expect matchers for charts. With assertions being made in common page object, the errors all point to the common file expect instead of the actual failed test suite.

Automated deployment of all pr and master.
Protected flow for 3rd party contributors for deployments. Manual approval required.
All deployment progress is shown in the pr and indicates when ready to view in the UI. Using GH's deployments api.
More isolated github workflow files
Uses reusable workflow for large test jobs like the pr checks and playwright tests.
Less noise on pr status checks using repository_dispatch events and splitting up prs and branch ci.
Both storybook and the e2e_server (aka Marco's fast vrt app) are deployed.
Firebase allows channel deployments to have an expiration, to be deleted automatically.
All e2e playwright tests run in parallel, could make this possible for jest tests too.
Local environment functions similar to before just a little more streamlined.
VRT screenshots are now structured in folders according to their test file and/or an array pattern matching (i.e. for all.test.ts by storybook folders).

The logic is a little verbose but enables running everything in GH actions without a 3rd party deployment provider.
The npm install process take a little longer on the deploy jobs because the caching action does not work with the event type. Could use https://docs.github.com/en/actions/using-workflows/advanced-workflow-features#caching-dependencies
A way to store diff content created in the ci (i.e. __image_diffs__) when failures occur. GCP is a good option but need to set this up moving away from jenkins.
Need to look into optimizing generate vrt logic
Playwright does not remove stale screenshots 😞

Feb 01 '22 15:02 nickofthyme

@nickofthyme could you please update the various lists? I think we can mark some items as done already

Jul 12 '22 08:07 markov00

Closed by https://github.com/elastic/elastic-charts/pull/1572 & https://github.com/elastic/elastic-charts/pull/1733

Jul 10 '23 15:07 nickofthyme