gutenberg Migrate E2E tests to Playwright

This issue tracks progress on migrating Gutenberg's E2E tests from Puppeteer (packages/e2e-tests) to Playwright (test/e2e).

Resources:

Migrating WordPress E2E tests to Playwright
test/e2e/README.md
test/e2e/MIGRATION.md
https://github.com/WordPress/gutenberg/pull/38570
Meeting notes: Discussion about Playwright as the e2e framework for WordPress
Proposal: Migrate e2e to Playwright!

Tests to migrate

Flaky tests

These tests are ranked by their estimated flaky rates. Experience developers should prioritize migrating them in this order.

[x] editor/blocks/image.test.js (https://github.com/WordPress/gutenberg/pull/40804)
[x] site-editor/template-part.test.js (#41048)
[x] editor/various/post-editor-template-mode.test.js (https://github.com/WordPress/gutenberg/pull/40175)
[x] site-editor/template-revert.test.js (#41310)
[x] site-editor/document-settings.test.js (#40258)
[x] editor/various/preview.test.js (#41380)
[ ] editor/blocks/navigation.test.js (@talldan will look at this)
[ ] editor/blocks/gallery.test.js
[ ] experiments/navigation-editor.test.js
[ ] editor/plugins/templates.test.js
[ ] editor/various/list-view.test.js
[ ] editor/plugins/iframed-multiple-block-stylesheets.test.js
[x] site-editor/style-variations.test.js (https://github.com/WordPress/gutenberg/pull/41427)
[ ] site-editor/settings-sidebar.test.js
[ ] editor/various/autosave.test.js
[x] experiments/blocks/comments-query.test.js (https://github.com/WordPress/gutenberg/pull/39826)
[ ] editor/various/writing-flow.test.js
[ ] editor/various/taxonomies.test.js
[ ] editor/various/block-grouping.test.js
[ ] editor/various/change-detection.test.js
[ ] editor/various/publish-button.test.js
[ ] editor/various/toolbar-roving-tabindex.test.js
[ ] widgets/editing-widgets.test.js
[x] editor/blocks/list.test.js (https://github.com/WordPress/gutenberg/pull/41555)
[ ] editor/various/embedding.test.js
[ ] editor/various/inserting-blocks.test.js
[ ] editor/various/block-editor-keyboard-shortcuts.test.js
[ ] editor/various/multi-block-selection.test.js
[ ] editor/plugins/custom-post-types.test.js
[ ] editor/various/block-deletion.test.js
[ ] editor/blocks/classic.test.js (https://github.com/WordPress/gutenberg/pull/42625)
[ ] editor/blocks/post-title.test.js (https://github.com/WordPress/gutenberg/pull/41107)
[x] editor/blocks/group.test.js (https://github.com/WordPress/gutenberg/pull/42801)
[ ] editor/blocks/site-title.test.js
[ ] editor/plugins/iframed-inline-styles.test.js
[x] editor/plugins/meta-boxes.test.js (https://github.com/WordPress/gutenberg/pull/41519)
[x] editor/various/copy-cut-paste-whole-blocks.test.js (https://github.com/WordPress/gutenberg/pull/39807)
[ ] editor/various/draggable-block.test.js
[ ] editor/various/keyboard-navigable-blocks.test.js
[ ] editor/various/links.test.js
[ ] editor/various/reusable-blocks.test.js
[ ] editor/various/rich-text.test.js (https://github.com/WordPress/gutenberg/pull/42628)
[ ] editor/various/undo.test.js
[ ] site-editor/multi-entity-saving.test.js
[ ] site-editor/site-editor-export.test.js
[x] widgets/customizing-widgets.test.js (https://github.com/WordPress/gutenberg/pull/39540)

Other tests

These tests don't have corresponding flaky reports. Newcomers can start migrating these since they are more stable and probably easier to start with.

[x] editor/blocks/buttons.test.js (https://github.com/WordPress/gutenberg/pull/41494)
[x] editor/blocks/code.test.js (https://github.com/WordPress/gutenberg/pull/41136)
[ ] editor/blocks/columns.test.js
[ ] editor/blocks/cover.test.js
[ ] editor/blocks/heading.test.js (https://github.com/WordPress/gutenberg/pull/41108)
[x] editor/blocks/html.test.js (https://github.com/WordPress/gutenberg/pull/41231)
[x] editor/blocks/missing.test.js (https://github.com/WordPress/gutenberg/pull/41680)
[x] editor/blocks/paragraph.test.js (https://github.com/WordPress/gutenberg/pull/40779)
[x] editor/blocks/preformatted.test.js (https://github.com/WordPress/gutenberg/pull/41128)
[ ] editor/blocks/query.test.js
[ ] editor/blocks/quote.test.js
[x] editor/blocks/separator.test.js (https://github.com/WordPress/gutenberg/pull/41130)
[x] editor/blocks/spacer.test.js (https://github.com/WordPress/gutenberg/pull/41590)
[x] editor/blocks/table.test.js (https://github.com/WordPress/gutenberg/pull/41945)
[ ] editor/plugins/align-hook.test.js
[ ] editor/plugins/allowed-blocks.test.js (https://github.com/WordPress/gutenberg/pull/39713)
[ ] editor/plugins/annotations.test.js
[x] editor/plugins/block-api.test.js (https://github.com/WordPress/gutenberg/pull/40217)
[ ] editor/plugins/block-context.test.js
[ ] editor/plugins/block-directory-add.test.js
[ ] editor/plugins/block-icons.test.js
[ ] editor/plugins/block-variations.test.js
[ ] editor/plugins/child-blocks.test.js
[ ] editor/plugins/container-blocks.test.js
[ ] editor/plugins/cpt-locking.test.js
[ ] editor/plugins/custom-taxonomies.test.js
[x] editor/plugins/deprecated-node-matcher.test.js (https://github.com/WordPress/gutenberg/pull/42759)
[x] editor/plugins/format-api.test.js (https://github.com/WordPress/gutenberg/pull/42035)
[x] editor/plugins/hooks-api.test.js (#42584)
[ ] editor/plugins/iframed-block.test.js
[ ] editor/plugins/iframed-masonry-block.test.js
[x] editor/plugins/image-size.test.js (https://github.com/WordPress/gutenberg/pull/40467)
[ ] editor/plugins/inner-blocks-allowed-blocks.test.js
[ ] editor/plugins/inner-blocks-render-appender.test.js
[ ] editor/plugins/meta-attribute-block.test.js
[ ] editor/plugins/nonce.test.js
[ ] editor/plugins/plugins-api.test.js
[ ] editor/plugins/register-block-type-hooks.test.js
[ ] editor/plugins/wp-editor-meta-box.test.js
[ ] editor/various/a11y.test.js (https://github.com/WordPress/gutenberg/pull/39638)
[ ] editor/various/adding-inline-tokens.test.js
[ ] editor/various/adding-patterns.test.js
[ ] editor/various/allowed-patterns.test.js
[ ] editor/various/block-hierarchy-navigation.test.js
[ ] editor/various/block-locking.test.js
[x] editor/various/block-mover.test.js (https://github.com/WordPress/gutenberg/pull/42039)
[ ] editor/various/block-switcher.test.js
[ ] editor/various/compatibility-classic-editor.test.js
[x] editor/various/convert-block-type.test.js (https://github.com/WordPress/gutenberg/pull/42760)
[ ] editor/various/core-settings.test.js
[ ] editor/various/datepicker.test.js
[ ] editor/various/dropdown-menu.test.js
[ ] editor/various/duplicating-blocks.test.js
[ ] editor/various/editor-modes.test.js
[ ] editor/various/font-size-picker.test.js
[ ] editor/various/format-library/text-color.test.js
[ ] editor/various/fullscreen-mode.test.js
[ ] editor/various/invalid-block.test.js
[ ] editor/various/is-typing.test.js
[ ] editor/various/keep-styles-on-block-transforms.test.js
[ ] editor/various/manage-reusable-blocks.test.js
[ ] editor/various/mentions.test.js
[ ] editor/various/navigable-toolbar.test.js
[ ] editor/various/new-post-default-content.test.js
[x] editor/various/new-post.test.js (https://github.com/WordPress/gutenberg/pull/39539)
[ ] editor/various/nux.test.js (https://github.com/WordPress/gutenberg/pull/39772)
[x] editor/various/popovers.test.js (https://github.com/WordPress/gutenberg/pull/39910)
[ ] editor/various/post-visibility.test.js
[ ] editor/various/preferences.test.js
[ ] editor/various/publish-panel.test.js
[ ] editor/various/publishing.test.js
[x] editor/various/rtl.test.js (https://github.com/WordPress/gutenberg/pull/41495)
[ ] editor/various/scheduling.test.js
[ ] editor/various/shortcut-help.test.js
[ ] editor/various/sidebar-permalink-panel.test.js
[ ] editor/various/sidebar.test.js
[ ] editor/various/splitting-merging.test.js
[x] editor/various/style-variation.test.js (https://github.com/WordPress/gutenberg/pull/40216)
[ ] editor/various/switch-to-draft.test.js
[ ] editor/various/typewriter.test.js
[ ] local/demo.test.js
[ ] performance/post-editor.test.js
[ ] performance/site-editor.test.js
[ ] site-editor/iframe-rendering-mode.test.js
[ ] site-editor/multi-entity-editing.test.js
[ ] site-editor/site-editor-inserter.test.js

Feb 16 '22 10:02 JustinyAhin

cc @kevin940726

Feb 16 '22 13:02 JustinyAhin

I reformatted this issue a little bit, added some links, and un-checked all the checkboxes since, to my knowledge, we haven't merged anything that migrates any tests yet. (https://github.com/WordPress/gutenberg/pull/38570 is just the infrastructure.)

Mar 22 '22 00:03 noisysocks

It would be nice to flag 🚩 which tests are flaky so that we can prioritise them. That will show us if Playwright offers any improvements there.

Mar 22 '22 02:03 noisysocks

I removed the e2e-utils checkboxes from the issue description, since they will be added if necessary when we are migrating tests.

Mar 22 '22 08:03 JustinyAhin

@JustinyAhin Can you please update the README.md and MIGRATION.md links in the description

https://github.com/WordPress/gutenberg/blob/trunk/test/e2e/MIGRATION.md https://github.com/WordPress/gutenberg/blob/trunk/test/e2e/README.md

Mar 23 '22 02:03 juhi123

@juhi123 Updated! Thanks!

Mar 23 '22 06:03 kevin940726

I started looking at the second most flakey test (site-editor/template-part.test.js).

I hit a roadblock that I think is worth trying to solve early on while there's a chance to put the right structure in place.

The site editor is using an iframe for the editor canvas. Because of this, none of the existing test utils for interacting with the editor canvas can be used for site editor tests. That's because they all use page.locator and the like, while for an iframe frameLocator needs to be used to find the iframe first, and then frame.locator can be used to select elements inside the iframe.

It's clear this was also a problem in Puppeteer, most of the puppeteer site editor tests have duplicate utility functions in the test file.

I'm not all that familiar with playwright, but I wonder if it could be an option to move the canvas utils into a class and do something like the following:

test.use( {
	editorCanvasUtils: async (
		{ page, pageUtils },
		use
	) => {
		await use(
			new EditorCanvasUtils( { page, pageUtils, isSiteEditor: true } )
		);
	},
} );

EditorCanvasUtils could have a private canvas property that uses either the iframe or the page depending on the isSiteEditor param. All utilities defined in the class then use canvas.locator.

Apr 19 '22 04:04 talldan

Hi @kevin940726 @JustinyAhin 👋

I have migrated image-size.test.js to Playwright, can you please review it and share your feedbacks https://github.com/WordPress/gutenberg/pull/40467

Apr 20 '22 11:04 juhi123

Hi @JustinyAhin 👋 I was wondering if we are still maintaining this list as it seems editor/blocks/quote.test.js this test is already migrated but not checked in the list.

Aug 05 '22 04:08 juhi123

Hello @juhi123.

Yes, this list is still our reference for tracking tests that are being migrated. I haven't done contributions to tests in a while because I switched jobs recently, and I'm still adapting to my new contribution schedules.

In the meantime, if you come across a test that is already migrated but not check, can you update it on the list? :)

Aug 08 '22 07:08 JustinyAhin

I don't think Quote tests have been migrated yet, the old puppeteer tests are still present, and there are quite a lot of test cases that haven't been ported - https://github.com/WordPress/gutenberg/blob/trunk/packages/e2e-tests/specs/editor/blocks/quote.test.js.

Aug 08 '22 07:08 talldan

Hey @JustinyAhin No problem, congratulations!

I can update but I don't think I have enough access to check the list as I am not able to click on that.

@talldan Yeah that's why I also got confused as it's still there, but I can see this in Playwright tests also https://github.com/WordPress/gutenberg/blob/trunk/test/e2e/specs/editor/blocks/quote.spec.js

Aug 09 '22 05:08 juhi123

@juhi123 I don't think that's a migration, just someone decided to write an individual test in playwright. The other test cases still need to be migrated.

Aug 09 '22 05:08 talldan

I started working on the editor/various/writing-flow.test.js migration.

Aug 25 '22 13:08 Mamaduka

I've updated the resource links to include new E2E tests handbook pages.

I would recommend it to anyone working or who already has a migration PR(s) to get familiar with the recommendations.

New resource links:

Apr 26 '23 15:04 Mamaduka

@Mamaduka @JustinyAhin I wanted to highlight here that the explanations page for Automated Testing is now outdated as mentioned here: https://github.com/WordPress/gutenberg/issues/48757

I am in favor of either updating the page or simply removing it if you all feel like the e2e resources should be simplified and kept in the Testing Overview section of the handbook. If removing the explanation doc is the desired outcome, let me know, and I can take care of that PR.

May 25 '23 13:05 ndiego

Every end-to-end test has been migrated to the Playwright. Thank you everyone for contributing!

Feb 23 '24 11:02 Mamaduka

Thank you all! ❤️ 💯

There is still some work left to do to clean up. We'll do them in follow-up PRs!

Feb 23 '24 11:02 kevin940726

gutenberg gutenberg copied to clipboard

Migrate E2E tests to Playwright

Tests to migrate

Flaky tests

Other tests

gutenberg
gutenberg copied to clipboard