Run e2e tests on Firefox

[swissspidy]

Context

We currently only run e2e tests on Chrome.

Summary

This PR adds experimental e2e testing on Firefox.

Relevant Technical Choices

No more request interceptions

Firefox does not currently support this with Puppeteer, so I replaced all usage of request interceptions with little e2e test plugins.

Console warnings

Firefox warns about lots of things in the devtools console. The PR adds tons of messages to the allowlist so that they're properly ignored.

Cross-origin isolation / navigation timeouts

Opening the editor in Firefox causes navigation timeouts in Puppeteer. Does not happen when disabling cross-origin isolation (video optimization)

Uploads not supported

Firefox does not support file uploads yet with Puppeteer. See https://bugzilla.mozilla.org/show_bug.cgi?id=1553847

Window size

Seems like --window-size doesn't work, but passing --width/--height does.

Skipped tests

Some tests don't yet work reliably on Firefox for one reason or another.

These are currently being skipped on Firefox, with the reason explained in an inline comment.

To-do

• Skip less tests on Firefox in the future

User-facing changes

N/A

Testing Instructions

• [x] This is a non-user-facing change and requires no QA

This PR can be tested by following these steps:

Reviews

Does this PR have a security-related impact?

No

Does this PR change what data or activity we track or use?

NO

Does this PR have a legal-related impact?

No

Checklist

• [ ] This PR addresses an existing issue and I have linked this PR to it in ZenHub
• [x] I have tested this code to the best of my abilities
• [x] I have verified accessibility to the best of my abilities (docs)
• [x] I have verified i18n and l10n (translation, right-to-left layout) to the best of my abilities
• [x] This PR contains automated tests (unit, integration, and/or e2e) to verify the code works as intended (docs)
• [x] I have added documentation where necessary
• [x] I have added a matching Type: XYZ label to the PR

---

Fixes #

[github-actions[bot]]

Size Change: +29 B (0%)

Total Size: 4.03 MB

ℹ️ View Unchanged

Filename	Size	Change
assets/css/carousel-view-rtl.css	701 B	0 B
assets/css/carousel-view.css	701 B	0 B
assets/css/vendors-chunk-getStoryPropsToSave-wp-story-editor-rtl.css	701 B	0 B
assets/css/vendors-chunk-getStoryPropsToSave-wp-story-editor.css	701 B	0 B
assets/css/web-stories-block-rtl.css	4.51 kB	0 B
assets/css/web-stories-block.css	4.55 kB	0 B
assets/css/web-stories-embed-rtl.css	288 B	0 B
assets/css/web-stories-embed.css	288 B	0 B
assets/css/web-stories-list-styles-rtl.css	2.38 kB	0 B
assets/css/web-stories-list-styles.css	2.4 kB	0 B
assets/css/web-stories-theme-style-twentyeleven-rtl.css	102 B	0 B
assets/css/web-stories-theme-style-twentyeleven.css	102 B	0 B
assets/css/web-stories-theme-style-twentyfifteen-rtl.css	251 B	0 B
assets/css/web-stories-theme-style-twentyfifteen.css	251 B	0 B
assets/css/web-stories-theme-style-twentyfourteen-rtl.css	287 B	0 B
assets/css/web-stories-theme-style-twentyfourteen.css	287 B	0 B
assets/css/web-stories-theme-style-twentyseventeen-rtl.css	288 B	0 B
assets/css/web-stories-theme-style-twentyseventeen.css	288 B	0 B
assets/css/web-stories-theme-style-twentysixteen-rtl.css	224 B	0 B
assets/css/web-stories-theme-style-twentysixteen.css	224 B	0 B
assets/css/web-stories-theme-style-twentyten-rtl.css	143 B	0 B
assets/css/web-stories-theme-style-twentyten.css	143 B	0 B
assets/css/web-stories-theme-style-twentytwelve-rtl.css	256 B	0 B
assets/css/web-stories-theme-style-twentytwelve.css	256 B	0 B
assets/css/web-stories-theme-style-twentytwenty-rtl.css	86 B	0 B
assets/css/web-stories-theme-style-twentytwenty.css	86 B	0 B
assets/css/web-stories-theme-style-twentytwentyone-rtl.css	325 B	0 B
assets/css/web-stories-theme-style-twentytwentyone.css	326 B	0 B
assets/css/web-stories-widget-rtl.css	485 B	0 B
assets/css/web-stories-widget.css	485 B	0 B
assets/css/wp-dashboard-rtl.css	658 B	0 B
assets/css/wp-dashboard.css	659 B	0 B
assets/css/wp-story-editor-rtl.css	608 B	0 B
assets/css/wp-story-editor.css	609 B	0 B
assets/js/carousel-view.js	3.72 kB	0 B
assets/js/chunk-colorthief.js	2.61 kB	0 B
assets/js/chunk-focus-visible.js	999 B	0 B
assets/js/chunk-fonts.js	54.2 kB	0 B
assets/js/chunk-getStoryPropsToSave.js	1.32 MB	-9 B (0%)
assets/js/chunk-web-stories-template-0.js	557 B	0 B
assets/js/chunk-web-stories-template-100.js	481 B	0 B
assets/js/chunk-web-stories-template-102.js	9.12 kB	0 B
assets/js/chunk-web-stories-template-104.js	536 B	0 B
assets/js/chunk-web-stories-template-108.js	509 B	0 B
assets/js/chunk-web-stories-template-110.js	6.96 kB	0 B
assets/js/chunk-web-stories-template-112.js	595 B	0 B
assets/js/chunk-web-stories-template-116.js	566 B	0 B
assets/js/chunk-web-stories-template-118.js	7.03 kB	0 B
assets/js/chunk-web-stories-template-12.js	523 B	0 B
assets/js/chunk-web-stories-template-120.js	556 B	0 B
assets/js/chunk-web-stories-template-124.js	528 B	0 B
assets/js/chunk-web-stories-template-126.js	7.69 kB	0 B
assets/js/chunk-web-stories-template-128.js	602 B	0 B
assets/js/chunk-web-stories-template-132.js	572 B	0 B
assets/js/chunk-web-stories-template-134.js	9.83 kB	0 B
assets/js/chunk-web-stories-template-136.js	554 B	0 B
assets/js/chunk-web-stories-template-14.js	8.55 kB	0 B
assets/js/chunk-web-stories-template-140.js	527 B	0 B
assets/js/chunk-web-stories-template-142.js	7.95 kB	0 B
assets/js/chunk-web-stories-template-144.js	597 B	0 B
assets/js/chunk-web-stories-template-148.js	569 B	0 B
assets/js/chunk-web-stories-template-150.js	8.77 kB	0 B
assets/js/chunk-web-stories-template-152.js	512 B	0 B
assets/js/chunk-web-stories-template-156.js	484 B	0 B
assets/js/chunk-web-stories-template-158.js	9.6 kB	0 B
assets/js/chunk-web-stories-template-16.js	600 B	0 B
assets/js/chunk-web-stories-template-160.js	563 B	0 B
assets/js/chunk-web-stories-template-164.js	535 B	0 B
assets/js/chunk-web-stories-template-166.js	8.19 kB	0 B
assets/js/chunk-web-stories-template-168.js	547 B	0 B
assets/js/chunk-web-stories-template-172.js	520 B	0 B
assets/js/chunk-web-stories-template-174.js	8.49 kB	0 B
assets/js/chunk-web-stories-template-176.js	536 B	0 B
assets/js/chunk-web-stories-template-180.js	508 B	0 B
assets/js/chunk-web-stories-template-182.js	6.91 kB	0 B
assets/js/chunk-web-stories-template-184.js	615 B	0 B
assets/js/chunk-web-stories-template-188.js	587 B	0 B
assets/js/chunk-web-stories-template-190.js	6.59 kB	0 B
assets/js/chunk-web-stories-template-192.js	531 B	0 B
assets/js/chunk-web-stories-template-196.js	501 B	0 B
assets/js/chunk-web-stories-template-198.js	10.4 kB	0 B
assets/js/chunk-web-stories-template-20.js	573 B	0 B
assets/js/chunk-web-stories-template-200.js	556 B	0 B
assets/js/chunk-web-stories-template-204.js	527 B	0 B
assets/js/chunk-web-stories-template-206.js	6.7 kB	0 B
assets/js/chunk-web-stories-template-208.js	612 B	0 B
assets/js/chunk-web-stories-template-212.js	584 B	0 B
assets/js/chunk-web-stories-template-214.js	6.29 kB	0 B
assets/js/chunk-web-stories-template-216.js	556 B	0 B
assets/js/chunk-web-stories-template-22.js	8.65 kB	0 B
assets/js/chunk-web-stories-template-220.js	528 B	0 B
assets/js/chunk-web-stories-template-222.js	7.14 kB	0 B
assets/js/chunk-web-stories-template-224.js	545 B	0 B
assets/js/chunk-web-stories-template-228.js	518 B	0 B
assets/js/chunk-web-stories-template-230.js	8.04 kB	0 B
assets/js/chunk-web-stories-template-232.js	575 B	0 B
assets/js/chunk-web-stories-template-236.js	547 B	0 B
assets/js/chunk-web-stories-template-238.js	8.07 kB	0 B
assets/js/chunk-web-stories-template-24.js	552 B	0 B
assets/js/chunk-web-stories-template-240.js	589 B	0 B
assets/js/chunk-web-stories-template-244.js	561 B	0 B
assets/js/chunk-web-stories-template-246.js	7.35 kB	0 B
assets/js/chunk-web-stories-template-248.js	512 B	0 B
assets/js/chunk-web-stories-template-252.js	485 B	0 B
assets/js/chunk-web-stories-template-254.js	9.14 kB	0 B
assets/js/chunk-web-stories-template-256.js	564 B	0 B
assets/js/chunk-web-stories-template-260.js	536 B	0 B
assets/js/chunk-web-stories-template-262.js	11.9 kB	0 B
assets/js/chunk-web-stories-template-264.js	506 B	0 B
assets/js/chunk-web-stories-template-268.js	479 B	0 B
assets/js/chunk-web-stories-template-270.js	8.51 kB	0 B
assets/js/chunk-web-stories-template-272.js	586 B	0 B
assets/js/chunk-web-stories-template-276.js	557 B	0 B
assets/js/chunk-web-stories-template-278.js	7.19 kB	0 B
assets/js/chunk-web-stories-template-28.js	524 B	0 B
assets/js/chunk-web-stories-template-280.js	575 B	0 B
assets/js/chunk-web-stories-template-284.js	547 B	0 B
assets/js/chunk-web-stories-template-286.js	8.44 kB	0 B
assets/js/chunk-web-stories-template-288.js	587 B	0 B
assets/js/chunk-web-stories-template-292.js	559 B	0 B
assets/js/chunk-web-stories-template-294.js	11 kB	0 B
assets/js/chunk-web-stories-template-296.js	538 B	0 B
assets/js/chunk-web-stories-template-30.js	7.79 kB	0 B
assets/js/chunk-web-stories-template-300.js	512 B	0 B
assets/js/chunk-web-stories-template-302.js	5.93 kB	0 B
assets/js/chunk-web-stories-template-304.js	583 B	0 B
assets/js/chunk-web-stories-template-308.js	556 B	0 B
assets/js/chunk-web-stories-template-310.js	7.48 kB	0 B
assets/js/chunk-web-stories-template-312.js	601 B	0 B
assets/js/chunk-web-stories-template-316.js	574 B	0 B
assets/js/chunk-web-stories-template-318.js	7.1 kB	0 B
assets/js/chunk-web-stories-template-32.js	576 B	0 B
assets/js/chunk-web-stories-template-320.js	570 B	0 B
assets/js/chunk-web-stories-template-324.js	542 B	0 B
assets/js/chunk-web-stories-template-326.js	8.66 kB	0 B
assets/js/chunk-web-stories-template-328.js	582 B	0 B
assets/js/chunk-web-stories-template-332.js	557 B	0 B
assets/js/chunk-web-stories-template-334.js	7.33 kB	0 B
assets/js/chunk-web-stories-template-336.js	535 B	0 B
assets/js/chunk-web-stories-template-340.js	506 B	0 B
assets/js/chunk-web-stories-template-342.js	6.57 kB	0 B
assets/js/chunk-web-stories-template-344.js	571 B	0 B
assets/js/chunk-web-stories-template-348.js	544 B	0 B
assets/js/chunk-web-stories-template-350.js	7.95 kB	0 B
assets/js/chunk-web-stories-template-352.js	596 B	0 B
assets/js/chunk-web-stories-template-356.js	568 B	0 B
assets/js/chunk-web-stories-template-358.js	9.76 kB	0 B
assets/js/chunk-web-stories-template-36.js	548 B	0 B
assets/js/chunk-web-stories-template-360.js	577 B	0 B
assets/js/chunk-web-stories-template-364.js	548 B	0 B
assets/js/chunk-web-stories-template-366.js	6.83 kB	0 B
assets/js/chunk-web-stories-template-368.js	542 B	0 B
assets/js/chunk-web-stories-template-372.js	515 B	0 B
assets/js/chunk-web-stories-template-374.js	4.56 kB	0 B
assets/js/chunk-web-stories-template-376.js	604 B	0 B
assets/js/chunk-web-stories-template-38.js	11 kB	0 B
assets/js/chunk-web-stories-template-380.js	576 B	0 B
assets/js/chunk-web-stories-template-382.js	7.9 kB	0 B
assets/js/chunk-web-stories-template-384.js	568 B	0 B
assets/js/chunk-web-stories-template-388.js	541 B	0 B
assets/js/chunk-web-stories-template-390.js	7.76 kB	0 B
assets/js/chunk-web-stories-template-392.js	527 B	0 B
assets/js/chunk-web-stories-template-396.js	500 B	0 B
assets/js/chunk-web-stories-template-398.js	9.37 kB	0 B
assets/js/chunk-web-stories-template-4.js	531 B	0 B
assets/js/chunk-web-stories-template-40.js	570 B	0 B
assets/js/chunk-web-stories-template-400.js	513 B	0 B
assets/js/chunk-web-stories-template-404.js	485 B	0 B
assets/js/chunk-web-stories-template-406.js	8.01 kB	0 B
assets/js/chunk-web-stories-template-408.js	541 B	0 B
assets/js/chunk-web-stories-template-412.js	513 B	0 B
assets/js/chunk-web-stories-template-414.js	9.54 kB	0 B
assets/js/chunk-web-stories-template-416.js	616 B	0 B
assets/js/chunk-web-stories-template-420.js	590 B	0 B
assets/js/chunk-web-stories-template-422.js	9.57 kB	0 B
assets/js/chunk-web-stories-template-424.js	564 B	0 B
assets/js/chunk-web-stories-template-428.js	535 B	0 B
assets/js/chunk-web-stories-template-430.js	5.35 kB	0 B
assets/js/chunk-web-stories-template-432.js	560 B	0 B
assets/js/chunk-web-stories-template-436.js	532 B	0 B
assets/js/chunk-web-stories-template-438.js	7.11 kB	0 B
assets/js/chunk-web-stories-template-44.js	541 B	0 B
assets/js/chunk-web-stories-template-440.js	588 B	0 B
assets/js/chunk-web-stories-template-444.js	558 B	0 B
assets/js/chunk-web-stories-template-446.js	6.22 kB	0 B
assets/js/chunk-web-stories-template-448.js	554 B	0 B
assets/js/chunk-web-stories-template-452.js	525 B	0 B
assets/js/chunk-web-stories-template-454.js	9.11 kB	0 B
assets/js/chunk-web-stories-template-456.js	541 B	0 B
assets/js/chunk-web-stories-template-46.js	8.9 kB	0 B
assets/js/chunk-web-stories-template-460.js	513 B	0 B
assets/js/chunk-web-stories-template-462.js	13.6 kB	0 B
assets/js/chunk-web-stories-template-464.js	566 B	0 B
assets/js/chunk-web-stories-template-468.js	539 B	0 B
assets/js/chunk-web-stories-template-470.js	5.24 kB	0 B
assets/js/chunk-web-stories-template-472.js	603 B	0 B
assets/js/chunk-web-stories-template-476.js	575 B	0 B
assets/js/chunk-web-stories-template-478.js	8.23 kB	0 B
assets/js/chunk-web-stories-template-48.js	582 B	0 B
assets/js/chunk-web-stories-template-480.js	522 B	0 B
assets/js/chunk-web-stories-template-484.js	494 B	0 B
assets/js/chunk-web-stories-template-486.js	8.51 kB	0 B
assets/js/chunk-web-stories-template-52.js	553 B	0 B
assets/js/chunk-web-stories-template-54.js	6.57 kB	0 B
assets/js/chunk-web-stories-template-56.js	581 B	0 B
assets/js/chunk-web-stories-template-6.js	10.3 kB	0 B
assets/js/chunk-web-stories-template-60.js	552 B	0 B
assets/js/chunk-web-stories-template-62.js	6.68 kB	0 B
assets/js/chunk-web-stories-template-64.js	581 B	0 B
assets/js/chunk-web-stories-template-68.js	552 B	0 B
assets/js/chunk-web-stories-template-70.js	7.92 kB	0 B
assets/js/chunk-web-stories-template-72.js	596 B	0 B
assets/js/chunk-web-stories-template-76.js	566 B	0 B
assets/js/chunk-web-stories-template-78.js	8 kB	0 B
assets/js/chunk-web-stories-template-8.js	551 B	0 B
assets/js/chunk-web-stories-template-80.js	545 B	0 B
assets/js/chunk-web-stories-template-84.js	518 B	0 B
assets/js/chunk-web-stories-template-86.js	6.52 kB	0 B
assets/js/chunk-web-stories-template-88.js	549 B	0 B
assets/js/chunk-web-stories-template-92.js	522 B	0 B
assets/js/chunk-web-stories-template-94.js	7.98 kB	0 B
assets/js/chunk-web-stories-template-96.js	510 B	0 B
assets/js/chunk-web-stories-textset-0.js	5.27 kB	0 B
assets/js/chunk-web-stories-textset-1.js	6.76 kB	0 B
assets/js/chunk-web-stories-textset-2.js	7.88 kB	0 B
assets/js/chunk-web-stories-textset-3.js	15.4 kB	0 B
assets/js/chunk-web-stories-textset-4.js	4.38 kB	0 B
assets/js/chunk-web-stories-textset-5.js	5.68 kB	0 B
assets/js/chunk-web-stories-textset-6.js	5.49 kB	0 B
assets/js/chunk-web-stories-textset-7.js	10.4 kB	0 B
assets/js/imgareaselect.js	4.14 kB	0 B
assets/js/lightbox.js	989 B	0 B
assets/js/tinymce-button.js	3.49 kB	0 B
assets/js/vendors-chunk-ffmpeg.js	5.61 kB	0 B
assets/js/vendors-chunk-getStoryPropsToSave-chunk-resize-observer-polyfill-wp-story-editor.js	2.55 kB	0 B
assets/js/vendors-chunk-getStoryPropsToSave-wp-story-editor.js	145 kB	0 B
assets/js/vendors-chunk-html-to-image.js	4.59 kB	0 B
assets/js/vendors-chunk-react-calendar.js	11.9 kB	0 B
assets/js/vendors-chunk-react-color.js	42.9 kB	0 B
assets/js/vendors-chunk-web-animations-js.js	14.6 kB	0 B
assets/js/vendors-wp-dashboard.js	95.6 kB	0 B
assets/js/web-stories-activation-notice.js	23.4 kB	0 B
assets/js/web-stories-block.js	19.4 kB	0 B
assets/js/web-stories-embed.js	493 B	0 B
assets/js/web-stories-widget.js	984 B	0 B
assets/js/wp-dashboard.js	121 kB	+1 B (0%)
assets/js/wp-story-editor.js	1.5 MB	+37 B (0%)

_{compressed-size-action}

[codecov[bot]]

Codecov Report

Merging #7748 (1fd04e8) into main (9bbede8) will increase coverage by 5.28%. The diff coverage is 0.00%.

:exclamation: Current head 1fd04e8 differs from pull request most recent head 08ac3f6. Consider uploading reports for the commit 08ac3f6 to get more accurate results

@@            Coverage Diff             @@
##             main    #7748      +/-   ##
==========================================
+ Coverage   61.57%   66.86%   +5.28%     
==========================================
  Files         325     1524    +1199     
  Lines        4539    29955   +25416     
==========================================
+ Hits         2795    20028   +17233     
- Misses       1744     9927    +8183     

Flag	Coverage Δ
karmatests	76.93% <ø> (+15.35%)	:arrow_up:
unittests	∅ <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ackages/e2e-tests/src/plugins/locked-post-mock.php	0.00% <ø> (ø)
...e2e-tests/src/plugins/status-check-200-invalid.php	0.00% <0.00%> (ø)
...ackages/e2e-tests/src/plugins/status-check-403.php	0.00% <0.00%> (ø)
...ackages/e2e-tests/src/plugins/status-check-500.php	0.00% <0.00%> (ø)
...story-editor/src/components/checklist/checklist.js	100.00% <ø> (ø)
packages/dashboard/src/app/api/apiProvider.js	15.38% <0.00%> (-2.80%)	:arrow_down:
packages/dashboard/src/app/api/useTemplateApi.js	4.54% <0.00%> (-0.46%)	:arrow_down:
packages/dashboard/src/app/api/useUserApi.js	0.00% <0.00%> (ø)
packages/dashboard/src/app/api/useMediaApi.js	0.00% <0.00%> (ø)
packages/dashboard/src/app/api/usePagesApi.js	0.00% <0.00%> (ø)
... and 1259 more

[swissspidy]

@spacedmonkey @choumx This is something I've been experimenting with over the last few months. Not 100% sold on the benefits of running tests on Firefox — especially considering its limitations when used with Puppeteer - but want to get some eyes on it nonetheless. So this may or may not land at all.

[spacedmonkey]

The value in running in Firefox, is running in firefox normally finds errors in the code. Firefox has a different render engine and handles JS differently. So running our code against, has value IMO.

[swissspidy]

The value in running in Firefox, is running in firefox normally finds errors in the code. Firefox has a different render engine and handles JS differently. So running our code against, has value IMO.

What I am mostly concerned about at the moment is the amount of work required to make tests work on Firefox, because Firefox on Puppeteer is so limited.

[dreamofabear]

Re: Puppeteer vs. Webdriver, I'd also keep an eye on test execution speed and reliability/flakiness. E.g. it's possible that the additional coverage is not worth the CI brittleness introduced, especially since current tests are written for Puppeteer.

[swissspidy]

Re: Puppeteer vs. Webdriver, I'd also keep an eye on test execution speed and reliability/flakiness. E.g. it's possible that the additional coverage is not worth the CI brittleness introduced, especially since current tests are written for Puppeteer.

This is in no relation to Webdriver, but yeah reliability is my worry here as well.

I suppose we can try it for a while and see how well it works.

Speed is about the same as for Chrome, perhaps a couple of minutes slower (15 vs 12 minutes in the last run)

[swissspidy]

All tests seem to pass here now 🎉 Albeit the Firefox ones seem to be a bit flakey (and thus marked as experimental). I'd love to merge this for now so we can keep an eye on it to see how well the tests work on Firefox.

@choumx @spacedmonkey If you could review this PR at some point, that'd be great. Not urgent though.

[swissspidy]

Looks like there is one bug that will be resolved in FF 95: https://bugzilla.mozilla.org/show_bug.cgi?id=1735131