Try running audio/video tests on more browsers and platforms

[stephenwade]

I'm trying to debug a Playwright issue with AudioContext on Windows runners (will file a separate issue if I can't figure it out), and I'm interested to see if Playwright's audio/video tests now work on all platforms and browsers.

This PR removes some it.fixme annotations from the audio/video tests in capabilities.spec.ts. All the it.fixme lines I removed were added more than a year ago, and all the linked issues look like they are resolved.

Related: https://github.com/microsoft/playwright/pull/30410

[stephenwade]

@microsoft-github-policy-service agree

[github-actions[bot]]

Test results for "tests 1"

2 flaky

:warning: [firefox-page] › page/page-click.spec.ts:848:3 › should not hang when frame is detached
:warning: [chromium-page] › page/page-event-request.spec.ts:139:3 › should report navigation requests and responses handled by service worker with routing

27392 passed, 668 skipped :heavy_check_mark::heavy_check_mark::heavy_check_mark:

Merge workflow run.

[stephenwade]

cc @mxschmitt I think this needs a tag to run the affected tests.

[github-actions[bot]]

Test results for "tests 1"

2 flaky

:warning: [playwright-test] › ui-mode-test-screencast.spec.ts:21:5 › should show screenshots
:warning: [playwright-test] › ui-mode-test-screencast.spec.ts:21:5 › should show screenshots

27407 passed, 671 skipped :heavy_check_mark::heavy_check_mark::heavy_check_mark:

Merge workflow run.

[mxschmitt]

Done!

[github-actions[bot]]

Test results for "tests 2"

1 fatal errors, not part of any test 6 failed :x: [firefox-library] › library/inspector/cli-codegen-2.spec.ts:340:7 › cli codegen › should record open in a new tab with url :x: [firefox-library] › library/inspector/cli-codegen-2.spec.ts:340:7 › cli codegen › should record open in a new tab with url :x: [webkit-library] › library/capabilities.spec.ts:129:3 › should not crash on page with mp4 @smoke :x: [webkit-library] › library/emulation-focus.spec.ts:203:12 › should trigger hover state concurrently :x: [webkit-library] › library/signals.spec.ts:62:7 › signals › should report browser close signal 2 :x: [webkit-library] › library/capabilities.spec.ts:129:3 › should not crash on page with mp4 @smoke

72 flaky

:warning: [chromium-page] › page/page-event-request.spec.ts:139:3 › should report navigation requests and responses handled by service worker with routing
:warning: [chromium-page] › page/page-event-request.spec.ts:110:3 › should report navigation requests and responses handled by service worker
:warning: [chromium-page] › page/page-set-input-files.spec.ts:57:3 › should upload large file
:warning: [chromium-library] › library/inspector/cli-codegen-1.spec.ts:625:7 › cli codegen › should await popup
:warning: [chromium-page] › page/page-set-input-files.spec.ts:57:3 › should upload large file
:warning: [chromium-page] › page/page-drag.spec.ts:272:5 › Drag and drop › should work if a frame is stalled
:warning: [chromium-page] › page/page-drag.spec.ts:330:5 › Drag and drop › should work with locators
:warning: [chromium-library] › library/inspector/cli-codegen-1.spec.ts:22:7 › cli codegen › should click
:warning: [chromium-library] › library/inspector/cli-codegen-1.spec.ts:291:7 › cli codegen › should fill textarea
:warning: [chromium-page] › page/page-drag.spec.ts:53:5 › Drag and drop › should not send dragover on the first mousemove
:warning: [chromium-page] › page/page-drag.spec.ts:272:5 › Drag and drop › should work if a frame is stalled
:warning: [chromium-page] › page/page-mouse.spec.ts:166:3 › should trigger hover state
:warning: [chromium-page] › page/page-mouse.spec.ts:186:3 › should trigger hover state on disabled button
:warning: [chromium-library] › library/selector-generator.spec.ts:425:5 › selector generator › should ignore empty role for candidate consideration
:warning: [chromium-library] › library/trace-viewer.spec.ts:893:1 › should display waitForLoadState even if did not wait for it
:warning: [firefox-page] › page/page-click-timeout-4.spec.ts:20:3 › should timeout waiting for stable position
:warning: [firefox-page] › page/elementhandle-wait-for-element-state.spec.ts:117:3 › should wait for stable position
:warning: [firefox-library] › library/video.spec.ts:730:5 › screencast › should capture full viewport on hidpi
:warning: [firefox-page] › page/page-click-timeout-4.spec.ts:20:3 › should timeout waiting for stable position
:warning: [firefox-page] › page/locator-misc-1.spec.ts:28:3 › should hover when Node is removed
:warning: [firefox-page] › page/page-add-locator-handler.spec.ts:88:5 › should work with locator.hover()
:warning: [firefox-library] › library/browsercontext-pages.spec.ts:58:3 › should not hang with touch-enabled viewports
:warning: [firefox-library] › library/browsertype-connect.spec.ts:932:3 › launchServer only › should be able to reconnect to a browser 12 times without warnings
:warning: [firefox-library] › library/inspector/cli-codegen-1.spec.ts:55:7 › cli codegen › should ignore programmatic events
:warning: [firefox-library] › library/inspector/cli-codegen-1.spec.ts:75:7 › cli codegen › should click after same-document navigation
:warning: [firefox-library] › library/inspector/cli-codegen-1.spec.ts:105:7 › cli codegen › should make a positioned click on a canvas
:warning: [firefox-library] › library/inspector/cli-codegen-1.spec.ts:158:7 › cli codegen › should work with TrustedTypes
:warning: [firefox-library] › library/inspector/cli-codegen-1.spec.ts:196:7 › cli codegen › should not target selector preview by text regexp
:warning: [firefox-library] › library/inspector/cli-codegen-2.spec.ts:138:7 › cli codegen › should upload multiple files
:warning: [firefox-library] › library/inspector/cli-codegen-2.spec.ts:169:7 › cli codegen › should clear files
:warning: [firefox-library] › library/inspector/cli-codegen-2.spec.ts:493:7 › cli codegen › should --save-trace
:warning: [firefox-library] › library/inspector/cli-codegen-3.spec.ts:92:7 › cli codegen › should generate frame locators
:warning: [firefox-library] › library/page-event-crash.spec.ts:73:1 › should be able to close context when page crashes
:warning: [firefox-library] › library/tracing.spec.ts:292:5 › should overwrite existing file
:warning: [firefox-page] › page/elementhandle-scroll-into-view.spec.ts:103:3 › should wait for element to stop moving
:warning: [firefox-page] › page/elementhandle-wait-for-element-state.spec.ts:117:3 › should wait for stable position
:warning: [firefox-page] › page/page-click-timeout-4.spec.ts:20:3 › should timeout waiting for stable position
:warning: [firefox-page] › page/page-click.spec.ts:99:3 › should click the 1x1 div
:warning: [firefox-page] › page/page-click.spec.ts:445:3 › should wait for stable position
:warning: [firefox-library] › library/browsertype-connect.spec.ts:815:7 › run-server › socks proxy › should proxy local.playwright requests
:warning: [firefox-library] › library/inspector/cli-codegen-javascript.spec.ts:87:5 › should save the codegen output to a file if specified
:warning: [firefox-library] › library/inspector/cli-codegen-python-async.spec.ts:39:5 › should print the correct context options for custom settings
:warning: [firefox-library] › library/proxy.spec.ts:229:3 › should use socks proxy
:warning: [firefox-page] › page/elementhandle-wait-for-element-state.spec.ts:117:3 › should wait for stable position
:warning: [firefox-library] › library/permissions.spec.ts:103:5 › permissions › should trigger permission onchange
:warning: [firefox-library] › library/tracing.spec.ts:412:14 › should produce screencast frames fit
:warning: [firefox-page] › page/elementhandle-wait-for-element-state.spec.ts:117:3 › should wait for stable position
:warning: [firefox-page] › page/page-click-timeout-4.spec.ts:20:3 › should timeout waiting for stable position
:warning: [firefox-page] › page/elementhandle-wait-for-element-state.spec.ts:117:3 › should wait for stable position
:warning: [firefox-page] › page/page-click.spec.ts:848:3 › should not hang when frame is detached
:warning: [chromium-library] › library/browsercontext-viewport.spec.ts:143:12 › should drag with high dpi
:warning: [chromium-page] › page/page-drag.spec.ts:53:5 › Drag and drop › should not send dragover on the first mousemove
:warning: [chromium-page] › page/page-drag.spec.ts:70:5 › Drag and drop › should work inside iframe
:warning: [webkit-library] › library/browsercontext-reuse.spec.ts:50:1 › should reset serviceworker
:warning: [webkit-library] › library/inspector/cli-codegen-1.spec.ts:227:7 › cli codegen › should fill
:warning: [webkit-page] › page/page-drag.spec.ts:70:5 › Drag and drop › should work inside iframe
:warning: [webkit-page] › page/page-drag.spec.ts:100:5 › Drag and drop › should cancel on escape
:warning: [webkit-page] › page/page-drag.spec.ts:289:5 › Drag and drop › should work with the helper method
:warning: [webkit-page] › page/workers.spec.ts:243:3 › should support offline
:warning: [webkit-library] › library/emulation-focus.spec.ts:203:12 › should trigger hover state concurrently
:warning: [webkit-library] › library/inspector/cli-codegen-3.spec.ts:22:7 › cli codegen › should click locator.first
:warning: [webkit-library] › library/popup.spec.ts:115:3 › should inherit viewport size from browser context
:warning: [webkit-page] › page/page-drag.spec.ts:330:5 › Drag and drop › should work with locators
:warning: [webkit-library] › library/browsercontext-reuse.spec.ts:206:1 › should reset mouse position
:warning: [webkit-library] › library/browsercontext-viewport.spec.ts:143:12 › should drag with high dpi
:warning: [webkit-library] › library/inspector/cli-codegen-3.spec.ts:415:7 › cli codegen › should generate getByPlaceholder
:warning: [webkit-library] › library/inspector/cli-codegen-pytest.spec.ts:33:5 › should print the correct context options when using a device and lang
:warning: [webkit-page] › page/page-request-fallback.spec.ts:239:5 › post data › should amend binary post data
:warning: [webkit-library] › library/trace-viewer.spec.ts:893:1 › should display waitForLoadState even if did not wait for it
:warning: [webkit-library] › library/tracing.spec.ts:412:14 › should produce screencast frames fit
:warning: [webkit-library] › library/tracing.spec.ts:412:14 › should produce screencast frames crop
:warning: [webkit-library] › library/tracing.spec.ts:412:14 › should produce screencast frames scale

2 interrupted

:warning: [firefox-page] › page/page-click-scroll.spec.ts:90:3 › should scroll into view span element
:warning: [firefox-page] › page/page-click-timeout-2.spec.ts:20:3 › should timeout waiting for display:none to be gone

179816 passed, 7825 skipped, 1283 did not run :heavy_check_mark::heavy_check_mark::heavy_check_mark:

Merge workflow run.

[github-actions[bot]]

Test results for "tests 1"

2 failed :x: [installation tests] › playwright-test-package-managers.spec.ts:38:5 › npm: @playwright/test + playwright-core should work :x: [installation tests] › screencast.spec.ts:18:5 › screencast works

1 flaky

:warning: [webkit-library] › library/browsercontext-clearcookies.spec.ts:116:3 › should remove cookies by path

27406 passed, 671 skipped :heavy_check_mark::heavy_check_mark::heavy_check_mark:

Merge workflow run.

[stephenwade]

@mxschmitt Looks like "tests 2" didn't run when I pushed another commit, and the Installation Test needs to be re-run. Could you help?

[github-actions[bot]]

Test results for "tests 2"

1 fatal errors, not part of any test 6 failed :x: [firefox-library] › library/inspector/cli-codegen-2.spec.ts:340:7 › cli codegen › should record open in a new tab with url :x: [firefox-library] › library/inspector/cli-codegen-2.spec.ts:340:7 › cli codegen › should record open in a new tab with url :x: [webkit-library] › library/capabilities.spec.ts:67:3 › should play video @smoke :x: [webkit-library] › library/capabilities.spec.ts:96:3 › should play audio @smoke :x: [webkit-library] › library/signals.spec.ts:62:7 › signals › should report browser close signal 2 :x: [webkit-library] › library/tracing.spec.ts:412:14 › should produce screencast frames fit

46 flaky

:warning: [chromium-library] › library/browsercontext-viewport.spec.ts:143:12 › should drag with high dpi
:warning: [chromium-page] › page/page-add-locator-handler.spec.ts:88:5 › should work with locator.hover()
:warning: [chromium-page] › page/page-drag.spec.ts:272:5 › Drag and drop › should work if a frame is stalled
:warning: [chromium-library] › library/trace-viewer.spec.ts:893:1 › should display waitForLoadState even if did not wait for it
:warning: [chromium-page] › page/page-event-request.spec.ts:110:3 › should report navigation requests and responses handled by service worker
:warning: [firefox-page] › page/page-click.spec.ts:445:3 › should wait for stable position
:warning: [firefox-library] › library/tracing.spec.ts:412:14 › should produce screencast frames crop
:warning: [firefox-page] › page/page-click.spec.ts:232:3 › should click on checkbox input and toggle
:warning: [firefox-page] › page/page-drag.spec.ts:25:5 › Drag and drop › should work @smoke
:warning: [firefox-library] › library/video.spec.ts:165:5 › screencast › should work with old options
:warning: [firefox-page] › page/page-click-timeout-4.spec.ts:20:3 › should timeout waiting for stable position
:warning: [firefox-page] › page/page-drag.spec.ts:178:5 › Drag and drop › should respect the drop effect
:warning: [firefox-library] › library/browsercontext-pages.spec.ts:58:3 › should not hang with touch-enabled viewports
:warning: [firefox-library] › library/download.spec.ts:80:5 › download event › should work with Cross-Origin-Opener-Policy
:warning: [firefox-library] › library/hit-target.spec.ts:398:3 › should click in iframe with padding
:warning: [firefox-library] › library/inspector/cli-codegen-1.spec.ts:625:7 › cli codegen › should await popup
:warning: [firefox-library] › library/inspector/cli-codegen-2.spec.ts:169:7 › cli codegen › should clear files
:warning: [firefox-library] › library/inspector/cli-codegen-2.spec.ts:200:7 › cli codegen › should download files
:warning: [firefox-library] › library/inspector/cli-codegen-2.spec.ts:493:7 › cli codegen › should --save-trace
:warning: [firefox-library] › library/inspector/cli-codegen-python.spec.ts:78:5 › should save the codegen output to a file if specified
:warning: [firefox-library] › library/selector-generator.spec.ts:494:5 › selector generator › should generate exact placeholder when necessary
:warning: [firefox-page] › page/page-click-timeout-4.spec.ts:20:3 › should timeout waiting for stable position
:warning: [firefox-page] › page/page-click.spec.ts:99:3 › should click the 1x1 div
:warning: [firefox-page] › page/page-click.spec.ts:445:3 › should wait for stable position
:warning: [firefox-page] › page/page-goto.spec.ts:81:3 › should work with Cross-Origin-Opener-Policy
:warning: [firefox-page] › page/elementhandle-wait-for-element-state.spec.ts:117:3 › should wait for stable position
:warning: [firefox-page] › page/page-event-request.spec.ts:171:3 › should return response body when Cross-Origin-Opener-Policy is set
:warning: [firefox-library] › library/permissions.spec.ts:103:5 › permissions › should trigger permission onchange
:warning: [firefox-page] › page/page-click-timeout-4.spec.ts:20:3 › should timeout waiting for stable position
:warning: [firefox-page] › page/page-click.spec.ts:848:3 › should not hang when frame is detached
:warning: [firefox-page] › page/elementhandle-wait-for-element-state.spec.ts:117:3 › should wait for stable position
:warning: [firefox-page] › page/page-click-timeout-4.spec.ts:20:3 › should timeout waiting for stable position
:warning: [firefox-page] › page/page-click-timeout-4.spec.ts:20:3 › should timeout waiting for stable position
:warning: [chromium-library] › library/downloads-path.spec.ts:105:5 › downloads path › should delete downloads when persistent context closes
:warning: [chromium-library] › library/inspector/cli-codegen-pytest.spec.ts:57:5 › should save the codegen output to a file if specified
:warning: [chromium-library] › library/browsercontext-viewport.spec.ts:143:12 › should drag with high dpi
:warning: [chromium-page] › page/page-drag.spec.ts:70:5 › Drag and drop › should work inside iframe
:warning: [chromium-page] › page/page-drag.spec.ts:100:5 › Drag and drop › should cancel on escape
:warning: [chromium-page] › page/page-drag.spec.ts:330:5 › Drag and drop › should work with locators
:warning: [webkit-page] › page/page-add-locator-handler.spec.ts:88:5 › should work with locator.hover()
:warning: [webkit-library] › library/inspector/cli-codegen-1.spec.ts:450:7 › cli codegen › should check
:warning: [webkit-library] › library/browsercontext-viewport.spec.ts:143:12 › should drag with high dpi
:warning: [webkit-library] › library/emulation-focus.spec.ts:203:12 › should trigger hover state concurrently
:warning: [webkit-library] › library/inspector/cli-codegen-1.spec.ts:323:7 › cli codegen › should fill [contentEditable]
:warning: [webkit-library] › library/inspector/cli-codegen-1.spec.ts:520:7 › cli codegen › should uncheck
:warning: [webkit-page] › page/page-mouse.spec.ts:186:3 › should trigger hover state on disabled button

2 interrupted

:warning: [firefox-page] › page/page-autowaiting-basic.spec.ts:46:3 › should await cross-process navigation when clicking anchor
:warning: [firefox-page] › page/page-autowaiting-no-hang.spec.ts:20:3 › clicking on links which do not commit navigation

179773 passed, 7822 skipped, 1355 did not run :heavy_check_mark::heavy_check_mark::heavy_check_mark:

Merge workflow run.

[stephenwade]

@mxschmitt Looks like the "should play audio" and "should play video" tests are flaky on macos-12. This lines up with what I found while testing audio playback on GitHub Actions - for some reason, they work on macos-11, macos-13, and macos-14 but work only about half the time on macos-12. What do you think about skipping these tests on that macOS version?

[stephenwade]

Hey @mxschmitt, just following up on this. I pushed a commit to skip the "should play audio" and "should play video" tests on Webkit on macos-12, since that's where I have observed flakiness. Please re-add the label to re-run the tests 😀

[github-actions[bot]]

Test results for "tests 1"

1 flaky

:warning: [playwright-test] › ui-mode-test-progress.spec.ts:165:5 › should update tracing network live

27249 passed, 670 skipped :heavy_check_mark::heavy_check_mark::heavy_check_mark:

Merge workflow run.

[mxschmitt]

Re-triggered the bot. Also landed this https://github.com/microsoft/playwright/pull/30634 which should make things simpler (I think it doesn't work for macOS 12 / maybe 13 but does a good job for 14).

[github-actions[bot]]

Test results for "tests 2"

6 fatal errors, not part of any test 3 failed :x: [firefox-library] › library/inspector/cli-codegen-2.spec.ts:340:7 › cli codegen › should record open in a new tab with url :x: [firefox-library] › library/inspector/cli-codegen-2.spec.ts:340:7 › cli codegen › should record open in a new tab with url :x: [firefox-library] › library/inspector/cli-codegen-2.spec.ts:340:7 › cli codegen › should record open in a new tab with url

48 flaky

:warning: [chromium-library] › library/inspector/cli-codegen-1.spec.ts:23:7 › cli codegen › should click
:warning: [chromium-page] › page/page-drag.spec.ts:34:5 › Drag and drop › should send the right events
:warning: [chromium-page] › page/page-mouse.spec.ts:166:3 › should trigger hover state
:warning: [chromium-library] › library/browsercontext-viewport.spec.ts:143:12 › should drag with high dpi
:warning: [chromium-page] › page/page-drag.spec.ts:25:5 › Drag and drop › should work @smoke
:warning: [chromium-library] › library/selector-generator.spec.ts:406:5 › selector generator › should work without CSS.escape
:warning: [chromium-library] › library/trace-viewer.spec.ts:893:1 › should display waitForLoadState even if did not wait for it
:warning: [chromium-page] › page/page-leaks.spec.ts:153:5 › waitFor should not leak
:warning: [firefox-page] › page/page-click.spec.ts:850:3 › should not hang when frame is detached
:warning: [firefox-library] › library/capabilities.spec.ts:258:3 › requestFullscreen
:warning: [firefox-library] › library/trace-viewer.spec.ts:893:1 › should display waitForLoadState even if did not wait for it
:warning: [firefox-library] › library/video.spec.ts:351:5 › screencast › should capture navigation
:warning: [firefox-library] › library/inspector/cli-codegen-1.spec.ts:159:7 › cli codegen › should work with TrustedTypes
:warning: [firefox-library] › library/tap.spec.ts:23:3 › should send all of the correct events @smoke
:warning: [firefox-library] › library/video.spec.ts:165:5 › screencast › should work with old options
:warning: [firefox-page] › page/page-click.spec.ts:232:3 › should click on checkbox input and toggle
:warning: [firefox-library] › library/hit-target.spec.ts:398:3 › should click in iframe with padding
:warning: [firefox-library] › library/inspector/cli-codegen-2.spec.ts:138:7 › cli codegen › should upload multiple files
:warning: [firefox-library] › library/inspector/cli-codegen-2.spec.ts:200:7 › cli codegen › should download files
:warning: [firefox-library] › library/inspector/cli-codegen-3.spec.ts:383:7 › cli codegen › should generate getByTestId
:warning: [firefox-page] › page/page-click-scroll.spec.ts:101:3 › should scroll into view element in iframe
:warning: [firefox-page] › page/page-click.spec.ts:99:3 › should click the 1x1 div
:warning: [firefox-library] › library/permissions.spec.ts:103:5 › permissions › should trigger permission onchange
:warning: [firefox-page] › page/page-click.spec.ts:850:3 › should not hang when frame is detached
:warning: [chromium-library] › library/browsercontext-viewport.spec.ts:143:12 › should drag with high dpi
:warning: [chromium-library] › library/inspector/cli-codegen-3.spec.ts:57:7 › cli codegen › should click locator.nth
:warning: [chromium-page] › page/page-add-locator-handler.spec.ts:90:5 › should work with locator.hover()
:warning: [chromium-page] › page/page-click.spec.ts:232:3 › should click on checkbox input and toggle
:warning: [chromium-page] › page/page-event-request.spec.ts:110:3 › should report navigation requests and responses handled by service worker
:warning: [firefox-page] › page/frame-goto.spec.ts:46:3 › should continue after client redirect
:warning: [webkit-library] › library/browsercontext-reuse.spec.ts:50:1 › should reset serviceworker
:warning: [webkit-library] › library/trace-viewer.spec.ts:1049:1 › should pick locator in iframe
:warning: [webkit-page] › page/page-drag.spec.ts:34:5 › Drag and drop › should send the right events
:warning: [webkit-page] › page/page-drag.spec.ts:70:5 › Drag and drop › should work inside iframe
:warning: [webkit-library] › library/inspector/cli-codegen-1.spec.ts:404:7 › cli codegen › should record ArrowDown
:warning: [webkit-library] › library/inspector/cli-codegen-2.spec.ts:518:7 › cli codegen › should fill tricky characters
:warning: [webkit-page] › page/page-add-locator-handler.spec.ts:90:5 › should work with locator.hover()
:warning: [webkit-page] › page/page-drag.spec.ts:70:5 › Drag and drop › should work inside iframe
:warning: [webkit-page] › page/page-drag.spec.ts:246:5 › Drag and drop › should work if the drag event is captured but not canceled
:warning: [webkit-library] › library/browsercontext-reuse.spec.ts:50:1 › should reset serviceworker
:warning: [webkit-library] › library/trace-viewer.spec.ts:122:1 › should contain action info
:warning: [webkit-page] › page/page-drag.spec.ts:25:5 › Drag and drop › should work @smoke
:warning: [webkit-page] › page/page-drag.spec.ts:246:5 › Drag and drop › should work if the drag event is captured but not canceled
:warning: [webkit-page] › page/page-drag.spec.ts:272:5 › Drag and drop › should work if a frame is stalled
:warning: [webkit-page] › page/page-drag.spec.ts:330:5 › Drag and drop › should work with locators
:warning: [webkit-page] › page/page-request-continue.spec.ts:227:5 › post data › should amend utf8 post data
:warning: [webkit-library] › library/browsercontext-reuse.spec.ts:50:1 › should reset serviceworker
:warning: [webkit-page] › page/page-goto.spec.ts:277:3 › should fail when navigating to bad SSL after redirects

167883 passed, 7560 skipped, 3960 did not run :heavy_check_mark::heavy_check_mark::heavy_check_mark:

Merge workflow run.

[pavelfeldman]

Archiving this