playwright
playwright copied to clipboard
microsoft
[BUG] screenshots hang on arm64 docker image
System info
- Playwright Version: v1.41.0
- Operating System: Ubuntu 22.04.3
- Browser: Chromium
- Other info: Tests a browser extension using
chromium.launchPersistentContext
Source code
https://github.com/stellar/freighter/pull/1084 Tests are here - https://github.com/stellar/freighter/tree/master/extension/e2e-tests
Steps
docker run --platform=linux/arm64 --ipc=host --rm --network host -v $(pwd):/work/ -w /work/ -it --cap-add=SYS_ADMIN --init mcr.microsoft.com/playwright:v1.41.0-jammy /bin/bash
yarn && yarn test:e2e
Expected Screenshots to be generated for linux.
Actual Runner hangs indefinitely
Info
I'm able to run these tests locally on my M1, and can include the toHaveScreenshot assertions and get the expected screenshots and desired behavior. When I try to run the same tests in the docker container, the runner hangs until it times out, unable to take the screenshot. I've tried increasing my timeout by a lot, and taking partial page screenshots, as well as disabling animations. Interestingly, the very first screenshot does get taken but any subsequent ones hang until a timeout happens.
stellar/freighter#1084
Note that the linux screenshots are missing, so the tests are bound to fail. Can you generate the expected images and retry the tests?
Btw, on github actions the default hardware is x64, not arm and the tests are failing there too in your PR.
Does it also hang without the extension?
The whole project is a browser extension so I don't think I can test that with this repo, but if I write a very rudimentary test suite targeting a web app context it does not hang so I think it's fair to be suspicious of using chromium.launchPersistentContext and trying to take screenshots on that image.
Does it also hang without the extension?
The whole project is a browser extension so I don't think I can test that with this repo, but if I write a very rudimentary test suite targeting a web app context it does not hang so I think it's fair to be suspicious of using
chromium.launchPersistentContextand trying to take screenshots on that image.
This happens when trying to generate the images for linux as well. The tests in my action are run on an ubuntu image, you can see in the output that it is expecting the images to be for linux.
If I run this with the debug flag, I do see this warning about the headless option even though I do have that option set to true
root@docker-desktop:/work/extension# npx playwright test --debug
Running 5 tests using 1 worker
✘ 1 [chromium] › onboarding.test.ts:8:5 › Welcome page loads (762ms)
1) [chromium] › onboarding.test.ts:8:5 › Welcome page loads ──────────────────────────────────────
Error: browserType.launchPersistentContext: Target page, context or browser has been closed
Browser logs:
╔════════════════════════════════════════════════════════════════════════════════════════════════╗
║ Looks like you launched a headed browser without having a XServer running. ║
║ Set either 'headless: true' or use 'xvfb-run <your-playwright-app>' before running Playwright. ║
║ ║
║ <3 Playwright Team ║
╚════════════════════════════════════════════════════════════════════════════════════════════════╝
Call log:
- <launching> /ms-playwright/chromium-1097/chrome-linux/chrome --disable-field-trial-config --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-back-forward-cache --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-component-update --no-default-browser-check --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=ImprovedCookieControls,LazyFrameLoading,GlobalMediaControls,DestroyProfileOnBrowserClose,MediaRouter,DialMediaRouteProvider,AcceptCHFrame,AutoExpandDetailsElement,CertificateTransparencyComponentUpdater,AvoidUnnecessaryBeforeUnloadCheckSync,Translate,HttpsUpgrades,PaintHolding --allow-pre-commit-input --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --no-service-autorun --export-tagged-pdf --disable-search-engine-choice-screen --no-sandbox --headless=new --disable-extensions-except=/work/extension/build --load-extension=/work/extension/build --user-data-dir=/tmp/playwright_chromiumdev_profile-mgOS8K --remote-debugging-pipe about:blank
- <launched> pid=2145
- [pid=2145][err] [2145:2159:0125/162140.080288:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
- [pid=2145][err] [2145:2162:0125/162140.099019:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
- [pid=2145][err] [2145:2162:0125/162140.099099:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
- [pid=2145][err] [2145:2159:0125/162140.107588:ERROR:bus.cc(407)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
- [pid=2145][err] [2145:2159:0125/162140.107929:ERROR:bus.cc(407)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
- [pid=2145][err] [2145:2159:0125/162140.108116:ERROR:bus.cc(407)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
- [pid=2145][err] [2145:2159:0125/162140.108143:ERROR:bus.cc(407)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
- [pid=2145][err] [2145:2145:0125/162140.121672:ERROR:policy_logger.cc(156)] :components/enterprise/browser/controller/chrome_browser_cloud_management_controller.cc(161) Cloud management controller initialization aborted as CBCM is not enabled. Please use the `--enable-chrome-browser-cloud-management` command line flag to enable it if you are not using the official Google Chrome build.
- [pid=2145][err] [2145:2159:0125/162140.146086:ERROR:bus.cc(407)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
- [pid=2145][err] [2145:2159:0125/162140.146119:ERROR:bus.cc(407)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
- [pid=2145][err] [2145:2231:0125/162140.273233:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
- [pid=2145][err] [2145:2231:0125/162140.273364:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
- [pid=2145][err] [2145:2231:0125/162140.273400:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
- [pid=2145][err] [2145:2231:0125/162140.273474:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
- [pid=2145][err] [2145:2231:0125/162140.273552:ERROR:bus.cc(407)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
at test-fixtures.ts:10
8 | context: async ({}, use) => {
9 | const pathToExtension = path.join(__dirname, "../build");
> 10 | const context = await chromium.launchPersistentContext("", {
| ^
11 | headless: true,
12 | args: [
13 | `--headless=new`,
at Object.context (/work/extension/e2e-tests/test-fixtures.ts:10:21)
Testing stopped early after 1 maximum allowed failures.
1 failed
[chromium] › onboarding.test.ts:8:5 › Welcome page loads ───────────────────────────────────────
4 did not run
If I run this with the debug flag, I do see this warning about the headless option even though I do have that option set to
true
--debug option implies --headed, this is why you see the error. You can fix it by bringing up an X server in the docker.
I tried running a subset of the tests with toHaveScreenshot uncommented and if I generate the expectations first with -u option, on the next run they will pass (ok, they seem to be flaky but that's a different problem). The command I ran are:
CI=1 yarn test:e2e -- onboarding.test.ts:8 onboarding.test.ts:17 onboarding.test.ts:59 -u
and after that one generated expected images, I ran
CI=1 yarn test:e2e -- onboarding.test.ts:8 onboarding.test.ts:17 onboarding.test.ts:59
with the following output:
# CI=1 yarn test:e2e -- onboarding.test.ts:8 onboarding.test.ts:17 onboarding.test.ts:59
yarn run v1.22.21
warning From Yarn 1.0 onwards, scripts don't require "--" for options to be forwarded. In a future version, any explicit "--" will be forwarded as-is to the scripts.
$ yarn workspace extension test:e2e onboarding.test.ts:8 onboarding.test.ts:17 onboarding.test.ts:59
$ playwright test onboarding.test.ts:8 onboarding.test.ts:17 onboarding.test.ts:59
Running 3 tests using 1 worker
✓ 1 [chromium] › onboarding.test.ts:8:5 › Welcome page loads (4.9s)
✓ 2 [chromium] › onboarding.test.ts:17:5 › Create new wallet (4.6s)
✘ 3 [chromium] › onboarding.test.ts:59:5 › Import 12 word wallet (9.8s)
✓ 4 [chromium] › onboarding.test.ts:59:5 › Import 12 word wallet (retry #1) (4.5s)
1) [chromium] › onboarding.test.ts:59:5 › Import 12 word wallet ──────────────────────────────────
Error: Screenshot comparison failed:
Timeout 5000ms exceeded.
Expected: /workspaces/playwright-browsers/freighter/extension/e2e-tests/onboarding.test.ts-snapshots/wallet-import-complete-page-chromium-linux.png
Call log:
- page._expectScreenshot with timeout 5000ms
- verifying given screenshot expectation
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- Timeout 5000ms exceeded.
89 |
90 | await expect(page.getByText("Wallet created successfully!")).toBeVisible();
> 91 | await expect(page).toHaveScreenshot("wallet-import-complete-page.png");
| ^
92 | });
93 |
94 | test("Import 24 word wallet", async ({ page }) => {
at /workspaces/playwright-browsers/freighter/extension/e2e-tests/onboarding.test.ts:91:22
attachment #1: wallet-import-complete-page-expected.png (image/png) ────────────────────────────
e2e-tests/onboarding.test.ts-snapshots/wallet-import-complete-page-chromium-linux.png
────────────────────────────────────────────────────────────────────────────────────────────────
1 flaky
[chromium] › onboarding.test.ts:59:5 › Import 12 word wallet ───────────────────────────────────
2 passed (24.4s)
Done in 25.32s.
increasing toHaveScreenshot's timeout to 10s fixed Import 12 word wallet.
We should not be retrying the error when expected image is missing, neither we should generate one unless --update-snapshots option is passed. We can fix that, this will make your tests fail earlier when the expectations are missing.
I see that when you provided linux expectations in the pr by renaming -darwin to -linux, the tests were failing with screenshots mismatch (see screenshot below), which is expected. Try generating the screenshots on linux, the error should disappear. If not, can you share a link to GHA output?
Also note that by using the same screenshot name "welcome-page.png" in multiple tests in the same file, you make all of them share the same expectation. Not sure if it's intentional.
Further debugging revealed that Page.captureScreenshot CDP command never returns when capturing screenshot of a page with the extension. This is an issue in Chromium implementation of the command.
Further debugging revealed that
Page.captureScreenshotCDP command never returns when capturing screenshot of a page with the extension. This is an issue in Chromium implementation of the command.
Thanks for digging into and the additional info. Do you happen to know if this is already an open issue on Chromium? Otherwise I can gather the details and open one with them.