[Bug]: Firefox tests in macos 14 time out / take a long time

[canstand]

Version

1.44.0

Steps to reproduce

Refer https://github.com/microsoft/playwright/actions/runs/8973892628

• All jobs of firefox in macos-14 (macos-latest) timeout after 1h30m
• Jobs of firefox in macos-12(or macos-13) only took about 30m

(another proof: https://github.com/canstand/playwright-go/actions/runs/8999400053/job/24721476663 )

Expected behavior

The test job takes similar time under different macos versions.

Actual behavior

Test job in macos-14(firefox) timeout.

Additional context

No response

Environment

OS: macos 14
Browser: firefox
playwright: v1.44.0

[mxschmitt]

Looking at the recent commits, it always times out.

[aslushnikov]

Investigation notes

We see that the frameAttached event for the main frame takes 5 seconds to arrive. This might be related to the slow-down to create a new process for the page. We only see this happenning under the following circumstances:

• in Firefox
• with MacOS 14
• on Github Actions

History: Firefox was timing out on macos-14 since the test bots were introduced: https://github.com/microsoft/playwright/commit/4affcdef71624e9c0982f9eb5be19c7e4d2251ab

Reproduction:

Github Actions provide different architectures for the different size runners (details here and here):

• macos-14 is aarch64 - ❌ has 5 seconds delay for each newPage
• macos-14-large is x86-64 - ✅ works as intended
• macos-14-xlarge is aarch64 - ❌ has 5 seconds delay for each newPage

Locally, on my macbook with MacOS-14 and aarch64 - ✅ works as intended.

• This behavior reproduces with pptr-firefox with Firefox nightly, and webdriver-bidi with Firefox Stable

WebDriver script

const { Builder } = require('selenium-webdriver');
const firefox = require('selenium-webdriver/firefox');

(async function example() {
    let options = new firefox.Options();
    // Enable BiDi
    options.setPreference('remote.bidi.enabled', true);

    let driver = await new Builder()
        .forBrowser('firefox')
        .setFirefoxOptions(options)
        .build();

    try {
        // Navigate to a URL
        await driver.get('about:blank');

    } finally {
        // Close the browser
        await driver.quit();
    }
})();

Puppeteer Firefox script

// make sure to run `npx puppeteer browsers install firefox` beforehand
import pptr from 'puppeteer';

const browser = await pptr.launch({
        product: 'firefox',
});
// Open a new page
const page = await browser.newPage();
await page.close();
await browser.close();

The easiest reproduction so far is just running the headless screenshot:

/Applications/Firefox.app/Contents/MacOS/firefox --headless --screenshot /tmp/foo.png about:blank

[aslushnikov]

This seems to be an issue with Apple Silicon virtualization + Firefox.

• Upstream bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1897892
• Detailed repro: https://github.com/aslushnikov/macos-14-firefox-super-slow/tree/main

[mxschmitt]

Note: We temporarily mitigated it for now by continuing to use macOS Intel for our testing infrastructure in order to not loose any test coverage. In https://github.com/microsoft/playwright/pull/31121.

[aslushnikov]

Note: this seems to be fixed with the latest Firefox Beta 128

[mxschmitt]

This seems to affect some feature which is not yet enabled on Firefox Stable.

[juliandescottes]

FWIW, Firefox had a sandboxing issue with GitHub action arm64 workers (same symptoms, 5s per tab opening) which was fixed in Firefox 129 (https://bugzilla.mozilla.org/show_bug.cgi?id=1893921)

[dgozman]

After the roll to Firefox 130, Playwright's own bots are back to normal again. Thanks to the Firefox team for fixing the issue!