[BUG] latest version 1.34.3 passes locally but fails in CI

[ghiscoding]

System info

System: OS: Windows 10 10.0.19045 CPU: (12) x64 Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
Memory: 5.57 GB / 15.85 GB Binaries: Node: 16.18.0 - C:\Program Files\nodejs\node.EXE Yarn: 1.22.19 - ~\AppData\Roaming\npm\yarn.CMD npm: 8.10.0 - C:\Program Files\nodejs\npm.CMD Browsers: Edge: Spartan (44.19041.1266.0), Chromium (113.0.1774.50) Internet Explorer: 11.0.19041.1566 npmPackages: @playwright/test: ^1.34.3 => 1.34.3

Source code

• [ ] I provided exact source code that allows reproducing the issue locally.

Link to the GitHub repository with the repro

[https://github.com/your_profile/playwright_issue_title]

or

Config file

// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  fullyParallel: true,
  // reporter: process.env.CI ? 'dot' : 'list',
  reporter: [['html', { outputFolder: '../playwright-report' }]],
  retries: process.env.CI ? 1 : 0,
  testDir: './e2e',
  timeout: 30 * 1000,
  expect: {
    timeout: 5000,
  },
  /* Fail the build on CI if you accidentally left test.only in the source code. */
  forbidOnly: !!process.env.CI,
  use: {
    baseURL: 'http://localhost:3000/',
    headless: true,
    viewport: { width: 1280, height: 720 },
    ignoreHTTPSErrors: true,
    trace: 'on',
    video: 'on-first-retry',
  },
  projects: process.env.CI
    ? [
        { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
        // { name: 'firefox', use: { ...devices['Desktop Firefox'] } },
      ]
    : [{ name: 'chromium', use: { ...devices['Desktop Chrome'] } }],
  webServer: process.env.CI
    ? {
        command: 'pnpm serve',
        port: 3000,
      }
    : undefined,
});

Test file (self-contained)

it('should check the box using setChecked', async ({ page }) => {
  await expect(1 + 1).toBe(2);
});

Steps See my open source project, the Playwright tests fails when ran in GitHub Actions CI. I use pnpm

the full GitHub Actions config can be found at https://github.com/ghiscoding/multiple-select-vanilla/blob/main/.github/workflows/main.yml

below is a summary

steps:
  - uses: pnpm/[email protected]
    with:
      version: 8
      run_install: true

  - name: Run Playwright E2E Tests
    uses: mathiasvr/[email protected]
    id: playwrightsummary
    with:
      run: pnpm test:e2e

You can see the GitHub Action failure at this link https://github.com/ghiscoding/multiple-select-vanilla/actions/runs/5086352694/jobs/9140741590

Expected

No error in both local and CI

Notes

If I install both @playwright/test and playwright, it passes in the CI but fails locally because of the new breaking change to not install both

Actual

Running 63 tests using 1 worker
×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F
×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F×F

  1) [chromium] › events.spec.ts:4:7 › Events Demo › executing multiple actions with ms-select should fire multiple events 

    Error: browserType.launch: Executable doesn't exist at /home/runner/.cache/ms-playwright/chromium-1060/chrome-linux/chrome
    ╔═════════════════════════════════════════════════════════════════════════╗
    ║ Looks like Playwright Test or Playwright was just installed or updated. ║
    ║ Please run the following command to download new browsers:              ║
    ║                                                                         ║
    ║     npx playwright install                                              ║
    ║                                                                         ║
    ║ <3 Playwright Team                                                      ║
    ╚═════════════════════════════════════════════════════════════════════════╝

[himself65]

same error here

[aslushnikov]

@ghiscoding you have both @playwright/test and playwright in your package.json; you should remove playwright to make things work.

See our 1.34 release notes.

[ghiscoding]

@aslushnikov you did not read all my issue, I know it was removed in 1.34, but I do not have playwright and my CI fails without it

[aslushnikov]

@ghiscoding I see. You also mentioned that things work for you locally, so I'm inclined to blame some caching policies you use for your workflows.

I'd like to avoid debugging your codebase; any chance you can reproduce the failure with a minimal setup? This would prove that we have issue at Playwright.

[ghiscoding]

The minimal setup is really the project I referenced when I opened the issue, it's rather small and I linked the failing PR that I have. I use pnpm both locally and in the CI for this project. I don't think the cache is in play when dependencies get bumped and the lock file changes.

For now my only solution is to stay with v1.33.0

[mxschmitt]

This looks expected, since you don't have npx playwright install chromium in your workflow, there is then no Chromium installed. And before you had the playwright package installed which automatically installs all the browsers (what we discourage).

So to fix this, simply add npx playwright install or since you are using pnpm exec playwright install chromium to your GitHub workflow file.

[ghiscoding]

@mxschmitt ok that does work but I wonder doesn't that bypass the npm/pnpm cache though since it's outside of the pnpm install flow? I would prefer to always use cache when possible. Maybe another out of subject but running the pnpm exec command doesn't even show up in my GitHub Action workflow window which is strange, I like to see all possible details

[mxschmitt]

Yes, installing browsers is a dedicated step. You can cache the browsers see here but we don't recommend it, since you need to install the browser OS dependencies anyways, APT dependencies can't be cached. As an alternative you could use the Docker container, which has both installed already.

We recommend installing the browsers for every CI run.

[ghiscoding]

ok then I guess I can close this issue as resolved and perhaps caching browser install could be improved in the future. Thanks for the help in fixing this

[mxschmitt]

If you want to contribute to our docs, especially the ci.md to help users on how to cache the browsers with pnpm/npm (both are similar) we would appreciate, thanks!