[Regression]: Reports are not merged.

[sleepmac]

Last Good Version

1.38

First Bad Version

1.42

Steps to reproduce

Hi. After upgrading from version 1.38 to 1.42 after running pipeline the reports are not merged. I have three projects desktop, iphone, lg. Each test is run for three projects, tests are run in parallel. Each project is divided into shards.

Expected behavior

After passing the pipeline, the report shows only tests for one project lg. I expect three projects in the report

Actual behavior

Report with one project.

Additional context

gitlab-ci

variables: 
  STAGE:
    value: "test"
  BASEURL: "https://${STAGE}.example.com/"
  PLAYWRIGHT_BLOB_REPORT: "blob-report"
  PLAYWRIGHT_TESTS_RESULTS: "test-results"
  PWTEST_BLOB_REPORT_NAME: "${PW_PROJECT}"
  PLAYWRIGHT_JUNIT_OUTPUT_NAME: "results.xml"
  PLAYWRIGHT_HTML_REPORT: "playwright-report"
  PROJECT:
    value: "all"
    options:
      - "all"
      - "desktop"
      - "mobile"
  GREP:
    value: ""

stages:
  - tests
  - reports

.playwright_job: &playwright_job
  image: mcr.microsoft.com/playwright:v1.42.0-jammy

.tests_job: &tests_job
  <<: *playwright_job
  stage: tests
  allow_failure: true
  script:
    - |
      options=()
      if [[ -n "$GREP" ]]; then
        options+=(--grep "$GREP")
      fi
      export PWTEST_BLOB_REPORT_NAME="${PW_PROJECT}"
      env
      PWTEST_BLOB_REPORT_NAME=$PW_PROJECT node -e "console.log(process.env.PWTEST_BLOB_REPORT_NAME)"
      node -e "console.log(process.env.PWTEST_BLOB_REPORT_NAME)"
      PWTEST_BLOB_REPORT_NAME=$PW_PROJECT npx playwright test \
        --retries=1 \
        --project=$PW_PROJECT \
        --shard=$SHARD \
        --reporter=blob,junit \
        "${options[@]}"
      apt update -y && apt install tree -y
      tree $PLAYWRIGHT_BLOB_REPORT
  artifacts:
    paths:
      - $PLAYWRIGHT_BLOB_REPORT
      - $PLAYWRIGHT_TESTS_RESULTS
      - $PLAYWRIGHT_JUNIT_OUTPUT_NAME
    reports:
      junit:
        - $PLAYWRIGHT_JUNIT_OUTPUT_NAME
    allow_failure:
      exit_codes: 1

all tests:
  <<: *tests_job
  parallel:
    matrix:
      - PW_PROJECT: [ 'desktop', 'iphone', 'lg' ]
        SHARD: [ '1/4', '2/4', '3/4', '4/4' ]
  rules:
    - if: $PROJECT == "all"

desktop tests:
  <<: *tests_job
  parallel:
    matrix:
      - PW_PROJECT: [ 'desktop' ]
        SHARD: [ '1/4', '2/4', '3/4', '4/4' ]
  rules:
    - if: $PROJECT == "desktop"

mobile tests:
  <<: *tests_job
  parallel:
    matrix:
      - PW_PROJECT: [ 'iphone', 'lg' ]
        SHARD: [ '1/4', '2/4', '3/4', '4/4' ]
  rules:
    - if: $PROJECT == "mobile"

pages:
  <<: *playwright_job
  stage: reports
  variables:
    PLAYWRIGHT_HTML_REPORT: merged-report
  script:
    - npx playwright merge-reports --reporter html ./$PLAYWRIGHT_BLOB_REPORT
    - tree $PLAYWRIGHT_BLOB_REPORT

Environment

Operating System: Ubuntu 22.04.3 LTS x86_64
Extra: node v20.5.1

[pavelfeldman]

We need a minimal repro to act on this issue, something we could run locally to see the error.

[sleepmac]

@pavelfeldman You can run your tests, for an example.

test('@user has title', async ({ page }) => {
	await page.goto('https://playwright.dev/');

	// Expect a title "to contain" a substring.
	await expect(page).toHaveTitle(/Playwright/);
});

test('@user get started link', async ({ page }) => {
	await page.goto('https://playwright.dev/');

	// Click the get started link.
	await page.getByRole('link', { name: 'Get started' }).click();

	// Expects page to have a heading with the name of Installation.
	await expect(page.getByRole('heading', { name: 'Installation' })).toBeVisible();
});

[pavelfeldman]

This works fine for me. I need exact steps I can follow to reproduce the issue.

[sleepmac]

@pavelfeldman Hi.

1. I am running a pipeline. And I want my tests to run on three projects [ 'desktop', 'iphone', 'lg' ]. I choose PROJECT == 'all'.

2. After running, I get a report. a. At 1.38. where we can see all the tests of the three projects. b. At 1.42 we see the report only with the lg project

3. Also in the logs after the version upgrade we see changes in the blob-report names a. At 1.38 b. At 1.42

[pavelfeldman]

At this time, merge-reports only supports merging the shards where all other variables are fixed (project is the same across shards). You have been using an undocumented env variable PWTEST_BLOB_REPORT_NAME that allowed working around it and was unclashing the names. This variable is now gone.

To work around your issue, you would need to configure blob reporter to use custom report file name. Modify your config like so:

export default defineConfig({
  ...
  reporter: process.env.CI ?
    [['blob', {
      outputDir: 'my-report',
      fileName: process.env.BLOB_REPORT_FILE_NAME,
    }]] :
    [['html']], // or any other way you have your current reporter configured.
});

BLOB_REPORT_FILE_NAME=report-$PW_PROJECT-$SHARD.zip npx playwright test \
        --retries=1 \
        --project=$PW_PROJECT \
        --shard=$SHARD \
        "${options[@]}"

[pavelfeldman]

We will follow up to make it less ugly.

• Let's make sure all the parameters for reporters used on CI are env-passable so that config is not modified
• Let's make sure that file name is a hash of command line arguments to cover basic clashing scenarios
• Let's allow passing extraTags in blob report that would deliver it via metadata and fan it out to all tests in the resulting report

[pavelfeldman]

Closing as per above, please feel free to open a new issue if this does not cover your use case.