[Feature] Combine sharding with html reporter

[bamanczak]

Currently there is no way to combine test results into one html report, while using sharding on multiple machines. (v1.17.0-rc1)

I end up with multiple reports which makes it harder for sharing with others in case of failures.

It would be nice to be able to combine multiple executions into 1 report.

I believe anyone who wants to scale tests horizontally might benefit from such feature

[JoelEinbinder]

Sounds reasonable to me! What kind of api would you like to combine the reports together?

[bamanczak]

I personally have no preferences for how to merge the reports together. Whatever I can do on CI in a fast way.

[MindaugasMateika]

Since I'm currently using https://plugins.jenkins.io/allure-jenkins-plugin/ to merge multiple Allure reports and it is as simple as 👇 then I'd favor a similar approach to HTML reporter but any other out-of-the-box no-brainer solution would work as well.

            script {
                allure([
                        commandline      : 'allure-2.x',
                        includeProperties: false,
                        jdk              : '',
                        results          : [
                                [path: 'allure-results1'],
                                [path: 'allure-results2']
                        ]
                ])

[liamdebeasi]

This feature would be great to have. We are looking to migrate our E2E testing solution to Playwright. We have a component library where each test runs several variations of a particular test to check for visual regressions. Some of the variants we have are:

1. LTR vs RTL modes
2. Desktop vs mobile viewports
3. Different themes for the components

Right now the results for these tests can get split up across multiple shards. It would be great to have a single result where we can see all of the test results.

[anooprav7]

I was able to build one solution for my use case (in Node.js). I had multiple reports from different shards (tests distributed with a custom logic) in a single run and wanted to send a single merged html report. Hope it helps.

Installation

npm install playwright-merge-html-reports --dev

Usage

const { mergeHTMLReports } = require("playwright-merge-html-reports");

mergeHTMLReports([
  process.cwd() + "/html_report-1",
  process.cwd() + "/html_report-2"
])

• It creates a separate HTML merged report, folder name - merged-html-report (customizable)

:warning: Tested on some use cases: Still at early stages of development

Github Repo NPM package

[btd]

@anooprav7 thanks kind men. I will try this in a week or two.

[ghost]

Is this feature still in progress?

[bamanczak]

My test suite becomes bigger and bigger every week, and this is a blocker for my team to start using sharding.

The solution mentioned by anooprav7 seems cool, but we would like to have a stable version of such toolkit. I was really hoping for it to appear in 1.22 :(

[mrmckeb]

We're running our tests on GitHub Actions. We'd ideally like a single report, but for now will create a report per shard/matrix runner.

I'm not sure how this should work - there's no easy mechanism for the runners to talk to each other to combine a report.

[mstepin]

wanting this too. We run our tests back to back with various different configurations and want to combine the final test results into a single html. So pls team at MS and Playwright, at some point in future, make this a native playwright feature. In the meantime, we have wired in the module provided by @anooprav7 . There are a bunch of issues with that module that we needed to address so just a heads up to anyone else try it.

1. We use NodeJS 12.X. This limits us to Playwright 1.21.X or lower. The module looks for fs/promises but cant find it. We've had to alias in the fs.promises module to that path.
2. If a test is successful and no data is generated as a result (in the 'data' directory) it also breaks, as it cant find the dir. So we've mitigated it by manually creating that dir no matter what. I believe there is a PR to fix both the above waiting to merged in gitHub for that module. But as for now there is no ETA. @anooprav7 pls let us know when the fix is available.

[mrmckeb]

Feel free to install from my (unpublished) fork in the meantime, @mstepin. We're using this on one of our repositories at Vercel right now and it seems like we have everything working. https://github.com/mrmckeb/playwright-merge-html-reports/tree/with-dist

npm i https://github.com/mrmckeb/playwright-merge-html-reports#aadf6a7

[anooprav7]

The fix has been merged and published - [email protected]. Thank you @mrmckeb and @nickofthyme for the fixes (https://github.com/anooprav7/playwright-merge-html-reports/pull/2 and https://github.com/anooprav7/playwright-merge-html-reports/pull/3).

[TimeInvestor]

Looking forward to this feature. Our tests take more than 1 hour to finish using one Jenkins job. We have to split the tests into mutiple Jenkins jobs to run across mutiple Jenkins agents using Playwright sharding feature. However, getting a single HTML test report is an issue with this solution. We need to pass HTML test report for release team to review before they do release.

[nickofthyme]

@TimeInvestor just to be clear, you can currently shard your playwright tests across multiple machines and use [email protected] npm package to combine all the html reports into one.

[TimeInvestor]

@TimeInvestor just to be clear, you can currently shard your playwright tests across multiple machines and use [email protected] npm package to combine all the html reports into one.

Thanks @nickofthyme. Will check it out although I am not sure if it works well for my scenario that the test results are residing with multiple Jenkins jobs over different Jenkins slaves.

[nickofthyme]

Yeah I have it running in parallel and merging all reports across 10 agents in buildkite. The only trick is to run all test with the --shard=n/x option then after each job completes, store the report in an archive or otherwise. Then have a job that depends on the parallelized jobs that runs only after all jobs are complete across all the agents. Then collect all the reports from all the parallel jobs.

Another difficultly is deciding how to build, run or host your application for testing. I decided on just building our react app first then pulling down the files in each parallel job and running a local webServer.

It takes a bit to get it just right but the whole file archive processes and terminology is different on every ci provider.

Hope that helps, but if you run into issues with reports merging with playwright-merge-html-reports please open an issue in the GH repo and I'll take a look.

[mrmckeb]

@JoelEinbinder I was chatting with my colleagues about this today and wanted to share thoughts.

What we're doing now is running multiple shards (GitHub Actions), uploading each HTML report as an artifact, and then we have a final step that combines all the uploaded reports using playwright-merge-html-reports.

How we think an official solution might look would be that there's a new reporter which outputs the information required by the HTML reporter (perhaps the raw-reporter), and again that would be uploaded/downloaded for processing. The HTML reporter would then process those into a single HTML report.

Considerations:

• Uploaded artifacts will add up over time, so users will probably want to delete the shard reports once they've combined them.
• If someone has to re-run a single shard, we'd ideally want a way to update the combined report with that shard's updated run without the other shard reports.

[angelo-loria]

What we're doing now is running multiple shards (GitHub Actions), uploading each HTML report as an artifact, and then we have a final step that combines all the uploaded reports using playwright-merge-html-reports.

I'm looking to do the same, but I'm a little confused as to how this works with each shard outputting an html report with the same name/filepath. Is each artifact uploaded with a different name? Thanks!

[nickofthyme]

I'm looking to do the same, but I'm a little confused as to how this works with each shard outputting an html report with the same name/filepath. Is each artifact uploaded with a different name? Thanks!

@angelo-loria Yes, exactly

Ideally you override the report output directory using PLAYWRIGHT_HTML_REPORT environment variable when you run the playwright test. This can be indexed by the shard number (i.e. ./html-reports/report-${n}). Then you could use node to merge all the available reports.

const fs = require('fs');
const { mergeHTMLReports } = require('playwright-merge-html-reports');

// get all archived/uploaded report directories locally into ./html-reports/report-*
getAllReports();

// find all directories in ./html-reports/*
const reportPathsToMerge = fs
    .readdirSync('./html-reports', { withFileTypes: true })
    .filter((item) => item.isDirectory())
    .map(({ name }) => path.resolve('html-reports', name));

mergeHTMLReports(reportPathsToMerge)

The report directory name is erroneous and does not impact the final report in any way, all the data used to merge comes only from the report content.

---

Another approach would be to run playwright test normally without an indexed output path. Then use some compression library like targz that allows you to set the indexed name when compressing and decompressing. This way you also save a little time when pulling down all the reports, with the reduced size.

At the end of the day the idea is simple but the implementation is highly dependent on the CI you are using and how all the jobs and steps play nicely together. It took a lot of trial and error for me to get it just right.

[angelo-loria]

Here's what I ended up with as far as an Actions workflow (sanitized) for this. It might end up being too convoluted for us to use in production and the merging somewhat negates the performance benefits of sharding, but it does work as expected and hopefully it'll help someone out. Thanks to everyone in this thread for their help.

jobs:
  playwright_test:
    name: Playwright Test
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3]
    defaults:
      run:
        working-directory: ./playwright
    steps:
      - uses: actions/checkout@v2

      - name: Setup NodeJS SDK ${{ env.NODEJS_VERSION }}
        uses: actions/[email protected]
        id: setup-node
        with:
          node-version: ${{ env.NODEJS_VERSION }}

      - name: Yarn Install
        run: yarn install

      - name: Setup
        run: yarn setup

      - name: Run Tests
        id: test
        run: yarn test --shard=${{ matrix.shard }}/${{ strategy.job-total }}

      - name: Upload Artifact
        uses: actions/upload-artifact@v2
        if: always()
        with:
          name: report-${{ matrix.shard }}
          path: ./playwright/playwright-report

  publish_pages:
    name: Publish to GitHub Pages
    if: always()
    needs: playwright_test
    runs-on: ubuntu-latest
    # needed for gh pages deploy
    environment:
      name: ${{ inputs.environment }}-report
      url: ${{ steps.deployment.outputs.page_url }}
    defaults:
      run:
        working-directory: ./playwright
    steps:
      - uses: actions/checkout@v2

      - name: Setup NodeJS SDK ${{ env.NODEJS_VERSION }}
        uses: actions/[email protected]
        id: setup-node
        with:
          node-version: ${{ env.NODEJS_VERSION }}
      
      - uses: actions/download-artifact@v2
        with:
          path: ./playwright/playwright-report

      - name: Yarn Install
        run: yarn install

      // this is a simple "node mergeReports.js" npm script
      - name: Merge Reports
        run: yarn merge

      - name: Upload Pages Artifact
        uses: actions/upload-pages-artifact@v1
        with:
          path: './playwright/html-report'

      - name: Setup Pages
        uses: actions/configure-pages@v1

      - uses: actions/download-artifact@v2
        with:
          name: 'github-pages'
  
      - name: Deploy to GitHub Pages
        id: pages-deploy
        uses: actions/deploy-pages@v1

      - name: Message PR of Artifact Published
        if: ${{ always() && github.event_name == 'pull_request' }}
        uses: thollander/actions-comment-pull-request@main
        with:
          message: |
            See Playwright Test Report at ${{steps.pages-deploy.outputs.page_url}}
          comment_includes: "See Playwright Test Report "
          pr_number: ${{ inputs.pr_number }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

My mergeReports.js:

const fs = require("fs");
const path = require("path");
const { mergeHTMLReports } = require("playwright-merge-html-reports");

const reportPathsToMerge = fs
    .readdirSync(process.cwd() + "/playwright-report", { withFileTypes: true })
    .filter((item) => item.isDirectory())
    .map(({ name }) => path.resolve(process.cwd() + "/playwright-report", name));

mergeHTMLReports(reportPathsToMerge, {
    outputFolderName: "html-report",
});

[ldy12347]

Can it support Python?

[costag1982]

Is this feature still being considered? Im doing a POC on Playwright at work and this feature would be brilliant for us

[segevfiner]

@mxschmitt I believe this was addressed?

And was reverted as far as I can see.

[costa-collibra]

The library that a lot of us are using no longer works as of the latest Playwright version 1.32

https://github.com/Rippling/playwright-merge-html-reports/issues/2

It would be great to see if theres any progress with this native support of merging reports

[nickofthyme]

@costa-collibra - Yes native is the best solution, but have you tried using playwright-merge-html-report on v1.32?

[costa-collibra]

@nickofthyme yes I am using that library but the latest playwright version caused it to break

someone fixed it here - https://www.npmjs.com/package/@nil1511/playwright-merge-html-reports

this has fixed our builds

[nickofthyme]

Gotcha, sorry I was unaware they moved and renamed the repo & npm package.

[costa-collibra]

@nickofthyme i think someone needed it urgently so forked it, id expect this fix to go in the original repo too

[nickofthyme]

That's what I thought at first too, but the original repo points to the new location as the Rippling repo. Anyways... 😅

[cenfun]

one more option to merge shard reports