[Feature] Track Code Coverage via istanbul

[mxschmitt]

Update: It can be easily integrated in the user-land now.

For that follow the guide here.

For any suggestions please comment inside this issue.

[barakbs1]

We currently use jest-playwrights' code coverage feature and we depend on it. We would love to migrate to new playwright test-runner but we have to have the code coverage feature, and would really hope it would be integrated. Also it would be amazing if there would be an integration with Kibana-Elastisearch for displaying the results. Moreover, we currently use this % test coverage as a bitbucket-pipeline gate, so an integration with bitbucket pipelines would be amazing!

Keep up the amazing work guys! 🚀 🥇

[mpuertao]

Thanks, i need coverage with istanbul. stay tuned

[mxschmitt]

Small update, we created an example guide how you can use it. It is currently there to get an understanding which additional features are necessary to provide a full story around code coverage collections. For any suggestions / feedback it would be highly appreciated to get input especially:

• If you are using the babel-plugin-istanbul?
• If single browser code coverage like Chromium would be enough for you?
• If you would prefer an approach with minified JS + source maps instead of using the babel plugin? (this would require to have the source maps either locally or available on the web-server)
• What are you doing with the coverage data? text-summary, uploading it to a provider, serving the html, only for manual checking etc.

[mpuertao]

For me:

• If you are using the babel-plugin-istanbul? R:// Yes.
• If single browser code coverage like Chromium would be enough for you? R:// Yes, the coverage is a quantitative aspect, and with a single browser I can determine what components I am covering from the E2E tests.
• What are you doing with the coverage data? text-summary, uploading it to a provider, serving the html, only for manual checking etc. R:// Generate lcov.info o json o html, we are send this data to elasticsearch and after grafana

[anfruiz]

I implemented the baseFixtures.ts in the test folder and I created my test like this:

import { test } from "./baseFixtures";
test('P', async ({ page }) => {
  await page.goto("http://localhost:3000");
  const commandLine = await page.innerText("p");
  test.expect(commandLine).toBe("Edit src/App.js and save to reload.");
});

[mxschmitt]

How did you add the babel-plugin? This requires a transcompilation step which is not out-of-the box available. This looks like create-react-app and needs by that an ejected installation.

[anfruiz]

This requires a transcompilation step which is not out-of-the box available

I only created the .babelrc file in the root project. That's means that I can't apply coverage if I use creat-react-app or ng new my-app?

[mxschmitt]

create-react-app does not use the babelrc file. For that you would need to either eject your create-react-app or use react-app-rewrired

[anfruiz]

Ok thanks Max, for an angular app is the same case or I can create a babelrc file?

[mxschmitt]

I'm not familiar with Angular, depends how/if they allow you to add a custom babel plugin.

[anfruiz]

Hi Max, when I create the baseFixtures.ts in an angular app, it doesn't recognize these 3 dependencies but if I create it in other project the dependencies are ok

[mxschmitt]

hi, this is only a typings warning. It should still be able to work correctly. To get rid of the typing warnings, replacing it with

import * as fs from 'fs';
import * as path from 'path';
import * as crypto from 'crypto';

should fix. Would be awesome if you could verify, then I can adjust it properly.

[anfruiz]

Excellent, this solution fix the warnings!! Thanks!!

[dmail]

According to my tests Istanbul coverage and v8 coverage cannot be merged together. I have opened an issue to track this: https://github.com/istanbuljs/v8-to-istanbul/issues/144. Maybe there is a way but I couldn't make it work.

If confirmed it means coverage collected using Istanbul cannot be merged with coverage returned by page.coverage.stopJSCoverage(). Dropping this here in case you was not aware of this issue.

[scottfwalter]

Excellent, this solution fix the warnings!! Thanks!!

Do you have a sample repo that shows integration with Angular?

[mxschmitt]

Closing since its possible in the user-land, see https://github.com/mxschmitt/playwright-test-coverage

[anishkny]

@mxschmitt It would be great if this were offered as part of Playwright itself via an option instead of in user-land. Thoughts?

[anishkny]

Thanks for reopening. Some points why I think this would be awesome:

• Playwright is positioned as primarily a test framework (which can also do other non-test stuff)
• A vast majority of modern web apps use babel (React, Vue, etc)
• Code coverage is a critical metric for determining test bed health and efficacy
• Playwright makes test related stuff super easy (screenshots, videos, browser downloads) and hides the underlying complexities from the user - this feature seems aligned with that

For now, until this feature is enabled, this can of course be achieved in user-land as shown in the demo or by using the playwright-test-coverage package.

[scottfwalter]

I'm struggling to get babel-plugin-istanbul to work with Angular 12 (or any version). Does anyone have an example on how to instrument with Angular? I attempted to set up extra-webpack-config.js like this:

module.exports = {
    module: {
      rules: [
        {
          // test: /\.m?js$/,
        test: /\.ts$/,

          /**
           * Exclude `node_modules` except the ones that need transpiling for IE11 compatibility.
           * Run `$ npx are-you-es5 check . -r` to get a list of those modules.
           */
          exclude: /[\\/]node_modules[\\/](?!(incompatible-module1|incompatible_module_2|some_other_nested_module)[\\/])/,
          use: {
            loader: 'babel-loader',
            options: {
                "plugins": [
                    ["@babel/plugin-proposal-decorators", {"legacy": true}],
                    "babel-plugin-istanbul"
                ],
                "presets": ["@babel/preset-typescript"]
            }
          }
        }
      ]
    }
  };

However, when I run my app I don't have the COVERAGE variable. I think just need to find out how to get babel properly working Angular.

[bgotink]

I'm in a similar situation as @scottfwalter where I'm not using babel or webpack directly but Angular is responsible for the configuration. I'm not planning on hooking/hacking into angular to add babel-plugin-istanbul, as that might break on every angular update.

So I've gone ahead and created a solution that uses v8's coverage instead: https://github.com/bgotink/playwright-coverage. It's very experimental, more of a proof-of-concept at this stage, but it works on a (pretty large) angular application.

To answer the questions of https://github.com/microsoft/playwright/issues/7030#issuecomment-867174653 for our case:

• We're not using babel. Using babel would require hacking into webpack configuration we do not control ourselves, which we are not prepared to do.
• Only having coverage in chrome is fine for us, as our CI solution currently only runs using chrome anyway due to resource constraints. Even if that were not the case, chromium-only coverage gives more info than no coverage, so I'd still gladly take it
• .
• Text-summary and HTML report for developers, LCOV file for CI and code analysis

[anfruiz]

@bgotink I'm trying to imlplement your solution but when I execute the test it presents the following error:

(node:79038) [DEP0148] DeprecationWarning: Use of deprecated folder mapping "./" in the "exports" field module resolution of the package at .../node_modules/@playwright/test/package.json. Update this package.json to use a subpath pattern like "./*". (Use node --trace-deprecation ... to show where the warning was created) Error: Error while reading @bgotink/playwright-coverage: Cannot find module '.../@bgotink/playwright-coverage'

Can you help me please?

[bgotink]

@anfruiz Could you open a ticket on https://github.com/bgotink/playwright-coverage? I'd rather discuss issues there than continuing in this feature request on the playwright repo, to prevent the discussion here from going off-topic.

[stevez]

Update: It can be easily integrated in the user-land now.

For that follow the guide here.

For any suggestions please comment inside this issue.

I found your repo can't create screen shot, please check the open issue I created in your repo: https://github.com/mxschmitt/playwright-test-coverage/issues/8

[cheapCoder]

@mxschmitt can i use baseFixtures.ts for @playwright/experimental-ct-svelte? I try to extend the context to the test from @playwright/experimental-ct-svelte, and add viteBabel to ctViteConfig in playwright.config.ts, but when using await page.pause() and logging window.__coverage__ in devtool console, i get undefined.

[SgtPooki]

I was able to get the solution mentioned above (https://github.com/microsoft/playwright/issues/7030#issuecomment-867174653) working on a large CRA project after roughly a day of finagling with the various solutions found.

check out the changes at https://github.com/ipfs/ipfs-webui/pull/2014

[sh977218]

I'm struggling to get babel-plugin-istanbul to work with Angular 12 (or any version). Does anyone have an example on how to instrument with Angular? I attempted to set up extra-webpack-config.js like this:

module.exports = {
    module: {
      rules: [
        {
          // test: /\.m?js$/,
        test: /\.ts$/,

          /**
           * Exclude `node_modules` except the ones that need transpiling for IE11 compatibility.
           * Run `$ npx are-you-es5 check . -r` to get a list of those modules.
           */
          exclude: /[\\/]node_modules[\\/](?!(incompatible-module1|incompatible_module_2|some_other_nested_module)[\\/])/,
          use: {
            loader: 'babel-loader',
            options: {
                "plugins": [
                    ["@babel/plugin-proposal-decorators", {"legacy": true}],
                    "babel-plugin-istanbul"
                ],
                "presets": ["@babel/preset-typescript"]
            }
          }
        }
      ]
    }
  };

However, when I run my app I don't have the COVERAGE variable. I think just need to find out how to get babel properly working Angular.

Hi, do you have a working solution to set up Playwright e2e code coverage for an Angular CLI project? If so, can you please share solution?

[CodingProgrammer]

@mxschmitt Hi mx, this coverage tool can only work within the business project ? Can I separate the e2e automation test from business code?

[gauravgandhi1315]

Update: It can be easily integrated in the user-land now.

For that follow the guide here.

For any suggestions please comment inside this issue.

@mxschmitt Do you have baseFixture.ts in js? My framework in JavaScript bases

[dhythm]

Does anyone collect the coverage of playwright component testing with babel-plugin-istanbul?

It doesn't collect the coverage in my local using vite + vite-plugin-istanbul + playwright-ct. (FYI: the normal playwright works well 👍 ) As mentioned in #14511, it would help us if baseFixture.ts supports playwright component testing.

I'm not sure but the following code should be updated because playwright-ct uses mount that wraps page instead of page directly.

[dhythm]

Does anyone collect the coverage of playwright component testing with babel-plugin-istanbul?

It doesn't collect the coverage in my local using vite + vite-plugin-istanbul + playwright-ct. (FYI: the normal playwright works well 👍 ) As mentioned in #14511, it would help us if baseFixture.ts supports playwright component testing.

I'm not sure but the following code should be updated because playwright-ct uses mount that wraps page instead of page directly.

Sorry, the above comment was my failure 🤦‍♂️ baseFixture.ts works well for component-testing too.