Support statement coverage

[avivkeller]

I think the Node.js coverage reporter should include support for reporting statement coverage.

My idea is to parse the source code into an Abstract Syntax Tree (AST) using acorn-walk, and then use the Statement callback to extract statements. They could then be converted to CoverageStatements, (similarly to how the getLines function works with lines to CoverageLines).

These statements could then be mapped to ranges in the same way that lines are handled in mapRangeToLines.

I saw a (extremely complicated) way of doing this in https://github.com/cenfun/monocart-coverage-reports/, so it appears to be possible.

[cjihrig]

I see at least two issues with doing this:

• This will hurt performance of coverage, particularly on large projects.
• We would be parsing the code with a different parser than the one V8 uses. This should almost always be fine, but could have some issues particularly when new syntax is introduced.

Of course there is some code complexity to doing this too. While I agree that statement coverage would be nice to have, I haven't seen any users of the test runner asking for it either. My understanding (which @molow seemed to confirm) is that C8 does not report this metric from V8 coverage. If they aren't doing it, I don't see a great reason for us to do it either.

[cenfun]

The latest version of C8 can do this with --experimental-monocart

c8 --experimental-monocart --reporter=v8 --reporter=console-details node foo.js

As an alternative, we can also use the custom test reporter

node --test-reporter=node-monocart-coverage --test tests

see node-monocart-coverage

Indeed, it is extremely complicated and hurt performance. There is a benchmark for reference, TypeScript is currently using c8 --experimental-monocart to generate coverage reports.

• The TypeScript source code have almost 17M
• It takes almost 19 seconds to generate the coverage reports using the default GitHub Action environment (test job)

Check TypeScript coverage job in CI See https://github.com/microsoft/TypeScript/pull/58850

[AriPerkkio]

If this was implemented directly in V8, whole JS/Node ecosystem could benefit from this. Adding AST based coverage analysis is not ideal for performance.

[github-actions[bot]]

There has been no activity on this feature request for 5 months. To help maintain relevant open issues, please add the https://github.com/nodejs/node/labels/never-stale label or close this issue if it should be closed. If not, the issue will be automatically closed 6 months after the last non-automated comment. For more information on how the project manages feature requests, please consult the feature request management document.

[ljharb]

This is pretty important since basically every other coverage tool in the ecosystem has this 4th metric. Perhaps it would make more sense to see if someone can contribute it to v8 directly? That way all of the valid downsides listed in https://github.com/nodejs/node/issues/54530#issuecomment-2308433263 wouldn't apply.

[cenfun]

The compromise is that the Node test runner can export all the native V8 coverage data, as well as the source code and sourcemap, in custom reports test:coverage event. This should be relatively easy and allow third-party tools to extend.

const { Transform } = require('node:stream');

const customReporter = new Transform({
  writableObjectMode: true,
  transform(event, encoding, callback) {
    switch (event.type) {
      // ...
      case 'test:coverage': {

        const { rawV8CoverageData, sourcesAndSourcemaps } = event.data;

        break;
      }
    }
  },
});

module.exports = customReporter;

[github-actions[bot]]

There has been no activity on this feature request for 5 months. To help maintain relevant open issues, please add the https://github.com/nodejs/node/labels/never-stale label or close this issue if it should be closed. If not, the issue will be automatically closed 6 months after the last non-automated comment. For more information on how the project manages feature requests, please consult the feature request management document.

[github-actions[bot]]

There has been no activity on this feature request and it is being closed. If you feel closing this issue is not the right thing to do, please leave a comment.

For more information on how the project manages feature requests, please consult the feature request management document.

[AriPerkkio]

A year later after my comment in https://github.com/nodejs/node/issues/54530#issuecomment-2322020403:

For Vitest we went ahead and created a library for doing the AST analysis based V8 coverage conversion. For 2-3 years we tried to avoid this but due to V8's limitations (like statement coverage) we needed more accurate approach. Compared to the 3rd party library mentioned in https://github.com/nodejs/node/issues/54530#issue-2484067809, this one is ~96% smaller and focuses only on the coverage convertion.

• https://github.com/vitest-dev/vitest/pull/7736

[pmarchini]

Hey @AriPerkkio , thank you very much for the follow-up and for the suggestion! I'll take a look asap 😁