Statements detected are not the same in unit-tests context

[gigaga]

Hi,

Currently, I use babel-plugin-istanbul to get code coverage after my E2E tests (selenium). The global variable in my browser window.__coverage__ is correctly filled but for a lot of files, the statements detected are not the same when I execute my unit-tests.

For example, when I generate lcov report for my unit-tests (from nyc mocha), each "import" lines are seen as statement but in my browser in window.__coverage__ variable, these lines are not seen as statement. Consequently, when I try to merge lcov files generated by my unit-tests and by my E2E tests I have a crash (see below)...

C:\Users\me\AppData\Roaming\npm\node_modules\istanbul\lib\object-utils.js:247 for (i = 0; i < retArray.length; i += 1) { ^ TypeError: Cannot read property 'length' of undefined

[bcoe]

@gigaga one thing that jumps out at me is that your using babel-plugin-istanbul with istanbul 1.0.

tldr; I think you should be able to output all the coverage files to a temporary directory, and then use nyc to output the combined report, rather than using Istanbul 1.0 to merge coverage files.

[gigaga]

@bcoe ok why not but even if I find a way to merge the statements are not the same, so the coverage measured is incorrect too. For example, I am able to merge these files into SonarQube and the coverage is wrong because on some files executed from my e2e tests the lines with "import" are marked as not covered (because these statements were not detect for e2e coverage measuring but were detect for unit tests).

[bcoe]

@gigaga it sounds like if I add an nyc merge command, which handled merging reports (so that you didn't need to use istanbul merge) you'd be unblocked?

[gigaga]

I thinks that the merge process is not the problem. My problem is why some statements are not detected by babel-plugin-istanbul whereas these same statements are detected with nyc / mocha (without the plugin)? If statements are the same, there will be no merge problem.

[bcoe]

@gigaga perhaps you could create a repo with a minimum example of babel-plugin-istanbul failing to detect statement coverage. It most certainly should be, so my guess is that it's a bad interaction between the test-runner and the library.

[gigaga]

@bcoe Hi. You can find my fork here : https://github.com/gigaga/babel-plugin-istanbul-issue In this fork, there are 2 ways to compute code coverage from my unit-tests:

• npm run test-babel-plugin-istanbul: where coverage is computed with babel-plugin-istanbul
• npm run test-nyc: where coverage is computed with nyc

The coverages are generated into "results\coverage" folder. You can see that the statements detected are not the same. For example, for nyc, the first line of reducer.jsx is seen as statement where as this same line for babel-plugin-istanbul is not a statement.

[bcoe]

@gigaga thanks for the reproduction hopefully someone will have time to dive into it soon.

[storenth]

@bcoe I have related same issue! Function data Missmatch!

1. My E2E tests instrumented sources with babel-plugin-istanbul and produce acceptace.info lcov from browser global __coverage__.
2. Unit tests produce .lcov coverage with Jest.

The problem that when I am merge that reports: lcov -a acceptace.info -a unit.info -o lcov_merged.info I have next output:

Combining tracefiles.
Reading tracefile acceptace.info
Reading tracefile unit.info
lcov: WARNING: function data mismatch at ../../frontend/src/app/webide/components/sessionEditor/SessionEditor.js:55
lcov: WARNING: function data mismatch at ../../frontend/src/app/webide/components/sessionEditor/SessionEditor.js:40

Because sources instrumented different ways. Unit tests lcov: E2E tests lcov:

[storenth]

Also, I experiment with istanbul-merge and figured out that if I used json reporter and then merge it with that package I get Cannot read property 'decl' of undefined error while try to look inside the final report, but coverage statistic (number of % of coverage) is fine.