Code Coverage reported as 0% after upgrading from 3.0.1 to 4.0.1

[ek9]

I am experiencing a weird issue where upgrading gulp-mocha from 3.0.1 to 4.0.1 causes code coverage to no longer work. The tests run as before, but istanbul is no longer able to pick things up and it reports 0% code coverage.

This affects this PR https://github.com/ek9/generator-license-cc/pull/25/files

Test coverage before update ( https://travis-ci.org/ek9/generator-license-cc/jobs/195271001 ):

  21 passing (1s)

-----------|----------|----------|----------|----------|----------------|

File       |  % Stmts | % Branch |  % Funcs |  % Lines |Uncovered Lines |

-----------|----------|----------|----------|----------|----------------|

 app/      |      100 |     93.1 |      100 |      100 |                |

  index.js |      100 |     93.1 |      100 |      100 |                |

-----------|----------|----------|----------|----------|----------------|

All files  |      100 |     93.1 |      100 |      100 |                |

-----------|----------|----------|----------|----------|----------------|

Test coverage after update ( https://travis-ci.org/ek9/generator-license-cc/jobs/205308602 ):

  21 passing (950ms)

-----------|----------|----------|----------|----------|----------------|

File       |  % Stmts | % Branch |  % Funcs |  % Lines |Uncovered Lines |

-----------|----------|----------|----------|----------|----------------|

 app/      |        0 |        0 |        0 |        0 |                |

  index.js |        0 |        0 |        0 |        0 |... 294,295,297 |

-----------|----------|----------|----------|----------|----------------|

All files  |        0 |        0 |        0 |        0 |                |

-----------|----------|----------|----------|----------|----------------|

gulpfile.js

'use strict';
var path = require('path');
var gulp = require('gulp');
var eslint = require('gulp-eslint');
var excludeGitignore = require('gulp-exclude-gitignore');
var mocha = require('gulp-mocha');
var istanbul = require('gulp-istanbul');
var nsp = require('gulp-nsp');
var plumber = require('gulp-plumber');
var coveralls = require('gulp-coveralls');

// ... some parts omitted ...

gulp.task('pre-test', function () {
  return gulp.src('app/*.js')
    .pipe(excludeGitignore())
    .pipe(istanbul({
      includeUntested: true
    }))
    .pipe(istanbul.hookRequire());
});

gulp.task('test', ['pre-test'], function (cb) {
  var mochaErr;

  gulp.src('test/**/*.js')
    .pipe(plumber())
    .pipe(mocha({reporter: 'spec'}))
    .on('error', function (err) {
      mochaErr = err;
    })
    .pipe(istanbul.writeReports())
    .on('end', function () {
      cb(mochaErr);
    });
});

For full source you can check the PR or repository https://github.com/ek9/generator-license-cc/pull/25#issuecomment-282495158

[ek9]

I tried to have a look on what is affecting this, but since it was a major rewrite, I don't think I can pin-down where the problem is without further assistance.

[agracio]

Just tested it in my project and coverage is also 0% on 4.0.1

gulpfile.js snippet:

gulp.task('pre-coverage',  function () {
        return gulp.src(paths.out + '/src/progress.js')
            .pipe(istanbul({includeUntested: true}))
            .pipe(istanbul.hookRequire());
});

gulp.task('istanbul', ['pre-coverage'], function () {
        return gulp.src(paths.test, {read: false})
            .pipe(mocha({reporter: 'spec'}))
            .pipe(istanbul.writeReports({
                dir: paths.coverage,
                reporters: [ 'json' ],
                reportOpts: { dir: paths.coverage}
            }));
});

[skarfacegc]

I think the issue is that gulp-mocha is now acting on files rather than on the stream from the istanbul pipeline. I think this was the tradeoff for making compilers work. :/

You can probably use instanbul to instrument your src into a tmp directory and run the tests against those.

[sindresorhus]

It always acted on the file paths, not the files object. Nothing has changed in that regard.

[chris-jones-pixitmedia]

I'm experiencing the same issue after updating

[zozzz]

The problem is that spawning mocha instead use of mocha api. The gulp-istanbul plugin patch require function to import patched sources. If spawn mocha process then the patched require function is now longer available in the new process.

I think a good solution is make a option that switch between spawn and api.

[corvinrok]

Will this ever be solved by a fix in the package?? If not, what are the steps we can take in our own test implementations to work with the new package?

[tinesoft]

Any update on this?

[Enet4]

Sounds unlikely. The discussions suggest that there isn't a trivial fix. Consider using gulp-mocha 3 in the mean time.

[edj-boston]

I would like gulp-mocha to restore compatibility with gulp-istanbul as well.

[jamesplease]

Looping in @shellscape , who was the primary author of the v4 updates. @shellscape , do you have any thoughts on how this could be fixed?

It seems like v4 swapped one big problem with another big problem. From the changelog:

Now spawns the mocha binary instead using its broken programmatic API.

another option would have been to update the mocha programmatic API so as to avoid the ecosystem issues.

---

If someone wanted to fix this, I'd recommend doing two things:

1. Fixing the upstream issue with Mocha
2. Reverting the changes made in v4 of this library

For now, my plan is to stick to v3 of this lib, but I also have a longer-term goal to switch to Jest. It doesn't seem to make much sense to rely on a patchwork of libs and plugins by many different authors to get basic testing functionality. Jest provides it all out of the box, so the n > 5 libs required to get mocha up and running with gulp (with code coverage) is reduced to 2 with Jest (Jest and gulp-jest)

[shellscape]

pong. I'm not really involved here much anymore, but I'm happy to chime in and then disappear in a puff of smoke...

Switching to using the binary was precisely to avoid upstream mocha problems. I can't recall the issue and I haven't the chance to search for the source right now, but the mocha project itself recommended not using the internal API. Unless @sindresorhus has had a change of heart (and I'd doubt it given the number of issues that change resolved) it's unlikely that the module would be reverted to once again use the internal mocha API.

~~You might try running mocha, istanbul, and coveralls via command line to test that it's not something else, somewhere else in your gulp pipeline that's causing the issue.~~ sounds like @zozzz pinpointed the issue with the gulp pipeline. Perhaps the blame lies with gulp-istanbul for making assumptions. You might get more traction with that project on a proper fix.

@jmeas I'm not really sure what the purpose of pushing hard for Jest here is - it comes across in poor taste.

[jamesplease]

Thanks for the response, @shellscape !

but the mocha project itself recommended not using the internal API.

Ah, okay. I didn't know this. My suggestion for how to solve this probably wasn't any good, then.

Perhaps the blame lies with gulp-istanbul for making assumptions. You might get more traction with that project on a proper fix.

This a good suggestion. I looked more into that lib, that it's using the "old" programmatic istanbul API. I'm not too sure, but maybe upgrading it to use the new istanbul-api could help? The creator of gulp-istanbul, @SBoudrias , said he'd be willing to merge in a PR to update to that version if anyone has the time.

@jmeas I'm not really sure what the purpose of pushing hard for Jest here is - it comes across in poor taste.

The purpose was to help other devs get unblocked by this issue. Right now it's not possible to use mocha+istanbul+gulp for coverage without investing a lot of time in the tooling ecosystem, given that many of these libs aren't actively maintained by their creators. I think Jest (or, really, any solution that bundles the integrations for you) is worth consideration for devs running into this specific problem.

There are a bunch of other solutions that might make sense for devs, too. You could just use the nyc CLI directly, and skip the gulp ecosystem. Or maybe the Ava/gulp ecosystem works well, too. Or maybe you would choose to invest time to get all of these mocha/coverage/gulp libs working together. I'm not sure – I'm just throwing some options out there for devs who come to this issue.

Thanks again, @shellscape !

[miparnisari]

Hey guys, what is the recommended way of getting code coverage after running gulp-mocha? gulp-istanbul doesn't work, gulp-cover doesn't either (just tried it and failed)