istanbuljs
Files required through `node --eval` or `node --require` do not get instrumented
Expected Behavior
Files required through node --eval or node --require should be instrumented.
Observed Behavior
They do not get instrumented.
The demo repository contains two empty files index.js and other.js.
Running the file instrument it correctly:
$ nyc node index.js
----------|----------|----------|----------|----------|-------------------|
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s |
----------|----------|----------|----------|----------|-------------------|
All files | 0 | 0 | 0 | 0 | |
index.js | 0 | 0 | 0 | 0 | |
----------|----------|----------|----------|----------|-------------------|
Requiring it through --eval does not instrument it:
$ nyc node -e 'require("./index")'
----------|----------|----------|----------|----------|-------------------|
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s |
----------|----------|----------|----------|----------|-------------------|
All files | 0 | 0 | 0 | 0 | |
----------|----------|----------|----------|----------|-------------------|
Requiring it through --require does not instrument it:
$ nyc node -r ./index.js other.js
----------|----------|----------|----------|----------|-------------------|
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s |
----------|----------|----------|----------|----------|-------------------|
All files | 0 | 0 | 0 | 0 | |
other.js | 0 | 0 | 0 | 0 | |
----------|----------|----------|----------|----------|-------------------|
Forensic Information
Operating System: Ubuntu 18.04
Environment Information: node v11.3.0, npm 6.4.1, nyc 13.1.0
Notes
nyc has a --require flag. However this only works when node is fired as the top-level command, not as a child process.
For example I am testing a library that has a register.js file that performs monkey patching on load (like @babel/register does). The only way to test it without polluting the global environment of the other tests is to run it as a child process. I.e. my unit test runs childProcess.spawn() using --require register.js.
I can work around the problem and avoid --eval or --require by loading a file that fires require() instead, but there might be other use cases that this limitation of nyc might impact.
I can answer part of this bug, it's not possible for files required via --require to be automatically instrumented. To get this working you must pre-instrument the file to be included via --require, along with any sources it require's (if you want those files to be covered). The --require options are processed before nyc can hook anything, partly due to startup order and partly due to the fact that --require options can actually be used to perform the instrumentation (see ES2015 tutorial for example).
The way that I know will work is if you have a module where sources are in ./lib/*.js and you want to test node --require ./lib/register.js, you would have to use the nyc instrument command to build an instrumented copy of the lib folder. Then you would tests with node --require ./instrumented/register.js.
As for node --eval, I'm not sure if it's possible to hook this. Like --require this is run very early in the process of starting a node process, before nyc can do anything. Again I think the only way to accomplish coverage testing of stuff loaded so early is to pre-instrument it.
I've marked this issue so the stale bot will not touch it anymore, but understand that no code fix is planned. The intent is to create a tutorial at istanbul.js.org for how to perform coverage testing of a --require script.
I understand, thanks for explaining the reason behind this. I do think this is a limitation in the current hooking system that should be documented.
A workaround I am doing which does not require an extra build step is to add a require() at the beginning of the first file to be executed. This only works if you can modify that file though (which is my case).
Another thing: I have seen that there has been some work in Istanbul towards offering NODE_V8_COVERAGE as an alternative for instrumentation. This still seems too early to use for me (no way to disable coverage through comments, and implementations still a little unstable from my own experience). But in the long run, would this solve that problem?
Native V8 coverage can be processed by c8 in place of nyc, it only uses istanbul to generate reports. From what I can tell it doesn't yet support any kind of comment hints for disabling coverage. I'd assume that native v8 coverage would pick up on node --require ./something.js, but I've never tried.
Yes exactly.
Thanks for your help, I am going to close this issue as this probably won't be fixed. However thanks for checking it out.
I am also going to delete the demo repository, please let me know if that is a problem.
@ehmicky thanks for your patience but I have transferred this issue to our website repo so that it exists in the right place as a need to be documented. Feel free to unsubscribe from the thread if you don't want notifications on this any more. I have also forked your demo repository, so it should be fine to delete... though it doesn't do much harm to keep. Either way, thanks so much for that!
