coverage not collected from sub-propcesses result from child_process APIs

[osher]

Describe the bug

Consider a package which is a CLI tool. Consider a test suite that does the testing-trophy: it runs e2e with few happy paths, generates coverage, and then write unit-tests to cover only the parts that are not covered by the e2e and component tests. Now consider that the e2e test cases run the CLI using any of the APIs of child_process: exec, spawn, fork, or their sync equivalents.

The tool fails to collect coverage from the e2e suites. The reproduction does only exec, because it is minimal. however, tried all of them.

I'd be delighted if it's just some setting that I'm failing to pass...

reproduction

scenario

1. given a project (full structure below) that uses child_process to run the bin of the package in e2e test of CLI tools, and unit tests to cover code-paths that do not appear in the e2e coverage

2. Run vitest run --coverage

expected

100% coverage. for all files.

Found

All tests run successfully, but only coverage collected using unit-tests is observed.

 RUN  v2.1.8 /home/user/ws/opensource/reproduction/vitest/child_process-not-covered
      Coverage enabled with v8

 ✓ test/calc.unit.mjs (1)
   ✓ calc (1)
     ✓ sub (1)
       ✓ should substract positive integers
 ✓ test/cli.e2e.mjs (2)
   ✓ fxgen command (2)
     ✓ when called with valid parameters (2)
       ✓ should not fail
       ✓ should return the right answer

 Test Files  2 passed (2)
      Tests  3 passed (3)
   Start at  17:13:40
   Duration  291ms (transform 18ms, setup 0ms, collect 16ms, tests 32ms, environment 1ms, prepare 128ms)

 % Coverage report from v8
-----------|---------|----------|---------|---------|-------------------
File       | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
-----------|---------|----------|---------|---------|-------------------
All files  |   14.28 |    33.33 |      25 |   14.28 |
 bin       |       0 |        0 |       0 |       0 |
  cli.mjs  |       0 |        0 |       0 |       0 | 1-3
 lib       |   18.18 |       50 |   33.33 |   18.18 |
  calc.mjs |     100 |      100 |      50 |     100 |
  cli.mjs  |       0 |        0 |       0 |       0 | 1-9
-----------|---------|----------|---------|---------|-------------------

Project structure:

vitest.config.js

import { defineConfig } from 'vitest/config';
export default defineConfig({
  test: {
    globals: true,
    reporters: ['verbose'],
    include: [
      'test/**/*.unit.mjs',
      'test/**/*.e2e.mjs',
    ],
  },
});

package.json

{
  "name": "child_process-not-covered",
  "version": "1.0.0",
  "bin": {
    "calc": "bin/calc.mjs"
  },
  "scripts": {
    "test": "vitest run --coverage"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "description": "",
  "devDependencies": {
    "@vitest/coverage-v8": "^2.1.8",
    "vitest": "^2.1.8"
  }
}

bin/cli

#!/usr/bin/env node
import cli from '../lib/cli.mjs';
cli({ process, console });

lib/cli

import * as calc from './calc.mjs';
export default ({ process, console }) => {
  const { argv: [ ,, op, a, b ] } = process;
  const result = calc[op](Number(a), Number(b));
  console.log(result);
}

lib/calc

export const add = (a, b) => a + b;
export const sub = (a, b) => a - b;

test/calc.unit.mjs

import * as calc from '../lib/calc.mjs';

describe('calc', () => {
  describe('sub', () => {
    it('should substract positive integers', () => {
      expect(calc.sub(44,2)).toEqual(42);
    });
  });
});

test/cli.e2e.mjs

import { execSync } from 'node:child_process';

describe('fxgen command', () => {
  describe('when called with two integers', () => {
    const ctx = {};
    beforeAll(() => {
      try { 
        ctx.result = execSync('node bin/cli.mjs add 20 22').toString().trim();
      } catch(err) {
        ctx.err = err;
      }
    });

    it('should not fail', () => expect(ctx.err).toBeFalsy());
    it('should return their sum', () => expect(ctx.result).toEqual('42'));
  });
});

System Info

$ vitest --version
vitest/2.1.8 linux-x64 node-v20.18.0

Used Package Manager

npm

Validations

• [X] Follow our Code of Conduct
• [X] Read the Contributing Guidelines.
• [X] Read the docs.
• [X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• [X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• [X] The provided reproduction is a minimal reproducible example of the bug.

disclaimer

Feels like de j'avoux. I encountered this before while trying to migrating a moch+nyc project, and that effort turned away from vitest. If I opened such a ticket before - excuse me, for some reason I cannot find it :( I'm now working with a new customer on a new codebase - new chance :)

Anyway, now I really put the effort to reproduce the but in isolation :)

[AriPerkkio]

Vite doesn't intercept node:child_process. Previous discussion is at https://github.com/vitest-dev/vitest/discussions/3851.

Also related to https://github.com/vitest-dev/vitest/issues/4899.

[jpshack-at-palomar]

I filed #7978 to propose a solution that works for my use case (command-line tools which spawn processes) and is similar to some of the other reports I see linked above.

I have found that when I set NODE_V8_COVERAGE, I get good coverage data on spawned process from which c8 can produce good reports. Since Vitest used to use NODE_V8_COVERAGE as its collection mechanism, it seems like we should be able to resurrect some code from a prior version that would handle the case where the NODE_V8_COVERAGE environment variable is set and combine coverage data from these files with the coverage currently collected. Surprising to me is that the raw coverage files produced by node v20.18.2 with NODE_V8_COVERAGE set contained source map data. This may or may not be needed as I haven't yet read how versions of vitest prior to #2786 worked.

While this may not address every use case, it seems it wouldn't be too heavy a lift--I am volunteering--to address this in the v8-coverage provider so there was at least some path forward for vitest users who need spawned process coverage.