Coverage information too high for large esbuild produced bundle

[jakebailey]

• Version: v20.9.0
• Platform: Linux JABAILE-DESK02 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

I can only apologize in advance for this one...

I work on TypeScript, and I've been wanting to set up coverage information for the compiler. c8 does a great job; significantly faster than nyc (of course). I use c8 in my own projects with success.

However, capturing coverage for the compiler produces results that are too high to believe. checker.ts is a 50k line file, and yet its coverage is:

checker.ts | 99.99 | 95.38 | 98.62 | 99.99 | 50577-50578

There's no way that we actually have tested every line of the checker. And indeed, adding a function to a known-dead path (like one ending in Debug.assertNever) indicates that the line is covered when it definitely is not, and the overall count doesn't change.

I don't know if I have a good repro for this besides npx c8 npm test -- --no-lint on our repo. Obviously that's huge but I don't really know where to start.

At best, I can observe that npx c8 npm test -- --no-lint --no-bundle works better, i.e. not using esbuild and instead running the tsc output produces a more believable coverage count:

checker.ts  |    97.5 |    95.87 |   98.62 |    97.5 | ...-48684,48686-48687,48761-48762,48 ...

[jakebailey]

One thing that is interesting is that comparing the bundled output HTML page with the unbundled output HTML page, the paths are different. Specifically, the bundled output starts right with compiler/checker.ts, whereas the unbundled output has src/compiler/checker.ts.

[jakebailey]

Yeah, the unbundled output is way better:

[ericmorand]

@jakebailey did you happen to solve the issue? I have the same problem with bundled test suite where, as soon as a portion of a source file is covered, the whole file is marked as covered.

[jakebailey]

No, besides using https://github.com/bcoe/c8/pull/521.

[ericmorand]

Thanks a lot. I tried it and it mostly solve the issue. I'll comment there, with a minimal reproducer repository to, hopefully, help tackle the remainder issues.

[gerardo-lima-moonfare]

Me too I was having 100% coverage when running tests using tsx, which uses esbuild under the hood:

tsx --enable-source-maps --test-reporter=spec --test 'src/**/*.spec.ts'

After some tests, I changed to @swc-node register:

node --import @swc-node/register/esm-register --enable-source-maps --test-reporter=spec --test 'src/**/*.spec.ts'

My preliminary conclusion is: the entry points for the tests were the test files, themselves, and this made any non-tested code to be marked as unreachable by eslint (and removed from the report). This would lead to 100% coverage, by definition.

This was not an exhaustive research, but it makes sense to me and the change produced accurate test coverage reports. I would be interested to understand if other people got different results.