[Bug] virtual module trigger extra compilation in webpack 5

[michenly]

• [x] I'd be willing to submit the fix

Describe the bug

Using the latest webpack 5 and the latest webpack-virtual-modules. I notice that webpack is logging out 2 compilation in watch mode.

To Reproduce

Run webpack --watch to reproduce using this brranch: https://github.com/michenly/webpack-5-demo/tree/webpack-virtual-modules-error

Minimally, use webpack-virtual-modules (I did this in both in a plugin with .apply and in webpack config's plugin directly with the same result)

new VirtualModulesPlugin({'test.js': `export default 'test123'`})

Log out compilation message by tapping into compiler.hooks.compile and see two compilation message as soon as webpack --watch is run

Screenshots

Environment if relevant (please complete the following information):

• OS: OSX 11.15.1
• Node version: v12.14.0
• Mochapack version: ?
• Webpack version: 5.50.0
• webpack-virtual-modules version: 0.4.3

Additional context

I also log out complier.removedFiles in compiler.hooks.watchRun, notice how the virtual module is listed as files that was removed.

[andy0130tw]

Took several days to bisect the problem. The problem lies in the fact that WatchFileSystem will scan the filesystem after a compilation has finished, and if it finds any differences, it triggers a rebuild. This scan obviously does not go through the cached input file system, so it reports that the virtual modules "missing in initial scan" (sample). It boils down to the following steps:

1. Watchpack goes through all "watchers" it keeps (1, 2), in our case, this eventually calls _onRemove method on virtual files.
2. Collected changes are buffered and emitted with an "aggregated" event.
3. The key is that watchFileSystem always calls the callback upon receiving the event, effectively triggers another watch cycle (with virtual modules purged from inputFileSystem so the virtual module plugin adds it back).

My idea to deal with this is to steal the watchFileSystem and roll a new one that filters out virtual files before proceeding. If none of the removals come from actual files, then we can safely ignore this event.

This involves precisely mimicing what Watchpack does (which slightly varies across versions of Webpack). The drawback is that we definitely need separate implementations for Webpack 3, 4, and 5, and the way is not future-proof.

Forgive me that writing clean TS is time-consuming for me. Moreover, under Webpack 5.64.1, when logging things down, I see some nonsense removals (like node_modules/babel-loader.js/package.json) that still trigger the rebuilds 2~3 times, possibly related to buggy cache, but I am running out of time budgets on studying this further. I will be grateful if someone can turn this idea into a PR.

Update: Did some quick tests, my fork works for me 5.60~5.64.0, and 5.64.1 with --no-cache.

[andy0130tw]

Update: I have worked out a more robust, Proxy-based solution that treats the original WatchFileSystem as an abstract object. Should be more resilient to breaking changes in Webpack and Watchpack.

Code: https://github.com/andy0130tw/webpack-virtual-modules/blob/24d823f3133da9f397224cdfedd39be341546b21/src/wfs.ts

[larixer]

@andy0130tw Awesome, do you still have no bandwidth for the pull request?

[andy0130tw]

@larixer Sure. I have opened a PR.

[larixer]

Closing the issue as stale, please retest on latest webpack-virtual-modules and reopen with reproduction steps if the issue still exists.