threads-plugin

threads-plugin copied to clipboard

Hello. I'm using this plugin and I have small problem. When I use it sometimes just crashes (with 1 worker it's random, it always crashes with 2).

Webpack Config: webpack.config.js

Here is crash report:

> [email protected] start /home/patbox/Pulpit/voxelsrv/client
> cd src/main/ && npx webpack-dev-server

ℹ ｢wds｣: Project is running at http://0.0.0.0:8080/
ℹ ｢wds｣: webpack output is served from /
ℹ ｢wds｣: Content not from webpack is served from /home/patbox/Pulpit/voxelsrv/client/docs
Warning (worker-plugin): output.globalObject is set to "window". It must be set to "self" to support HMR in Workers.

<--- Last few GCs --->
ca[17128:0x3301540]    94766 ms: Mark-sweep 2048.7 (2054.9) -> 2048.4 (2056.2) MB, 1134.6 / 0.0 ms  (+ 128.5 ms in 32 steps since start of marking, biggest step 9.3 ms, walltime since start of marking 1280 ms) (average mu = 0.121, current mu = 0.013) alloca[17128:0x3301540]    96041 ms: Mark-sweep 2050.0 (2056.2) -> 2049.6 (2057.9) MB, 1157.7 / 0.0 ms  (+ 103.5 ms in 25 steps since start of marking, biggest step 9.6 ms, walltime since start of marking 1275 ms) (average mu = 0.069, current mu = 0.011) alloca

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x13a71b9]
    1: StubFrame [pc: 0x1328831]
    2: ConstructFrame [pc: 0x13282cf]
Security context: 0x3500dd0c08d1 <JSObject>
    3: parsePropertyAccessExpressionRest(aka parsePropertyAccessExpressionRest) [0x3a3bea02a349] [/home/patbox/Pulpit/voxelsrv/client/node_modules/typescript/lib/typescript.js:~32289] [pc=0x1c109f6de69a](this=0x0e9f4e8004b1 <undefined>,1645143,0x049964a70431 <TokenObject map = 0x10f40fdc4219>,...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20201127.102801.17128.0.001.json
Node.js report completed
 1: 0x9ef190 node::Abort() [node]
 2: 0x9f13b2 node::OnFatalError(char const*, char const*) [node]
 3: 0xb5da9e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb5de19 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd0a765  [node]
 6: 0xd0adf6 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xd1760a v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xd18515 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xd1afcc v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xce19bb v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
11: 0x10246ce v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
12: 0x13a71b9  [node]
Aborted (core dumped)
npm ERR! code ELIFECYCLE
npm ERR! errno 134
npm ERR! [email protected] start: `cd src/main/ && npx webpack-dev-server`
npm ERR! Exit status 134
npm ERR! 
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/patbox/.npm/_logs/2020-11-27T09_28_01_623Z-debug.log

Hey @Patbox, thanks for reporting!

I am not sure if it's actually the threads-plugin that's crashing, though.

3: parsePropertyAccessExpressionRest(aka parsePropertyAccessExpressionRest) [0x3a3bea02a349] [/home/patbox/Pulpit/voxelsrv/client/node_modules/typescript/lib/typescript.js:~32289] [pc=0x1c109f6de69a](this=0x0e9f4e8004b1 ,1645143,0x049964a70431 <TokenObject map = 0x10f40fdc4219>,...

Looks to me as if typescript is crashing.

Btw, since we are talking about build time crashes… How exactly do you mean "with 1 worker it's random, it always crashes with 2"? I suppose you mean having two different worker entrypoint files (like having a worker1.ts and a worker2.ts), not calling spawn() once/twice as this would only make a difference at runtime.

When I created only one worker from file a, it didn't always crashed (mostly on x rebuilds). After adding worker from file b, it always crashes on first build.

I also have this problem and have reported to Webpack at https://github.com/webpack/webpack/issues/12252

They said to report here

Version: webpack 4.44.2
Time: 12948ms
Built at: 12/22/2020 8:05:07 AM
             Asset      Size  Chunks             Chunk Names
0.bundle.worker.js  5.98 MiB          [emitted]
 + 1 hidden asset
Entrypoint main = bundle.js
[./node_modules/threads-plugin/dist/loader.js?{"name":"0"}!./src/browser.worker.ts] 63 bytes {main} [not cacheable] [built]
    + 272 hidden modules
Child WorkerPluginLoader:
                 Asset      Size  Chunks             Chunk Names
    0.bundle.worker.js  5.98 MiB       0  [emitted]  0
    Entrypoint 0 = 0.bundle.worker.js
    [./node_modules/ts-loader/index.js?!./src/browser.worker.ts] ./node_modules/ts-loader??ref--4!./src/browser.worker.ts 856 bytes {0} [built]
    [./src/actionMap.ts] 11.1 KiB {0} [built]
        + 228 hidden modules
ℹ ｢wdm｣: Compiled successfully.
ℹ ｢wdm｣: Compiling...

<--- Last few GCs --->

[23148:0x102d4e000]   228827 ms: Mark-sweep 2045.8 (2052.4) -> 2045.3 (2052.9) MB, 1445.3 / 0.0 ms  (average mu = 0.049, current mu = 0.006) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x1009ce0f9]
Security context: 0x137f899008d1 <JSObject>
    1: parseDelimitedList(aka parseDelimitedList) [0x137fdca82a29] [/Users/kevzettler/code/webpack-memory-leak/node_modules/typescript/lib/typescript.js:~29145] [pc=0x24a11b769072](this=0x137f7b2404b1 <undefined>,16,0x137fdca832e9 <JSFunction parseParameter (sfi = 0x137f5bb65599)>,0x137f7b2404b1 <undefined>)
    2: parseParametersWorker(aka parseParametersWorke...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x1011bdf85 node::Abort() (.cold.1) [/Users/kevzettler/.nvm/versions/node/v12.18.1/bin/node]
 2: 0x10009d119 node::Abort() [/Users/kevzettler/.nvm/versions/node/v12.18.1/bin/node]
 3: 0x10009d27f node::OnFatalError(char const*, char const*) [/Users/kevzettler/.nvm/versions/node/v12.18.1/bin/node]
 4: 0x1001de7b7 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/kevzettler/.nvm/versions/node/v12.18.1/bin/node]
 5: 0x1001de757 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/kevzettler/.nvm/versions/node/v12.18.1/bin/node]
 6: 0x100364225 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/kevzettler/.nvm/versions/node/v12.18.1/bin/node]
 7: 0x100365a7a v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/Users/kevzettler/.nvm/versions/node/v12.18.1/bin/node]
 8: 0x1003624fe v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/kevzettler/.nvm/versions/node/v12.18.1/bin/node]
 9: 0x1003602b0 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/kevzettler/.nvm/versions/node/v12.18.1/bin/node]
10: 0x10036c0da v8::internal::Heap::AllocateRawWithLightRetry(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/kevzettler/.nvm/versions/node/v12.18.1/bin/node]
11: 0x10036c161 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/kevzettler/.nvm/versions/node/v12.18.1/bin/node]
12: 0x10033a5ea v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/kevzettler/.nvm/versions/node/v12.18.1/bin/node]
13: 0x100689068 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/kevzettler/.nvm/versions/node/v12.18.1/bin/node]
14: 0x1009ce0f9 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/kevzettler/.nvm/versions/node/v12.18.1/bin/node]
Abort trap: 6

I have created a repo that reproduces the issue at: https://github.com/kevzettler/webpack-memory-leak/tree/master

@Patbox any way you could also provide a minimal reproduction repository? sounds like your proj may be simpler that what i've delivered.

@andywer any further thoughts here /\

Not really, unfortunately… This seems like one of those really nasty issues. Especially as it only concerns some users :(

@patbox @kevzettler, are you importing a dependency into your main project files and also into your worker? This led to an infinite loop for us with [email protected]. Specifically we were importing a dependency contained within our monorepo.

I did, as it is somewhat required for me

yes.

@Patbox it's required for us too. The only solution I found was to duplicate the code. Fortunately in my case there was not a lot - but also I haven't implemented workers as much as I'd like because of this issue.

Not sure what your use case is, if you're using the threads rpc interface this suggestion might not work for you. But, I ended up dropping threads.js and upgrading to webpack version 5. Which has built-in support for loading webWorkers and worker_threads in both environments.

@kevzettler does that work in Electron?

I'm not eager to give up on the wonders of threads.js, but I'm at least intrigued.

I just noticed that webpack 5 comes with better web worker support, as you pointed out, @kevzettler: https://webpack.js.org/guides/web-workers/

Maybe we can drop the threads-plugin now… Not sure if it has any effect on this particular issue here, though.

@andywer if dropping the threads-plugin, and configuring to use Webpack's built in capabilities, our code written with threads.js in mind should theoretically run correctly?

In that case would we use import { Worker } from 'worker_threads'; or would it make no difference where we import from?

I'll do a bit more research into using Webpack 5 without the threads plugin. I think it could potentially resolve #37, if not this issue too.

@kevzettler any configuration code you're able to share could be a huge help!

I was able to resolve my issue! Huge thanks to @kevzettler for bringing the feature to my attention.

I had some issues switching away from threads-plugin because Webpack didn't like trying to transpile one of my worker's dependencies. We resolved that by excluding that particular file from our babel-loader, but otherwise the usage is nearly identical. Just change your worker import to emulate the example code: https://github.com/webpack/webpack/tree/master/examples/worker

Awesome stuff. Was the last thing I replied to last night and the first thing I saw today, so didn't have time to get my hands dirty yet.

if dropping the threads-plugin, and configuring to use Webpack's built in capabilities, our code written with threads.js in mind should theoretically run correctly?

Definitely. Seems like you are already there! 🙌

hi @Slapbox @andywer , do we have any updates about this ticket? Wondering how to use webpack 5 new worker feature with threadsjs :) thanks for any advice

@lancety check this out: https://github.com/andywer/threads-plugin/issues/37#issuecomment-784648267

thanks @Slapbox , I tried the solution in #37 but did not work - maybe it is because I am using typescript as well like stephencorwin.
If there is no support to transpile worker ts, I will try use a separate ts task to convert worker ts to js first, then link that transpiled js for any usage of those workers.

We go the same error. Turned out it was a missing semicolon in CSS.

if I remember correctly, there should be a issue in this repository or threads.js repo where I replied about my solution, but I could not find it, here I explain it again for anyone who having same issue:

what I did:

1. remove threads plugin from webpack
2. make separate webpack config for the worker bundle, instead of letting threads plugin and webpack to pack the bundle. you can use multi worker entry if these workers have same pack config
3. since now the 'new Worker' js reference wont be handled by webpack and threads.js, the js path is now pointing to server page root, you need to change the js path relative to server page domain root
4. you can keep using 'Worker' from threads.js or use native 'Worker'

benefits: it fixed the build crash, out of memory problem for me it works in webpack 5 + typescript dev environment by using separate webpack config for worker, the build/pack speed is much faster than using threads-plugin. With --watch under develop mode, single file change can be built within 1 second