node
node copied to clipboard
nodejs
Support Web Workers
What is the problem this feature will solve?
Creating cross-platform (Node.js + browsers) code has never been more important, but there are still some sharp edges. fetch support was recently added, but there's another important and popular API; Web Workers. Node.js does have worker_threads, but the API differs in many ways and it's really difficult to properly bridge them. There are attempts at bridging these APIs in user-land, but the most popular one is incomplete and not actively maintained.
What is the feature you are proposing to solve the problem?
I propose adding support for Web Workers in Node.js. The Web Workers API is essential to keep apps and servers responsive by moving CPU heavy work off the main thread. I strongly feel it should be part of Node.js.
What alternatives have you considered?
Continue using one of the available polyfills, but that means larger dependency trees, more bugs, and more workaround code.
Is it something you'd be ready to implement this yourself?
/cc @nodejs/workers
The linked module appears to mostly be adding things node workers already support (web events, data uris). Looking at MDN the only thing that really jumps out to me is type: classic/module and name.
The linked module appears to mostly be adding things node workers already support (web events, data uris). Looking at MDN the only thing that really jumps out to me is
type: classic/moduleandname.
Also the web version only supports URLs (strings) in the constructor, while the Node.js one supports paths and URL objects.
IIRC, when @addaleax implemented worker_threads, she modeled the API on Web Workers but it was not possible to support the entire API, so it diverged in places. I'm not sure if the specifics are documented anywhere or if anything has changed.
@sindresorhus Any chance you have specific API pain points you'd highlight? If we can't do everything but we can do some things, it would be good to know what is (at least in your view) the high-priority stuff.
but the most popular one is incomplete and not actively maintained.
That’s arguably a sign against inclusion in Node.js core, unless you have a reason to believe that development as part of Node.js would improve this situation (isn’t obvious to me why that would be).
she modeled the API on Web Workers but it was not possible to support the entire API, so it diverged in places
The goal was to implement an API that matches Node.js’s abilities and requirements. You can probably support all or almost all of the Web API if you try hard enough, and the Node.js API is certainly inspired by the Web API on the parent thread side. On the child thread side, you are just running code in a completely different environment to begin with.
That’s arguably a sign against inclusion in Node.js core, unless you have a reason to believe that development as part of Node.js would improve this situation (isn’t obvious to me why that would be).
Definitely have to agree with this... It could also be a sign that worker_threads are "close enough" to discourage further effort there.
I'd certainly be open to PRs that move the current worker_threads implementation closer to the standard alignment. I don't think we really need to do a new implementation of anything, just incremental changes here and there.
tl;dr ... PRs welcome ;-)
IIRC, when @addaleax implemented worker_threads, she modeled the API on Web Workers but it was not possible to support the entire API, so it diverged in places.
I'm curious why it was not possible to support the entire API? And what parts?
That’s arguably a sign against inclusion in Node.js core, unless you have a reason to believe that development as part of Node.js would improve this situation (isn’t obvious to me why that would be).
The maintainer released something they needed and got busy. I don't think that's evidence for whether or not it would prosper as a part of Node.js. The repo does have a lot of pull requests, which suggests people would be willing to help improve it.
It could also be a sign that worker_threads are "close enough" to discourage further effort there.
Why add fetch then? http.get is close enough. Why add Web Streams? Node streams are close enough. There are huge benefits for Node.js and browser sharing some APIs. Familiarity. Code sharing. Less dependencies.
Deno supports Web Workers.
I'd certainly be open to PRs that move the current
worker_threadsimplementation closer to the standard alignment.
Moving worker_threads closer to the Web Workers API would be a welcome change, but Node.js should still have a spec compliant Web Worker API. worker_threads cannot ever be fully spec compliant.
Why add fetch then? http.get is close enough. .
No, I meant that it could explain why the userland module wasn't advanced further, not offering any kind of reason why we wouldn't continue to make improvements in core. I'm all for that, just need someone to volunteer to do the work
I don't think anyone is against improving this, we're more trying to figure out what specifically you are looking for in the worker api. Like what prompted you to open this issue? That information can help us triage and understand what scope of work is needed.
WebWorker is probably the single most important Web API that I miss not having in Node. The use case for me is just wanting to write code once that works everywhere, if possible.
worker_threads exists, and it works, but it just doesn't allow me to write code once that works both in the browser and in Node, not without more code to abstract the differences anyway.
Somewhat interestingly worker_threads is not usable in Electron's renderer thread with Node integration enabled, it's just disabled for whatever reason, so potentially one would have to support both WebWorker and worker_threads even when running code only in environments where Node is available, but I guess Electron's renderer threads with Node integration enabled are a bit niche.
The web-worker package is good for shimming out the differences between Node and the Browser but I think it doesn't really support bundling. I have written my own little shim that supports bundling (https://github.com/fabiospampinato/webworker-shim), but it only supports loading a base64-encoded module and really I only wanted to write my original module in an isomorphic way, in order to do that I had to write this thing and now I have to maintain it, ideally I should just delete this because Node would support WebWorker.
Whether WebWorker should be implemented or not seems like a no-brainer to me, whether this is the right time to implement it or whether there are more important problems to allocate resources to I don't know.
My use-case: I try to make my packages work in both Node.js and browser whenever possible. Sometimes I would like to offload CPU heavy work to a worker (important to keep servers and apps responsive). However, because it's such a pain to deal with the differences of worker_threads and Web Worker APIs, I have generally just not done it, which is a shame. I did end up doing it in my latest package which prompted this issue.
I love Node.js, but the constant differences in essential APIs is a daily pain-point. Mostly Buffer, crypto (being fixed), and workers.
This is basically my Node.js wishlist and Node.js is getting closer every year: https://deno.land/manual/runtime/web_platform_apis#web-platform-apis
I'm certain Node.js will have a spec compliant Web Worker API at some point (whether it's now or in 10 years). Same as I was certain Node.js would get fetch at some point. It just makes sense. My hope is for this issue to attract some interest from the core team and community so that it can happen sooner rather than later.
I love Node.js, but the constant differences in essential APIs is a daily pain-point. Mostly Buffer, crypto, and workers.
I think Node now has a WebCrypto implementation. Personally I just load those APIs via this trivial module and have code that works both in the browser and in Node trivially. I'd love to do the same, or better, for WebWorker and other Web APIs.
FWIW I think generally if an API is supported both by the browser and by Deno the question should be flipped: why shouldn't Node also implement it?
Maybe start with a simple question: What does the global scope inside a Web Worker spawned by Node.js look like? Does it match WorkerGlobalScope? Does it match a regular Node.js global scope?
I used for node and browsers this package: https://www.npmjs.com/package/threads
Works pretty cool 👌🏻
Maybe start with a simple question: What does the global scope inside a Web Worker spawned by Node.js look like? Does it match WorkerGlobalScope? Does it match a regular Node.js global scope?
I think it should match WorkerGlobalScope as close as possible.
If you need to access any Node.js-specific stuff, you can just use import.
To make it easier to implement this, I would personally be fine with it only supporting module-type workers: new Worker('x.js', {type: 'module'});
I used for node and browsers this package: https://www.npmjs.com/package/threads
This package would benefit from Node.js supporting Web Workers as it would not need separate code paths for Node.js and browsers.
Definitely have to agree with this... It could also be a sign that
worker_threadsare "close enough" to discourage further effort there.
Speaking from my own experience, the fact that worker_threads is different enough from Worker often makes me decide just not to bother with threads at all in code (even though it could often benefit) that is polymorphic for Node/Browsers.
Although I will say, even if Node did provide a web-compatible Worker, the whole Worker API leaves a lot to be desired due to how much boilerplate the event/messagechannel API requires to do relatively simple things. It would make worker wrappers easier though if Worker was global and had a more compatible surface though (particularly a common way to actually listen to events).
@bnoordhuis @cjihrig As folks who have worked for Deno Land Inc. and therefore are presumably familiar with challenges and benefits of supporting Web Workers in a JavaScript runtime, do you have any recommendations, cautions, enthusiastic endorsements, etc.?
I wasn't around when Web Workers were implemented in Deno, nor have I maintained them at all, so I don't have anything to add in that regard.
-
A plain Web Worker environment offers rather limited functionality. I think it's telling deno decided to expose the (obviously non-standard)
Denoobject to provide a better out-of-the-box experience. -
Deno goes through some unnatural contortions (by faking up a
window.location) to make imports sorta-kinda work like in the browser. I don't know how node would tackle that. -
There's a long tail of lesser issues where node is subtly incompatible with the Web Worker standard. Stuff like timers, rejection handling, error reporting, etc. Nothing that can't be fixed but whether the effort and complexity pay off? 🤷
If the Web Worker specification is hard to adapt to non-browser JS, then maybe a simpler subset of the Web Worker API could be designed and proposed to browsers as a simpler building block? (Skipping the .location and similar)
There is also other work going on when it comes to workers, like @surma's JS Module Blocks proposal, which I raised this issue in to discuss ways it which it could maybe be usable in non-browser contexts: https://github.com/tc39/proposal-js-module-blocks/issues/61
Having a common way of running such Module Blocks would of course be helpful then.
A plain Web Worker environment offers rather limited functionality. I think it's telling deno decided to expose the (obviously non-standard) Deno object to provide a better out-of-the-box experience.
Which is why I suggested that Node.js core modules could be importable (but not globals).
Deno goes through some unnatural contortions (by faking up a window.location) to make imports sorta-kinda work like in the browser. I don't know how node would tackle that.
I don't see any reasons to support that. You can just do the following (as the Deno docs also point out):
new Worker(new URL("./worker.js", import.meta.url).href, { type: "module" });
There's a long tail of lesser issues where node is subtly incompatible with the Web Worker standard. Stuff like timers, rejection handling, error reporting, etc. Nothing that can't be fixed but whether the effort and complexity pay off?
Ideally, these things should be fixed regardless as it will benefit any kind of Node.js-browser shared code, not just Web Workers usage.
I don't see any reasons to support that.
We're talking about different things. My comment is on how to handle imports inside the script.
Think import "foo" - where is foo loaded from? Node and browsers (and deno) give different answers.