WebWorker support?

[ebrensi]

Hi I just discovered esbuild, coming from Parcel. One thing I like about parcel is that if I instaniate a WebWorker with the string literal filename, like

const ww = new Worker ('./myWorker.js')
ww.postMessage('work!')

Parcel will recognize the Worker constructor and create another bundle starting at ./myWorker.js. It also handles cache busting for the filename. So the bundle would be called ./myWorker8a68r8912q.js or something and that string would be updated in the code above.

Does ESBuild do something like that? If not, where would I look to implement that?

[evanw]

Oh, interesting. That's pretty cool. I didn't know about that feature of Parcel. I assume this also works for the SharedWorker constructor. You should be able to get this to work with the current version of esbuild by just adding myWorker.js as an additional entry point in the list of entry points passed to esbuild.

However, the cache busting part won't work yet. None of the entry points for esbuild are cache-busted right now. I'm working on enabling this as part of general code splitting support (see #16) but it's still a work in progress. Introducing cache busting in entry point names is also a breaking change and I've been waiting to do this until the next batch of breaking changes. Once that's in, I think it should be relatively straightforward to make something like this work.

[ggoodman]

This sort of feature is sorely needed in Rollup and can be achieved in Webpack with something like worker-plugin.

Having this built into esbuild as an opt-in would be absolutely fantastic! I think deferring to plugins would likely not do it 'right' as these would need to parse an AST, figure out bindings, etc.. (or just RegEx and yolo).

[ggoodman]

There are other web apis that would benefit from this form of non require, non import()-based code splitting like audio worklets.

[rtsao]

I think this would be a really nice feature.

As a stopgap, I was thinking this could be implemented as an esbuild plugin. This would impose some rather unfortunate syntax, but in a pinch, something like this would work:

import workerUrl from "workerUrl(./worker.js)";

const worker = new Worker(workerUrl)

import path from "path";
import { build } from "esbuild";

let workerLoader = (plugin) => {
  plugin.setName("worker-loader");
  plugin.addResolver({ filter: /^workerUrl\((.+)\)/ }, (args) => {
    return { path: args.path, namespace: "workerUrl" };
  });
  plugin.addLoader(
    { filter: /^workerUrl\((.+)\)/, namespace: "workerUrl" },
    async (args) => {
      let match = /^workerUrl\((.+)\)/.exec(args.path),
        workerPath = match[1];
      let outfile = path.join("dist", path.basename(workerPath));
      try {
        // bundle worker entry in a sub-process
        await build({
          entryPoints: [workerPath],
          outfile,
          minify: true,
          bundle: true,
        });

        // return the bundled path
        return { contents: `export default ${JSON.stringify(workerPath)};` };
      } catch (e) {
        // ...
      }
    }
  );
};

[blixt]

Besides path resolution within new Worker(…) etc, it would also be nice to enable an entrypoint to be generated as a worker script, which means it will not use import * as x from "./x.js" but rather importScripts("./x.js") – at least until all browsers implement new Worker(…, { type: "module" }) (https://github.com/whatwg/html/issues/550).

[gkjohnson]

As of version 5.x webpack can bundle web workers out of the box with the following syntax:

new Worker( new URL( "./worker", import.meta.url ), { type: "module" } )

import.meta.url is needed so the worker file is explicitly loaded relative to the current file. Otherwise it's loaded relative to the href of the page. And of course type: module is needed in order to optionally support modules. IMO it's the right way to support bundling Workers that because it promotes practices that work in browsers, as well. Parcel doesn't support this yet but I'm still hoping they will in v2.

This bundler looks great and I'd really look forward to seeing this feature added! It's been too hard to use WebWorkers generically for too long...

[endreymarcell]

Update: I created a minimal demonstration for how this plugin would work at https://github.com/endreymarcell/esbuild-plugin-webworker

~An important difference to Webpack is that this one does not inline the code of the web worker script into the main bundle. Instead, it is built as a separate JS file and has to be shipped next to the main bundle.~

[RReverser]

An important difference to Webpack is that this one does not inline the code of the web worker script into the main bundle. Instead, it is built as a separate JS file and has to be shipped next to the main bundle.

Webpack does not inline it either, it's emitted as a separate file too.

[endreymarcell]

@RReverser oh OK, thanks for the correction. I updated the original comment to not mislead anyone. Also created a little example at https://github.com/endreymarcell/esbuild-plugin-webworker for anyone who might find it useful.

[dy]

Side note: it's possible to (partially) minify worker code with the inline-worker technique:

let worker = new Worker(URL.createObjectURL(new Blob([
  (function(){

  // ...worker code goes here

  }).toString().slice(11,-1) ], { type: "text/javascript" })
));

so as a workaround, workers can be stored in usual js files and included as follows:

// worker.js
export default URL.createObjectURL(new Blob([(function(){
  // ...worker code
}).toString().slice(11,-1)], {type:'text/javascript'}))

// main.js
import workerUrl from './worker.js'
let worker = new Worker(workerUrl)

[evanw]

It works but uses a rather ugly hack to pass both the importer path and the imported file's path to be onLoad handler which makes me think there must be a better way...

The pluginData feature might make this easier.

Side note: it's possible to (partially) minify worker code with the inline-worker technique:

This is fragile and can easily break with esbuild. For example, setting the language target to es6 converts async functions into calls to a helper called __async that implements the required state machine using a generator function. However, that function is defined in the top-level scope and won't be available in the worker. So that code will crash.

As of version 5.x webpack can bundle web workers out of the box with the following syntax:

new Worker( new URL( "./worker", import.meta.url ), { type: "module" } )

This is my current plan to support this feature. This approach is sufficiently general and doesn't need to know anything about web workers. That would make this issue a duplicate of #795. It also looks like Parcel might drop support for the syntax that was originally proposed in this thread: https://github.com/parcel-bundler/parcel/issues/5430#issuecomment-770132484. So that's another count against the original approach.

[RReverser]

That would make this issue a duplicate of #795.

Note that while there is some overlap, those are not really duplicates, because general asset handling and Worker handling need to differ.

In particular, with general asset handling Worker JS file would be treated as raw binary, and wouldn't get minified, the imports wouldn't resolve and so on - which is not what user normally wants. For this reason other bundlers have separate implementation paths for these two features.

[evanw]

I was assuming that the new URL('./some/file.js', import.meta.url) syntax should cause a new JavaScript entry point to be created independent of whether there is new Worker nearby or not. The .js file extension is normally associated with the js loader so it wouldn't be interpreted as binary.

[RReverser]

The .js file extension is normally associated with the js loader so it wouldn't be interpreted as binary.

Fair enough. That's not how it's handled in any of those bundlers AFAIK, but seems to be a sensible route too (probably also needs to include mjs, ts etc. that can result in JavaScript code as well).

[blixt]

new Worker( new URL( "./worker", import.meta.url ), { type: "module" } )

This is my current plan to support this feature.

This would also support the version without { type: "module" } right? So the bundle would use importScripts(…) instead of ES Module import syntax?

While I would love for this to all be ES Modules, neither Firefox nor Safari have any support for ES Module workers, which would make esbuild unsuitable for Worker bundling in most cases if it only emits ES Module-based Worker scripts.

Firefox issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1247687 WebKit issue: https://bugs.webkit.org/show_bug.cgi?id=164860 (but it turns out it was very recently "FIXED" so it might hit technical preview soon! 🎉)

[RReverser]

While I would love for this to all be ES Modules, neither Firefox nor Safari have any support for ES Module workers, which would make esbuild unsuitable for Worker bundling in most cases if it only emits ES Module-based Worker scripts.

Not necessarily, the point of type: "module" is to tell bundler that it contains ESM, so that it can bundle Worker code as a separate entry point. It's not meant to preserve imports in the final bundle anyway.

[blixt]

Not necessarily, the point of type: "module" is to tell bundler that it contains ESM, so that it can bundle Worker code as a separate entry point. It's not meant to preserve imports in the final bundle anyway.

I guess that makes sense, assuming it always bundles down into "old school" Workers. I'd expect the flag to be kept in the output though, as the choice can have pretty big effects on other things, for example <link rel="modulepreload"> tags in the HTML will only work for entrypoints bundled for an ES Module worker, so if the bundler gets to choose freely between the two outputs it wouldn't be possible to set up an HTML file with the right preloading logic, for example.

Basically, an author should be able to always pick "old school" workers for cross-browser support, but also optionally use ES Module workers for Chrome (which has supported this for a long time).

[RReverser]

I'd expect the flag to be kept in the output though, as the choice can have pretty big effects on other things

Usually it has to be removed precisely because it has effect on other things.

E.g. if bundler supports code splitting, then chunk loading usually has to be transformed down to importScripts, which is only available in regular Workers, and not when type:"module" is used. For this reason other bundlers remove this property.

[endreymarcell]

It works but uses a rather ugly hack to pass both the importer path and the imported file's path to be onLoad handler which makes me think there must be a better way...

The pluginData feature might make this easier.

Oh right. Thanks! I updated my implementation at https://github.com/endreymarcell/esbuild-plugin-webworker to use pluginData instead of the hack.

[NullVoxPopuli]

for me, ESM works as a work build target for the most part when combined with 'bundle' except, I need to remove all export keywords. Is there a way to do that?

---

I submitted a PR to https://github.com/microsoft/vscode/pull/123739 so if that's resolved, I don't need a way to strip exports from workers 🙃

[pierre10101]

I managed to resolve this after following the docs at link.

[lgarron]

```js
new Worker( new URL( "./worker", import.meta.url ), { type: "module" } )

This is my current plan to support this feature. This approach is sufficiently general and doesn't need to know anything about web workers. That would make this issue a duplicate of #795. It also looks like Parcel might drop support for the syntax that was originally proposed in this thread: parcel-bundler/parcel#5430 (comment). So that's another count against the original approach.

It seems Firefox and Safari are newly making progress on module workers. Given the hours I've wasted on CJS workarounds, I'd love for esbuild to be ready for this ahead of their releases so we can live in an ESM-only world. And for what it's worth, such behaviour would already be very useful right now, since it already works in node and one major browser (Chrome) — enough to use for local development and try out things ahead of other browsers being ready. 😃

But I more importantly for me, I would like to note that that it would be most useful if this was not specifically tied to the implicit/explicit global Worker constructor. For those of us writing libraries that also work in node, workarounds like like web-worker may be necessary. In that case, call sites may look something like:

new CustomWorkerConstructor( new URL( "./worker", import.meta.url ), { type: "module" } )

(It's possible to assign CustomWorkerConstructor to a variable named Worker in the current scope, so that the AST locally resembles a direct browser worker construction. But that's not always desirable.)

Handling this at the new URL() level instead of the new Worker() level would avoid the issue.

[RReverser]

Another update to this issue is that Emscripten on C++ side and wasm-bindgen-rayon on Rust side both now emit new Worker(new URL('...', import.meta.url)) pattern for Workers internally used for WebAssembly threads, so supporting this syntax would automatically make bundling Wasm easier, too.

[lgarron]

I started trying to figure out how to implement this by mirroring dynamic imports e.g. here and here. I also started trying to detect the new Worker(...) AST pattern at https://github.com/evanw/esbuild/blob/cd5597d0964305a5d910342ac790fc38cbe7f407/internal/js_parser/js_parser.go

I'm not sure I'm going about it the best way either:

isWorkerConstructor := p.lexer.Raw() == "Worker"
if isWorkerConstructor {
  p.lexer.Next()
  if p.lexer.Token == js_lexer.TOpenParen {
    p.lexer.Next()
    if p.lexer.Token == js_lexer.TNew {
      p.lexer.Next()
      if p.lexer.Raw() == "URL" {
        p.lexer.Next()
        if p.lexer.Token == js_lexer.TOpenParen {
          p.lexer.Next()
          if p.lexer.Token == js_lexer.TStringLiteral {
            newURLPathArg := p.lexer.Raw()
            p.lexer.Next()
            if p.lexer.Token == js_lexer.TComma {
              p.lexer.Next()
              if p.lexer.Token == js_lexer.TImport {
                p.lexer.Next()
                if p.lexer.Token == js_lexer.TDot {
                  p.lexer.Next()
                  if p.lexer.Token == js_lexer.TIdentifier && p.lexer.Raw() == "meta" {
                    p.lexer.Next()
                    if p.lexer.Token == js_lexer.TDot {
                      p.lexer.Next()
                      if p.lexer.Token == js_lexer.TIdentifier && p.lexer.Raw() == "url" {
                        p.lexer.Next()
                        if p.lexer.Token == js_lexer.TCloseParen {
                          // ...

More importantly, detecting new Worker(new URL(...), ...) is unfortunately completely inadequate for my use case, because a web worker hosted on a CDN cannot be instantiated cross-origin and must use a trampoline:

<!-- example.com -->
<script src="https://cdn.cubing.net/js/cubing/twisty" type="module" defer></script>

// e.g. https://cdn.cubing.net/js/cubing/twisty
const workerURL = new URL("./twisty-worker.js", import.meta.url);
const importSrc = `import "${workerURL}";`;
const blob = new Blob([importSrc], {
  type: "text/javascript",
});
new Worker(URL.createObjectURL(blob), { type: "module" });

@evanw, do you have any thoughts on how to structure a PR so that this pattern works?

[Waxolunist]

The most important thing currently is to get a minimal version working. Don't expect to many features in the first place. Any syntax is fine for me in a first version. But that missing feature is currently holding us back from using esbuild. One question I do have: Will every worker url be handled like an own entrypoint?

[lgarron]

@evanw, do you have any thoughts on how to structure a PR so that this pattern works?

poke

It looks like the main person working on this for Firefox has been working pretty actively on this, and it seems again that we might have wide support among modern browsers for module workers in the near future.

[lgarron]

So, I've spent a lot of time on figure out workarounds, and if you're all-in on ESM then I think the following is the simplest solution with clear semantics that is handled properly by a variety of tools and works cross-origin.

In particular, it already works in esbuild as-is.

// index.js
const workerURL = (await import("./worker.js")).WORKER_ENTRY_FILE_URL;
const blob = new Blob([`import "${workerURL}";`], { type: "text/javascript" });
new Worker(URL.createObjectURL(blob), { type: "module" });

// worker.js
export const WORKER_ENTRY_FILE_URL = import.meta.url;

(If you know worker.js will always share the same origin with any page it's used on, you can pass workerURL directly into the Worker constructor.)

This works because:

• Bundlers will almost always preserve entry files for dynamic imports.
• There are no hacks to the the URL of the worker, since this is provided by the runtime semantics.

This works directly in the browser, and I've successfully tested it against vite, wmr, and esbuild for a small test site after it was passed through esbuild and published to npm as a library. (Unfortunately, it doesn't work in Parcel but I've asked them to fix it: https://github.com/parcel-bundler/parcel/issues/7623)

One caveat is that this will pull static imports from worker.js into the static dependency graph for index.js. If that code is heavy, you can avoid this by using dynamic imports (which is what we do). This can cause a race condition with tools like comlink, but I found okay workarounds for that as well.

If you want something that works in esbuild today, this approach might work for you. For something better in the future, consider supporting the blank worker or module blocks proposals.

[RReverser]

It works today, but it's pretty unfortunate that it doesn't support the same convention chosen by most other popular bundlers, that also works directly in browsers. It makes it harder for library authors to write syntax that works everywhere.

Did you test this against Webpack and Rollup btw?

[lgarron]

It works today, but it's pretty unfortunate that it doesn't support the same convention chosen by most other popular bundlers, that also works directly in browsers. It makes it harder for library authors to write syntax that works everywhere.

Indeed. I've dug into the esbuild source and offered on this thread to contribute a fix, but unfortunately it isn't getting traction yet. 😔

Did you test this against Webpack and Rollup btw?

No, I've sworn off both of those, but I'd love to know if they work! If someone knows the Webpack/Rollup configs de jour, https://github.com/lgarron/parcel-import-meta-url-bug has a simple test case.

[lgarron]

@evanw What are the chances that esbuild could support new URL("./worker.js", import.meta.url) as in this?

// index.js
const workerURL = new URL("./worker.js", import.meta.url);
const blob = new Blob([`import "${workerURL}";`], { type: "text/javascript" });
new Worker(URL.createObjectURL(blob), { type: "module" });

// worker.js
// ...

I'd be glad to contribute a lot of time to such functionality, if I know that it's an appropriate direction for esbuild and where to start.

I'm in the unfortunate situation where it's impossible for me to publish our library without breaking functionality with some bundlers. I've just learned that apparently all other major bundlers support the relative path resolution of other source entry files like in the snippet above.

For the time being, I'm sticking with the approach that works with esbuild (and still a significant amount of other bundlers), but it sounds like this would finally allow us to support shipping a library that is performant and compatible for module workers.

[lgarron]

I've just learned that apparently all other major bundlers support the relative path resolution of other source entry files like in the snippet above.

Just to expand on this, @devongovett helpfully pulled together some docs links at https://github.com/parcel-bundler/parcel/issues/7623#issuecomment-1025385072 that show that this has gained a lot of support:

This works natively and in most bundlers (e.g. Webpack, Vite, Parcel).

[RReverser]

Just to expand on this, @devongovett helpfully pulled together some docs links at parcel-bundler/parcel#7623 (comment) that show that this has gained a lot of support

Yeah that's what I was referring to when talking about other bundlers. I wrote an article on that pattern as de-facto convention on web.dev a few months ago: https://web.dev/bundling-non-js-resources/

FWIW in your last snippet the Blob URL is not required either.

[lgarron]

FWIW in your last snippet the Blob URL is not required either.

How do you mean? I don't know of anything else (or at least anything simpler) that works cross-origin.

[RReverser]

Ah yeah, for cross-origin it probably is helpful. I was referring to a more general pattern.

[DanielHeath]

Perhaps I have misread, but it seems to me that if support for treating new URL( "./worker", import.meta.url ) as another form of import that creates a separate entrypoint were implemented, the rest would fall into place.

[mbrevda]

@lgarron seems your method, while great, will cause the worker to be executed in the context of the page, and hence its onmessage handler will receive any message sent via window.postMesage.

If anyone is trying to introduce workers using this method, at the very least your going to want to guard the onmessage handler with something like this:

// ensure we are not in the browser context
if (typeof window === 'undefined') {
  onmessage = async ({data}) => {
    // do something here
  }
}

new URL support would definitely be welcome!

[lgarron]

@lgarron seems your method, while great, will cause the worker to be executed in the context of the page, and hence its onmessage handler will receive any message sent via window.postMesage.

If anyone is trying to introduce workers using this method, at the very least your going to want to guard the onmessage handler with something like this:

Yeah, there are a few ways to do this. We do something more careful and have a shared import that we use to signal to the worker module when it is supposed to expose the API or not:

// worker-guard.ts
export const exposeAPI: { expose: boolean } = {
  expose: true,
};

// worker-entry.ts
import { exposeAPI } from "./worker-guard";

if (exposeAPI.expose) { (async () => { /* ... */ })(); }
export const WORKER_ENTRY_FILE_URL = import.meta.url;

// app code
import { exposeAPI } from "./worker-guard";

exposeAPI.expose = false;
const workerURL = (await import("./worker-entry")).WORKER_ENTRY_FILE_URL;

// simplified cross-origin instantiation code
const blob = new Blob([`import "${workerURL}";`], { type: "text/javascript" });
new Worker(URL.createObjectURL(blob), { type: "module" })

This keeps the logic entirely self-contained within our library. If you are writing an app where you control everything, though, using window or WorkerGlobalScope detection is probably fine, though.

[mhsdesign]

Vite seems to handle it like: import EditorWorker from 'monaco-editor/esm/vs/editor/editor.worker.js?worker'; one could make an easy plugin for this...

but I generally like that esbuild doesn't offer to much magic. So I'm okay with creating another build manually with multiple entry points too.

[jasikpark]

Vite seems to handle it like: import EditorWorker from 'monaco-editor/esm/vs/editor/editor.worker.js?worker'; one could make an easy plugin for this...

but I generally like that esbuild doesn't offer to much magic. So I'm okay with creating another build manually with multiple entry points too.

I wonder what vite is doing since they depend on esbuild as a dep 🤔

[itibbers]

Side note: it's possible to (partially) minify worker code with the inline-worker technique:

let worker = new Worker(URL.createObjectURL(new Blob([
  (function(){

  // ...worker code goes here

  }).toString().slice(11,-1) ], { type: "text/javascript" })
));

so as a workaround, workers can be stored in usual js files and included as follows:

// worker.js
export default URL.createObjectURL(new Blob([(function(){
  // ...worker code
}).toString().slice(11,-1)], {type:'text/javascript'}))

// main.js
import workerUrl from './worker.js'
let worker = new Worker(workerUrl)

template strings works for me.

let worker = new Worker(URL.createObjectURL(new Blob([`
 onmessage = (event) => console.log(event)
 `], { type: "text/javascript" })
));