URL assets bundling

[edoardocavazza]

Hello!

Is there any plan to support assets bundling using the URL constructor and import.meta?

Something like this:

const myImg = new URL('./assets/my-img.png', import.meta.url);

would be equivalent to:

import myImg from './assets/my-img.png';

but with native browser support.

This is how we can do that with rollup https://modern-web.dev/docs/building/rollup-plugin-import-meta-assets/. I think it could fit with the "convention over configuration" policy of esbuild.

[ggoodman]

@edoardocavazza you might want to take a look at #312.

[evanw]

Does any bundler currently do this? What would be the semantics specifically? Would this require the loader to be dataurl or file? The syntax import myImg currently returns a string, not a URL object. Would this be equivalent to import string from './assets/my-img.png'; const myImg = new URL(string, import.meta.url) instead?

[ggoodman]

Not the OP, but I'd love to see something like Webpack@5 Asset Modules. The specific behaviour that I think would be a great fit is the following:

1. When in a JavaScript file, a call to the global URL constructor whose first argument can be evaluated to a path at build time (I'd be fine w/ only relative paths) and whose second argument is a reference to import.meta.url will trigger a chunk boundary in the dependency graph similar to an import() expression.
2. This new edge will have a new type and would run through onResolve plugins. Plugins could do custom resolution if they want, or set a custom loader. The default behaviour would be to select the same loader as if the resolved path were an entrypoint.
3. The edge will also go through onLoad plugins.
4. The new URL('./relative/path', import.meta.url) expression will be replaced with a constant string as if it were an asset reference. This behaviour would then be consistent with how the same code would behave in a modern browser.
5. The resulting chunk would be treated like another entrypoint or async chunk.

Benefits:

• Parity with how the same code behaves in the browser
• Extensible across use-cases for static assets, Workers, ServiceWorkers, SharedWorkers, Worklets, ...etc.
• Clean integration with existing plugin hooks

Really excited to see where you land on this @evanw.

[evanw]

The new URL('./relative/path', import.meta.url) expression will be replaced with a constant string as if it were an asset reference.

It seems weird to me that the bundler would replace an object with a string. Then e.g. new URL(...).pathname would unexpectedly be undefined.

[ggoodman]

Excellent point. That doesn't work at all. I wonder if the same behaviour, but without the substitution would work?

Edit: or perhaps only the first argument of the URL constructor could be adjusted with the resulting chunk path.

[lifaon74]

Actually, on my side, I would be really happy to have a way to retrieve the 'pre-bundled' import.meta.url as it is currently really useful to load ressources relative to our source code.

[rdmurphy]

Adding a +1 to this request, but also wanted to share an example of how the Rollup plugin does this that may address your .pathname concern @evanw.

https://modern-web.dev/docs/building/rollup-plugin-import-meta-assets/

If you look at the plugin itself it's incredibly simple and leans on Rollup's own asset output functionality, but I think the key thing it does is return a modified new URL call itself. That way it still works as written as an instance of URL and will correctly chain any further property calls. We've been using this in our Rollup-powered build system and it has been great.

(FWIW I do think this is how webpack 5 does this as well — it returns a modified URL, not a string.)

I was actually gonna submit a separate issue on whether esbuild can interact with new URL calls via plugins to see if this was implementable as a plugin. 😅

[rdmurphy]

I now see that Rollup plugin was what was linked in the original issue. 😶 Sorry @edoardocavazza!

[ggoodman]

Excellent point. That doesn't work at all. I wonder if the same behaviour, but without the substitution would work?

Edit: or perhaps only the first argument of the URL constructor could be adjusted with the resulting chunk path.

@evanw do you think this might offer a viable path forward for introducing 'arbitrary' code split boundaries?

[evanw]

@evanw do you think this might offer a viable path forward for introducing 'arbitrary' code split boundaries?

It seems similar to two existing features in esbuild. One of those is the file loader which copies the file into the bundle directory and returns a relative path to the file. The other one is a dynamic import() expression which (when code splitting is enabled) creates a new entry point and becomes code that loads that entry point and returns a promise. This feature sort of seems like the combination of both: create a new entry point but just return the relative path to it instead of starting to load it. I'm not sure if I'd say that it offers a viable path forward for introducing 'arbitrary' code split boundaries since it would behave similarly to import(), which already exists. Maybe I don't understand exactly what you're saying. But I do think it's a viable feature, especially with other bundlers converging on common behavior for this convention.

[ggoodman]

I think we're thinking the same thing then but perhaps I haven't done a good job articulating it. I don't think 'arbitrary' was a good word choice.

Calling out the parallels to the file loader and dynamic import() is exactly what I had in mind though. I think the feature belongs in esbuild where access to the AST and the ability to reliably identify appropriate new URL() expressions would be more efficient. I think that the feature, as I envision it, could be implemented in user-space today by either a hacky regex or full AST traversal to find new URL('./path', import.meta.url) expressions. It would require some trickery that may not even be possible to get it all wired into the main asset graph.

I'm looking forward to being able to use something like this for frictionless web-worker bundling.

[lgarron]

This feature sort of seems like the combination of both: create a new entry point but just return the relative path to it instead of starting to load it. I'm not sure if I'd say that it offers a viable path forward for introducing 'arbitrary' code split boundaries since it would behave similarly to import(), which already exists. Maybe I don't understand exactly what you're saying. But I do think it's a viable feature, especially with other bundlers converging on common behavior for this convention.

This would be invaluable for web workers (#312), and would personally save me a lot of debugging time and pain.

At this point, it's impossible to use esbuild to publish a library that uses web workers successfully when passed through Parcel or Webpack. I know this can be solved to some extent using plugins, but I really want our libraries to work everywhere without caveats. It sounds like including new URL("./relative-source-file", import.meta.url); in the module graph as an entry file is the most compatible way forward.

As someone familiar with JS and Go but who hasn't contributed, what could someone do to help move this forward?

[lgarron]

As someone familiar with JS and Go but who hasn't contributed, what could someone do to help move this forward?

Okay, matching the appropriate syntax in the AST wasn't too bad: https://github.com/evanw/esbuild/compare/master...lgarron:new-url-entry

I'm having some trouble adding it to the import graph, as I don't fully understand EImportCall and ImportRecord. What I'm currently trying:

• Add ERelativeURL, patterned after EImportCall.
• Reuse ImportRecord for this (at least to get things working).
• Add an ImportKind of RelativeURL.

https://github.com/evanw/esbuild/compare/master...lgarron:new-url-entry-import-graph is as far as I've gotten so far.

[mbrevda]

@evanw - any guess when something like this might land? Really want it for #312!

[firien]

based on @lgarron work (as I have zero Golang experience) here is a demo of:

const workerEntryFileURL = new URL("./worker.test.js", import.meta.url);
new Worker(workerEntryFileURL, { type: "module" });

being converted into:

var workerEntryFileURL = new URL("./worker.test-P7GTNY24.js");
new Worker(workerEntryFileURL, { type: "module" });

The entry is handled exactly like a dynamic import (await import("../path/to.js")) - as mentioned by @evanw above (https://github.com/evanw/esbuild/issues/795#issuecomment-786065290) - it is only added as a new entry with splitting option

https://github.com/evanw/esbuild/compare/main...firien:relative-url

[lgarron]

based on @lgarron work (as I have zero Golang experience) here is a demo of:

😍

I was able to use this locally, with a few issues:

• It doesn't actually specifically check for .js or .ts files, so other new URL() calls cause errors. I was able to fix this with if strings.HasSuffix(helpers.UTF16ToString(s.Value), ".js") || strings.HasSuffix(helpers.UTF16ToString(s.Value), ".ts") in the relativeJS calculation. -esbuild seems to build successfully for me, but it hangs after building (whether called from CLI for building / CLI for serving / JS for either). Not sure why.
• ~The file name was rewritten in the build, but it doesn't have the correct relative path when the import happens in a subfolrder. For example, importing new URL("./b.js", import.meta.url) from inside ./nested/in/a/folder/a.js should sometimes give new URL("../../../../b-[HASH].js", import.meta.url) rather than new URL("./b-[HASH].js", import.meta.url) (but this can depend on whether the importer and/or importee are also entry files). Haven't figured out how to address this yet.~
  • Actually, looks like the issue is that the import.meta.url is dropped from the new URL, which breaks relative resolution (i.e. will not in browsers if the call site's JS file is not in the same folder as the current page). I'm able to address that by assigning e.Args[0] instead of e.Args when adding the ERelativeURL to the AST.

[rauschma]

My use case is static site generation: The same code has to run on the client and in Node.js. And currently new URL(relPath, import.meta.url) is the best cross-platform pattern for associating assets with JavaScript modules.

If the asset files are copied next to the bundle, then the code can remain unchanged. If the files are renamed (e.g. to append digest strings), then only the first argument of new URL() has to be changed during compilation.

More information on associating assets: https://web.dev/bundling-non-js-resources/

[evanw]

I'm planning on releasing a form of this sometime soon. You can see my work in progress version here: #2508. This will initially require code splitting to be enabled (as well as bundling and ESM output of course) as esbuild's internals currently mean each input module only shows up in one output file. It will also require paths to be explicitly relative (i.e. start with ./ or ../) to avoid ambiguity. Otherwise it will work very similar to how import() already works. So for example if you do new URL('./foo.bar', import.meta.url) and .bar files are loaded with the copy loader then it will be copied to the output directory. But if .bar files are loaded with the json loader then it will be parsed as JSON and converted to a JavaScript file with that JSON data as the default export. Let me know what you think.

[edoardocavazza]

I think it would be great.

It will also require paths to be explicitly relative

Since we are using the URL constructor, is this really mandatory? Isn't new URL('foo.bar', import.meta.url) equivalent to new URL('./foo.bar', import.meta.url)?

[evanw]

Since we are using the URL constructor, is this really mandatory? Isn't new URL('foo.bar', import.meta.url) equivalent to new URL('./foo.bar', import.meta.url)?

Two reasons why I was thinking explicitly relative could be a good way to start:

1. Leaving off the ./ is confusing because with node it typically means that the path is a package path instead of a relative path. People may otherwise expect that putting a package name there will function as a package path instead of a relative path. See for example #2470, specifically this part:

   • Import paths should work, e.g. @some/package/worker if @some/package exports ./worker.

   Here's another example of someone wanting to do this: https://stackoverflow.com/questions/70688128/. It's straightforward to get this to work by writing a shim module that re-exports from a package and passing a relative URL to the shim, so importing directly from a package isn't critical to support.

2. What to do for paths that aren't explicitly relative isn't clear to me right now. Not supporting them for now gives space for esbuild to support them later without breaking use cases that started working earlier. And not supporting them doesn't lose any expressive power because you can just insert ./ at the beginning.

[evanw]

I just investigated what Webpack/Parcel/Rollup do for this feature:

• Parcel: Using new URL() with a .js file bundles the .js file. This is what I was thinking of doing. If you leave off the ./ it still works but it's treated as a relative URL. Trying to use a package path with new URL() fails the build because path resolution fails.

• Webpack: Using new URL() with a .js file copies the .js file without bundling it. This is not what I was thinking of doing. In addition, Webpack appears to try both relative and package paths when the ./ is omitted. It appears that maybe relative paths take precedence over package paths? This is similar to what happens with CSS @import which also has this problem regarding ./ being optional.

• Rollup: Doesn't appear to handle this at all? But this isn't surprising because it hardly handles anything without plugins. I assume it's possible to write a plugin to do anything, but that doesn't help me figure out what behavior the community prefers.

So perhaps I should allow omitting the ./ and have that mean "try relative then try package" like Webpack, despite that not being how the URL constructor actually behaves. I'll think more about this.

I also need to finalize what new URL() means. If it's similar to import() but just without evaluating the import, then you'll need to configure a loader to use it. And you probably wouldn't want to configure a loader other than copy for all non-JavaScript files otherwise esbuild will convert the asset into a JavaScript file (e.g. that exports the file as a string if it's the text loader) which probably isn't too helpful. But if it just always copies the file without bundling then it's not useful for creating additional entry points, which also isn't too helpful. I'll think more about this too.

[rauschma]

I’d start with minimal functionality and see where it goes.

My feeling is that how URLs work should stay as close to uncompiled code as possible, which would indeed mean only copy. I don’t think URLs are needed for importing code because import() can be used for that.

[mbrevda]

I just investigated what Webpack/Parcel/Rollup do for this feature:

Plain new URL(), or wrapped in a Worker/import?

[evanw]

Plain new URL(). I'm not yet sold on hard-coding new Worker() recognition. In addition to it not being general-purpose, there are some scenarios where you want a bundled JS file but recognizing the syntax is not really practical such as the audio worklet API. See also https://github.com/parcel-bundler/parcel/issues/1093.

[mbrevda]

I'm not yet sold on hard-coding new Worker() recognition

couldn't the same be said for new URL()? I.e. what if one wants the browser to execute the call and not have the bundler interpit it?

[evanw]

couldn't the same be said for new URL()? I.e. what if one wants the browser to execute the call and not have the bundler interpit it?

It’s the same thing for require() and import(). Bundlers will interpret and bundle these things if there’s a hard-coded string literal inline that hasn’t been marked as external. Otherwise they will leave it alone and pass it through. So you can mark it as external or move it into a variable if you don’t want bundlers to bundle it.

[mbrevda]

Got it. So your concern is with recognizing new Worker, not bundling it. Sure hope that can be overcome!

[rdmurphy]

@evanw re: how Rollup does this — I don't know how popular of a package this is but @web/rollup-plugin-import-meta-assets (docs) is the only plugin I've found that implements something similar, and it worked well enough when I needed it. As its name suggests it only works with assets but I'm unsure if it requires relative paths. (Though I agree with your logic on requiring them in the first pass!)

[marionebl]

To provide another use case this would be useful for; working around cached error responses to retry async module imports, illustrative code:

async importWithRetry(id, attempt = 1) {
    const url = new URL(id, import.meta.url);
    url.searchParams.set('attempt', attempt);

    try {
       return await import(href); 
    } catch (err) {
       return importWithRetry(id, i + 1);
    }  
}

[dominic-p]

I'm currently evaluating esbuild, and I'm wondering if it was decided whether the upcoming functionality would bundle or copy the referenced assets.

For my particular use case, I'm importing Workers. The code in the Worker does further imports so it would definitely be beneficial for me to bundle instead of copy, but that's just my 2 cents. I'm excited to see this getting close to release as it would be nice to be able to drop my old build system in favor of esbuild.