esbuild
esbuild copied to clipboard
evanw
support for top level await
I am currently working with deno which has top level support, is there any way to use esbuild with top level await?
Sorry, top-level await is not supported. It messes with a lot of things and adding support for it is quite complicated. It likely won't be supported for a long time.
Out of interest / for the issue history, is it possible to give some idea of the sort of issues that arise here?
On Sun, Jul 12, 2020 at 14:36 Erick Sosa Garcia [email protected] wrote:
Closed #253 https://github.com/evanw/esbuild/issues/253.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/evanw/esbuild/issues/253#event-3537454071, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAESFSS2NEUIRROFVJVY7GLR3IUGHANCNFSM4OX6XTIQ .
Right now there is an equivalence in esbuild between CommonJS modules and ES6 modules. A module can be written in either syntax and can even mix both syntaxes. That no longer works with CommonJS because the implementation of require() can't suspend execution in JavaScript. So I'll have to figure out something for that. It's probably some form of conditionally disabling features along the module dependency graph when a module contains a top-level await.
I'd like to punt this feature until later because there are already a lot of edge cases to work through with all of the features added so far (tree shaking, scope hoisting, code splitting, star re-exports, etc.) and this feature adds another dimension to the test case matrix. This feature is also lower priority for me than other features such as watch mode, plugins, finishing code splitting, and adding additional CSS and HTML file types, especially given that top-level await is a new feature and not yet widely used.
Let's keep this issue open because I'm sure it'll come up again. At the very least esbuild should be able to parse top-level await and pass it through in non-bundling mode, which will be useful for people using esbuild as a transformer and not a bundler.
I haven't added parsing support for top-level await yet because it's not clear exactly how to parse it. The way it works the JavaScript spec is that you're supposed to know beforehand whether the parsing goal is "script" or "module" and then only parse the await operator with the "module" goal. However, esbuild doesn't work that way because it makes no distinction between the two goals.
TypeScript is another tool that does this so I've been waiting to see how they handle things. When I looked into this last they had a pretty severe bug so I'm going to wait a while longer for their implementation to settle before I consider how to approach this in esbuild. It looks like they still have open bugs on this such as this one, which interestingly appears to require a symbol table to parse correctly kind of like C. Not sure how I feel about that approach.
Also, while I'm on the topic, apparently this feature also requires supporting top-level for await loops. Just adding that as a note to myself for when I get to implementing this.
That no longer works with CommonJS because the implementation of require() can't suspend execution in JavaScript
This was one of the major issues that the Node.js integration of ES modules had to deal with. The result was we did not permit CJS modules to load ES modules although bundlers like Webpack do. I believe previous dicussions here for eg Webpack have been around allowing require of ES modules so long as they do not use top-level await. Because of the timeline for adoption of top-level await, we may actually be ok here in terms of real world usage.
I haven't added parsing support for top-level await yet because it's not clear exactly how to parse it. The way it works the JavaScript spec is that you're supposed to know beforehand whether the parsing goal is "script" or "module" and then only parse the await operator with the "module" goal. However, esbuild doesn't work that way because it makes no distinction between the two goals.
This was something else the Node.js modules integration had to carefully define with "type": "module" package.json boundaries. Mainly for the strict-mode-by-default handling for modules without module syntax, but also for other reasons like this too.
I would avoid trying to follow anything TypeScript does personally as they have made it clear they will not lead the ecosystem but instead follow the winning conventions when they emerge. As such TypeScript is a typical friction point in modules workflows these days.
One of the other implementation difficulties is that when bundling parallel dependencies, they are supposed to behave like a Promise.all, but in theory that is just a matter of wrapping.
Definitely agreed this is a low priority, but thanks for the detailed explanation and for keeping this open for future reference, however long that might be.
I'm planning to add pass-through support for this now that node is enabling it by default. It won't work while bundling but it will still be useful for certain use cases such as using esbuild as a library for converting TypeScript to JavaScript.
TypeScript is another tool that does this so I've been waiting to see how they handle things.
It looks like TypeScript's implementation ended up being to do an initial parse where top-level await is an identifier, then to reparse all expressions containing those identifiers as await expressions instead if the AST has an import or export statement. This approach isn't a good fit for esbuild's parser for many reasons including lack of a persistent token stream and the fact that parse errors are streamed in real-time and cannot be un-emitted. It's also a very complex approach because it may involve rewriting subsequent expressions (e.g. await \n foo; becomes await foo; instead of await; foo;). And it seems too weird to me to not be able to use await import() in a file without having to add export {} like you have to in TypeScript.
I've decided to just try unconditionally parsing the await operator at the top-level. I'm guessing that the number of modules in the wild that actually expect await to be an available top-level identifier is very small and the benefit of supporting top-level await syntax far outweighs making those modules work, especially for a modern tool like esbuild.
@evanw an added benefit is that TLA is only supported in the module goal and await is a reserved keyword in the module goal 🎉
Note that while transforming code containing top-level await is supported, bundling code containing top-level await is not yet supported.
:(
Is the situation for this still the same?
Is the situation for this still the same?
Yes, top-level await still doesn't work with bundling.
TLA is so complicated. I'm still trying to figure out how to support it but it has been really challenging. I'm trying to match all of the real evaluation order rules including parallel evaluation and microtask ordering while still doing scope hoisting and my attempts keep failing. I might need to undo some of esbuild's existing optimizations to get this in.
One of the other implementation difficulties is that when bundling parallel dependencies, they are supposed to behave like a Promise.all, but in theory that is just a matter of wrapping.
It would be amazing if that actually worked but joining together modules using promises causes things to evaluate in an incorrect order. I have written a TLA correctness fuzzer to verify that a TLA bundling strategy is correct by comparing it against V8, which I assume is correct. The only strategy that I've gotten to work is manually creating a module registry and using it to track which dependencies are remaining for each module. Presumably this is how the runtime works as well. But that's too much generated code size and run-time performance overhead for what I'm going for.
I haven't quite gotten to the bottom of exactly what's so hard about this, but I have a hunch that the problem might be a fundamental difference between how the spec works and how promises work. I think it might be something like "the spec adds dependencies as modules are evaluated but the Promise API adds dependencies at the time .then() is called." I'm still investigating this.
FWIW I couldn't get Webpack or SystemJS to match V8's behavior either. Does anyone know if there is a bundler-like implementation of TLA that is able to match V8's behavior? Also before I get even deeper into this, does anyone know if V8's behavior is incorrect somehow?
Small update: It turns out that the top-level await specification and/or V8 have some subtle bugs that cause undesirable behavior. More details are here: https://github.com/evanw/tla-fuzzer/issues/1. Many thanks to @sokra on the Webpack team for getting to the bottom of these issues.
The newly-released version 0.10.0 has preliminary support for bundling with top-level await (previously top-level await was not allowed at all when bundling). From the release notes:
Initial support for bundling with top-level await (#253)
Top-level await is a feature that lets you use an
awaitexpression at the top level (outside of anasyncfunction). Here is an example:let promise = fetch('https://www.example.com/data') export let data = await promise.then(x => x.json())Top-level await only works in ECMAScript modules, and does not work in CommonJS modules. This means that you must use an
importstatement or animport()expression to import a module containing top-level await. You cannot userequire()because it's synchronous while top-level await is asynchronous. There should be a descriptive error message when you try to do this.This initial release only has limited support for top-level await. It is only supported with the
esmoutput format, but not with theiifeorcjsoutput formats. In addition, the compilation is not correct in that two modules that both contain top-level await and that are siblings in the import graph will be evaluated in serial instead of in parallel. Full support for top-level await will come in a future release.
It works and this is awesome !
Now I can use const { a, b } = await import('c'); on top of all my modules !
It saves lots of duplicates and increases readability :)
Hi,
Into my project, I experimented an issue with a top-level await, on the 0.11.13 version.
Actually, esbuild still let some await on the top-level of the bundle (--format=esm).
I solved it by adding a wrapper like --banner:js='(async () => {' --footer:js='})()'.
Is there a reason to don't wrap the entire bundle, by default (or just if it have some top-level await)?
If the project sources can help: https://glitch.com/edit/#!/immutable-isomorphic-demo?path=package.json%3A7%3A53
(The problematic await is located into the dist/worker.js bundle and the expression to find await dependency('renderer'), originally into the templates)
Is there a reason to don't wrap the entire bundle, by default (or just if it have some top-level await)?
Yes. Doing that would be incorrect. Top-level await is supposed to delay the evaluation of the importing module until the promise has been resolved, and the only way of doing that is to use a top-level await. In other words, top-level await cannot be polyfilled for environments that don't have it, at least as long as you support the possibility that the output file may be imported by something else.
Yes. Doing that would be incorrect. Top-level await is supposed to delay the evaluation of the importing module until the promise has been resolved, and the only way of doing that is to use a top-level await. In other words, top-level await cannot be polyfilled for environments that don't have it, at least as long as you support the possibility that the output file may be imported by something else.
Ok for that, but since the ES modules aren't supported into a worker, the top-level await can't be supported.
Maybe need a --platform=worker to handle a specific behavior and considerations? :) (or another flag to enforce that wrapping, whatever the context)
Ultimately top-level await will be supported with the iife format, which is really what you're trying to do. It will work similarly to --banner:js='(async () => {' --footer:js='})()' but that should be a suitable substitute for now.
ES modules aren't supported into a worker
Yes they are, in both worker and service worker. Use the {type: module} option when instanciating a worker or a service worker.
https://blog.chromium.org/2021/04/chrome-91-handwriting-recognition-webxr.html (look at bottom of the page)
ES Modules for service workers ('module' type option) JavaScript now supports modules in service workers. Setting 'module' type by the constructor's type attribute, worker scripts are loaded as ES modules and the import statement is available on worker contexts. With this feature, web developers can more easily write programs in a composable way and share them among a page and workers
Yes they are, in both worker and service worker. Use the {type: module} option when instanciating a worker or a service worker.
https://blog.chromium.org/2021/04/chrome-91-handwriting-recognition-webxr.html (look at bottom of the page)
ES Modules for service workers ('module' type option)
Unsupported, actually, on Firefox ;)
Just to let people know : I ran into a deadlock issue today using top level await + dynamic imports. Then I switched to dynamic imports inside functions, and the deadlock was gone. It appears that dynamic imports do not play well with top level await in case modules A and B call functions from each others. I don't think this is related to esbuild, this more a language construct.
To avoid such problems, I would recommend for now to avoid using top level await + dynamic imports. Yes, the code is uglier with dynamic imports inside functions (lots of repetition), but at least it works without any deadlock.
The primary motivation of TLA was supporting static imports. Requiring dynamic imports will leak promises at which point you might as well use an async IIFE
One thing we discussed in committee was using linters to avoid deadlock, as we decided against throwing early on deadlock.
On Sat, May 1, 2021, 2:24 PM Denis TRUFFAUT @.***> wrote:
Just to let people know : I ran into a deadlock issue today using top level await + static imports. Then I switched into fully dynamic imports (inside functions) and the deadlock was gone. It appears that static imports do not play well with top level await in case modules A and B call functions from each others. I don't think this is related to esbuild, this more a language construct.
To avoid such problems, I would recommend to avoid using top level import
- static imports. Yes, the code is uglier with pure dynamic imports (lots of repetition), but at least it works without any deadlock.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/evanw/esbuild/issues/253#issuecomment-830674286, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADZYV5VGCMRXGE2MRF3WFLTLRBOLANCNFSM4OX6XTIQ .
Current implementation of bundling TLA is still not matching the un-bundled code. I re-thinked how code runs, and find out there maybe no way to keep doing scope hoisting on "awaited" modules:
// a.js
import './b.js'
import './c.js'
console.log('a')
// b.js
await Promise.resolve()
console.log('b')
// c.js
console.log('c1')
await Promise.resolve()
console.log('c2')
The output of executing a.js should be c1 b c2 a, which seems equal to:
// equal code
let taskB = (async () => {
await Promise.resolve()
console.log('b')
})()
let taskC = (async () => {
console.log('c1')
await Promise.resolve()
console.log('c2')
})()
await taskB
await taskC
console.log('a')
While esbuild gives:
/// esbuild a.js --bundle --format=esm
// b.js
await Promise.resolve();
console.log("b");
// c.js
console.log("c1");
await Promise.resolve();
console.log("c2");
// a.js
console.log("a");
Although putting things in closure is like what webpack does, there's no denying that it produces the most correct code.
The callback approach you are describing is subtly incorrect or at least different from real JavaScript VMs in certain cases, but the correct approach is something similar to that. I did an investigation around this here: https://github.com/evanw/tla-fuzzer. Last time I looked at it, even V8 had ordering bugs that needed to be resolved and then shipped so I paused work on it then. I still need to pick this up again at some point which is why this issue is still open.
What esbuild currently does (linearizing all TLA modules) is obviously incorrect. I still plan to do the full version of this and have already done a lot of the underlying work to enable ES modules to be asynchronously initialized (specifically the changes that shipped in version 0.11.0). Some of the remaining work is to re-examine the current state of TLA implementations, which are hopefully more consistent now, and re-design and then implement an algorithm for bundling ES modules into an appropriately maximally flattened bundle while still preserving correct TLA semantics.
@evanw Would it be possible to add an experimental flag and simply not attempt to transpile TLA expressions? I want to tell esbuild not to worry about it, just emit the statements as they exist in the source.
Previous comments
This is highly desirable for Node 16+ environments (and is why a lot of people are requesting this). I've read your comments on the pain of polyfilling top-level await and totally get it, but Node can support these statements out of the box—I am not asking esbuild to transpile them, just to not crash when it encounters them if I pass a special flag or something. I am using this to build public packages, and would want all transpilation to occur downstream anyway. My goal is to just emit raw ESNext, even if that means a bunch of opt-in flags so as to not be a breaking change.It's interesting because I can use the transform() member to transpile TS and feed it into Node as a loader and it works perfectly (!), but build() is where it throws and there's no escape hatch to get out. I can work around by iterating over files in JS, transforming them with esbuild, and then writing them back to disk, but I imagine there will be performance costs.
Edit: esbuild can already do this!
I was able to silence esbuild errors regarding top-level await by shutting off transpilation entirely with the following BuildOptions flags:
{
target: "esnext",
module: "esm"
}
With node 14 as target you usually get
Top-level await is not available in the configured target environment ("node14")
But now AWS support top level await even for node 14. https://aws.amazon.com/about-aws/whats-new/2022/01/aws-lambda-es-modules-top-level-await-node-js-14/?nc1=h_ls
Is there an option to somehow add an exeption for that feature?
Like --always-allow=top-level-await for example.
Edit: Nevermind. When using CDK with nodejs-function just set
bundling: {
target: 'node14.8'
}
i deadly need TLA to work in a bundler, my use case scenario:
- Project code is written in TS, and build(bundle) for 2 targets: nodejs CLI, "node14", & web in browser
- I use some native nodejs native addons(*.node) in node env, and i want to convert them to wasm to use in web browser env, so i use Emscripten, however, the wasm seems to be async module, but i need them to init before app code normal starts up, (app code needs a wasm module inject)
- To make situations complex, the wasm modules is not one, but many...
The perfect fit is to just use TLA to block before all essential async loads completed, if use Promise.all, then app code has to be put in a then block? That's very weird...
kind of a roadblock for me atm... are there any progress on making a async iife bundle? can't generate ESM atm cuz firefox do not support modules in Service Workers...
@ctjlewis when trying your suggestion:
Invalid build flag: "--module=esm"
I found --format=esm maybe works.
Yeah, that flag is now called —format. I’ll update the answer shortly.
ESM target appropriately includes top-level await because ES Modules allow top-level await. Just saw your edit.
I’m going to be running ESM through pkg soon and I’ll add notes here.