loader
loader copied to clipboard
whatwg
Node core dev thinks loader will not work for Node
See https://github.com/WebAssembly/design/issues/256 where @trevnorris outlines his issues. I think it would be useful to discuss them in this repo. I am pretty sure they are not quite accurate (for example you can definitely do conditional loading at runtime, by customizing the loader), so it would be good to address such misconceptions and to make sure all use cases are covered.
Yes they mostly seem incorrect arguments. @caridy @dherman it would be great if one of you could answer these questions. I'll chime in the next day or so otherwise.
@domenic from a lot of what I have heard/talked to people Loader would be instrumented as part of the es6 loading mechanism; which is where these concerns come from, having a module/global Loader object is somewhat benign but does have an odd case of when async scripts get evaluated by loader (on the microtask queue?) vs require doing them at point of require.
as per the conditional loader, it is more limited since the resolution is part of the loader, you cannot have easy user logic like if (!Promise) require("bluebird") to my knowledge
@domenic Conditional loads, which require a customized loader, don't give the same flexibility as the user being able to freely define the path at runtime. Also, that comment was about the "import syntax in general", not about loader specifically.
The point is valid, but it didn't have anything to do with loader specifically. Was only mentioned to point out issues of common practices in node (e.g. lazy loading). It shouldn't have been included in the bullet points. Apologies.
@bmeck @trevnorris I don't think you guys are wrong, particularly about performance side-effects from an asynchronous loader.
But conditional loading can be done at runtime by the user. They cannot use the import syntax for this, but they could do something like:
if(needThing) {
let thing = await loader.import("thing");
}
@matthewp for places that support await , which is not available at top level or module level of the https://github.com/tc39/ecmascript-asyncawait proposal
Hm, I don't see that in the spec but will take your word for it. Nevertheless all places support Promises and will be able to be load modules with a loader.import form regardless of whether a friendly syntax is available or not.
@matthewp If there is no performance advantage, is breakage with the existing ecosystem and added complexity, why would we consider implementing it the server-side? I'm at a loss to find any advantage over the current method.
@trevnorris Because the ecosystem is JavaScript, not just Node, and that ecosystem is already broken. I would suggest that we find a way to address the legacy needs so that JavaScript can have a module system rather than multiple incompatible ones for each host environment.
@matthewp If the spec requires import to be async, then we'll just program around it. Just like we've had to program around other parts of the spec so JS can be reasonably used on the server. The ecosystem uses what works best, and we're not constrained like the browser so the flexibility of getting around useless (to us) parts is easy.
@matthewp https://tc39.github.io/ecmascript-asyncawait/#modified-productions as per the lack of it at a module level
We would love to have it work with existing code! Which is why we are here, to discuss what measures can be taken to do so. We just need to be informed to what the advantages vs. incompatibilities are. There seems to be a lot of disregard or putting the burden on existing code to change which only makes our concern grow. I think a good first step is to try and find issues that are fairly simple to reproduce like the circular dependencies and discuss them as @domenic said.
I would think you would be able to keep backwards compatibility with require using the loader.install API.
@matthewp Still would not fix circular dependencies or the timing concern of task queue vs immediate side effects as I read it. Maybe you can elaborate?
That's true, any module being required would need to go through the Loader APIs as a module.
I can't think of a reason why the loader hooks have to be async. If you used a synchronous Promise you should see side-effects occur in the same order as they do today.
@matthewp currently there are multiple stages of promises during the evaluation phase that would have to be placed on the task queue if we use a fulfillment handler :
- https://whatwg.github.io/loader/#reflect-loader-import
with a fulfillment handler that, when called with argument _key_, runs the following steps - https://whatwg.github.io/loader/#request-ready
@trevnorris @bmeck ...
@domenic Conditional loads, which require a customized loader, don't give the same flexibility as the user being able to freely define the path at runtime.
Conditional loading does not requires a customized loader or any sort, you can perfectly use the runtime built-in loader which will be accessible from any module executed in the runtime, and available as global somehow.
Can someone provide more details about the circular dependencies issue, I don't see the problem :)
Other thread that you have been replying to contains the comments and relevant links:
- https://github.com/WebAssembly/design/issues/256#issuecomment-120481934
- https://github.com/WebAssembly/design/issues/256#issuecomment-120558466
On Fri, Jul 10, 2015 at 6:45 PM, Caridy Patiño [email protected] wrote:
Can someone provide more details about the circular dependencies issue, I don't see the issue :)
— Reply to this email directly or view it on GitHub https://github.com/whatwg/loader/issues/54#issuecomment-120551704.
@caridy
you can perfectly use the runtime built-in loader
Let's work through a hypothetical scenario.
Take modules A, B and C. Such that C imports B and B imports A.
C declaratively imports B
B declaratively imports A
Simple enough. Until:
A must now imperatively import another module to finish its export
B needed to create a new instance of A immediately, and now must wait
C is wondering what happened and why it has to change from declarative to imperative
When attempting to simulate synchronous imports it only takes one module in the chain to require an imperative import to then force the remaining to do the same. This means developers will have mixed declarative and imperative imports in the header to accommodate each module. This is added complexity that devs won't want to deal with.
Amust now imperatively import another module to finish its export
@trevnorris I don't fully understand this statement. export declaration are "always" declarative, and bindings per export declaration are created during parsing, not need to wait for something to be imported (declarative or imperative) like node does. I don't see why B or C will have to change at all. Can you elaborate more?
Are you talking about something like this for A.js?
let thing = await import.default('thing');
export {thing};
It's becoming difficult to track the discussion both here and on WebAssembly. So I'm migrating all my comments here.
I would like to clarify something. There are comments about loading the files synchronously from disk, but will that still be done within a Promise executor?
@trevnorris sure, I mentioned that the pipeline allows hooking any loading style for CommonJS (including synchronous loading and execution). The way that would work is that instantiate gets called for a top-level module, that the loader knows from other means (meta configuration) is CommonJS. It then calls into the custom CommonJS loader, which doesn't need to communicate at all with the ES loader pipeline - cjsLoad(entryPoint) or whatever. Once the module (and all its dependencies) are loaded, executed and defined by that custom pipeline, it just returns the defined module. The only promise wrapping is the standard one around the top-level entry point. The same principle applies for ES modules and CJS without a dynamic loader.
There is miscommunication going on about all of this; I will be on a Google Hangout on Air today from 1PM CDT 14 Jul 2015 to discuss with anyone who wishes to.
Event Link: https://plus.google.com/u/0/events/cin8ppflalbfnsi9ilegfugjnu8
Hangout Link: https://plus.google.com/hangouts/_/hoaevent/AP36tYfG4iSLuHeoT4iah5E_Yeqz6eFnsGFOVCHOMrlySzEIdy4T8A?authuser=0&hl=en
On Tue, Jul 14, 2015 at 5:17 AM, Guy Bedford [email protected] wrote:
@trevnorris https://github.com/trevnorris sure, I mentioned that the pipeline allows hooking any loading style for CommonJS (including synchronous loading and execution). The way that would work is that instantiate gets called for a top-level module, that the loader knows from other means (meta configuration) is CommonJS. It then calls into the custom CommonJS loader, which doesn't need to communicate at all with the ES loader pipeline - cjsLoad(entryPoint) or whatever. Once the module (and all its dependencies) are loaded, executed and defined by that custom pipeline, it just returns the defined module. The only promise wrapping is the standard one around the top-level entry point. The same principle applies for ES modules and CJS without a dynamic loader.
— Reply to this email directly or view it on GitHub https://github.com/whatwg/loader/issues/54#issuecomment-121192353.
@bmeck thanks for bringing a discussion together. I won't be able to make it unfortunately, but will follow the notes!
@guybedford I think it's still a problem that ES modules can't be loaded synchronously. Users are used to doing:
if(needThing) {
exports.thing = require("thing");
}
With ES modules they could do this with:
var thing;
export thing;
if(needsThing) {
thing = await loader.import("thing");
}
So any code importing this module can't use "thing" right away.
Maintaining compatibility with CommonJS is not enough. We need to maintain compatibility with Node coding style.
@matthewp I used multi-lines just to facilitate the discussion, but you can always go 1-liner. Btw, just a side note: I haven't seeing a lot of that (if (needsThing) at the top level) in the wild, that's more of conditional loading. Instead, I see a lot of this:
module.exports.thing = function () {
var something = require('something');
something();
}
That's the one that I'm more interested to, since it represents the ability to load things on demand.
That's the one that I'm more interested to, since it represents the ability to load things on demand.
If that were an ES module that would become:
export function thing(){
let something = await loader.import('something');
return something();
}
So any code consuming this module must await the result of calling thing(). Please correct me if I'm really wrong here, loader.import will always return a Promise or no?
It's the usual problem of once something becomes async there is a snowball effect causing everything else to need to as well. I don't know if the Node community will like/accept this.
I don't know if the Node community will like/accept this.
I appreciate this concern for the module ecosystem. You're assessment of the async snowball effect is correct. This would be a significant deterrent preventing adoption.
It needs to be understood that even after this lands in V8, the existing module loading mechanism will continue to work and to be used. If this spec breaks interoperability with the now 170k modules it's likely to gain little adoption.
Any type of asynchronous behavior (i.e. needing to use a callback or await) breaks current behavior. By synchronously loading a dep in the Promise executor it may possibly work if dependencies are only one level deep, but after that it's hosed.
While Loader loads one level of modules at a time, currently modules are loaded along each dependency branch until it reaches the leaf. Where it then unwinds until another dependency is required. The difference between the two is significant.
@trevnorris certainly any minor break in behaviour is not acceptable for an upgrade path and will inhibit adoption of ES modules in Node. If it helps to clarify, here is an example loader hook implementation that would be backwards-compatible in Node:
hook('resolve', function(key, parent) {
// check file, file.js, file.json, package.json etc etc
// this can be sync, but yes it is wrapped in an async wrapper function
return nodeLookup(key, parent);
});
hook('instantiate', function(key) {
// already determined during lookup process from package.json
if (isCJS(key)) {
// exact Node require, returns Module instance object
return Module(nodeRequire(key));
}
else {
return undefined; // ES6 module
}
});
The above pipeline would support loading CJS from within ES6 and dynamic imports of CJS via the module loader.
The nodeRequire above is exactly the NodeJS require function. In no way does it have to expose its internals to the loader. That is, the CJS loading pipeline remains synchronous and backwards compatible. The loader pipeline only loads one module - entrypoint.js, the CJS require does the rest. In the browser we load CJS asynchronously yes, and with the limitations of that approach, but that is only because of the nature of the browser environment. Sorry if I'm explaining this badly, feel free to ask any questions further.