[Discussion] Simplification of Loader

[caridy]

Disclaimer: the following is a compilation of many thoughts and discussions about the future of the loader spec, it is not an actual plan, just food for thoughts.

Rationale on why we might want to simplify the Loader API

• Importing relative and global modules on-demand is a primary use-case to level-up with node/CJS capabilities (import() imperative form is needed).
• A module may want to create a new loader instance via new Reflect.Loader(), useful for frameworks and sandboxes.
• A module should never attempt to use System.loader because it makes the module less portable.
• The default global loader instance (e.g.: System.loader) is rarely needed.
• <script type="module"> or node's --module (or it equivalent) are far better options to kick in your app than the global loader.
• fetch and translate hooks can be implemented in user-land via service workers or Realm's evaluation hook.
• instantiate hook can be removed as well if the registry is exposed via the current loader reference.

Examples

Imperative vs Declarative Import

Imperative form of import, equivalent to declarative import statement:

import {something} from "./foo.js";
something(1);

import("./foo.js").then(foo => ({something}) {
    something(2);
});

Even easier when using await:

const {something} = await import("foo");
something(2);

note: the catch is that something may or may not be a live binding depending on how you import it.

Configuring Loader

Extending the current loader via hooks:

import {resolveHook} from "custom-resolver";
import.loader[Reflect.Loader.resolve] = resolveHook;
import("foo").then(foo => ({something}) {
    something(2);
});

note: the catch here is that module "custom-resolver" and its dependencies are imported before attempting to configure the loader, while "foo" and its dependencies are imported after the fact, which means they will be subject to the new resolution rules defined by the new hook.

Similarly, you can apply AOP on current resolver hooks:

import {resolveFactory} from "custom-resolver";
const oldResolver = import.loader[Reflect.Loader.resolve];
import.loader[Reflect.Loader.resolve] = resolveFactory(oldResolver);
import("foo").then(foo => ({something}) {
    something(2);
});

Controlling the Registry

If we open up the resolve hook, then we will have to open on the registry as well:

const loader = import.loader;
loader[Reflect.Loader.resolve] = (name, referrer) => {
    if (name === "foo" && !loader.registry.has(name)) {
        const fooNS = new Reflect.Module(...);
        loader.registry.set(name, fooNS);
    }
    return name;
};
import("foo").then(foo => {
    foo; // ... it is just a reference to the exotic object `fooNS` created in the resolver.
});

note: this also show that the @@instantiate hook may not be needed after all.

Default Loader

Not need to have a System.loader initially, instead, you can use <script type="module"> to gain access to the global loader, e.g:

<script type="module">
const loader = import.loader;
</script>

Custom Loader

A module may want to create a loader instance, pre-populate its registry, configure the hook, and start importing other modules using the newly created loader instance, e.g.:

<script type="module">
const l1 = new Reflect.loader();
// configure the loader in any way you want
l1.import('foo').then(...);
</script>

note: "foo" module and its dependencies will have access to l1 via import.loader.

Open questions

Should the current loader be exposed initially?

const loader = import.loader;
export {loader};

note: it seems that this is needed to simplify the mental model of the example above.

What about other intermedia states for relative modules?

We may want to expose a resolve and load mechanism for relative modules, e.g.:

// relative vs top level?
import.resolve('./foo'); // the referrer (2nd arg) is implicit
import.loader.resolve('foo'); // top level resolve operation
// and load?
import.load('./foo');      // relative
import.loader.load('foo'); // top level

note: the resolve() and load() methods are important to apply performance optimizations.

Alternative, if the key of the module in the loader registry is exposed somehow, then we might not need relative resolve and relative load after all. e.g.:

import.loader.resolve('./foo', import.key); // relative
import.loader.resolve('foo');               // top level resolve operation
import.load('./foo', import.key);           // relative
import.loader.load('foo');                  // top level load operation

note: this second option is probably better because it retains the mental model of dealing with the loader power object.

[caridy]

/cc @wycats @dherman @ajklein @bterlson

[caridy]

Removing the fetch hook will solve https://github.com/whatwg/loader/issues/132

[caridy]

related to https://github.com/whatwg/loader/issues/121, https://github.com/whatwg/loader/issues/69 and https://github.com/whatwg/loader/issues/130

[caridy]

this will also answer the question from https://github.com/whatwg/loader/issues/72, since there will be no global loader to be mutated.

[caridy]

it also touch on the imperative form of import defined in https://github.com/whatwg/loader/issues/36

[caridy]

this might entirely solve https://github.com/whatwg/loader/issues/89

[domenic]

This seems like a great direction to me.

Further simplification may be achievable by not allowing mutation of import.loader. instead you load scripts with custom loaders using thatLoader.import (which also applies to their dependencies).

That way you can't affect the way other code loads, unless you yourself load it.

[matthewp]

In regards to the fetch hook and using service workers, would there be a way to know, in the service worker fetch event, that the FetchEvent is coming from a loader request (vs. an XHR request, a <link> request, etc.)? That would be important to know in order to do the types of things that you are likely to do in a loader fetch or translate hook.

[guybedford]

Nice to see the simplification effort here! Is 'import.loader.load' here just a normal import function? On Fri, 01 Jul 2016 at 02:11, Matthew Phillips [email protected] wrote:

In regards to the fetch hook and using service workers, would there be a way to know, in the service worker fetch event, that the FetchEvent is coming from a loader request (vs. an XHR request, a request, etc.)? That would be important to know in order to do the types of things that you are likely to do in a loader fetch or translate hook.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/whatwg/loader/issues/147#issuecomment-229822816, or mute the thread https://github.com/notifications/unsubscribe/AAkiyvg05xg2E8pB-Uj0bXQgWZZiLIVPks5qRFtDgaJpZM4JCrlP .

[caridy]

@guybedford no, import.loader.load is just loader.load(), which is available today to fetch a module and its dependencies, instantiate all of them, and get them ready to be evaluated.

[guybedford]

Thanks @caridy that makes sense. Will loader.define still be available for population of modular source text? Would be necessary to allow translation workflows to esm I think? On Fri, 01 Jul 2016 at 15:49, Caridy Patiño [email protected] wrote:

@guybedford https://github.com/guybedford no, import.loader.load is just loader.load(), which is available today to fetch a module and its dependencies, instantiate all of them, and get them ready to be evaluated.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/whatwg/loader/issues/147#issuecomment-229951606, or mute the thread https://github.com/notifications/unsubscribe/AAkiysV34houNttd0IycMJ9MFrQuT81Gks5qRRrogaJpZM4JCrlP .

[caridy]

@guybedford no, you can push module records into the registry (check the example "Controlling the Registry"), but as today, there is no way to process a source text module record from the loader. This might or might not be a problem, we have some ideas. Alternative, you can always rely on the service worker to produce the string to be evaluated for a particular key/url, in which case the loader will evaluate it.

[guybedford]

That does remove some use cases for in-browser transpilation eg macros for modules. While it can be done in the service worker, a modular 'eval' through loader.define to a given module name would be ideal to properly support this. On Fri, 01 Jul 2016 at 16:46, Caridy Patiño [email protected] wrote:

@guybedford https://github.com/guybedford no, you can push module records into the registry (check the example "Controlling the Registry"), but as today, there is no way to process a source text module record from the loader. This might or might not be a problem, we have some ideas. Alternative, you can always rely on the service worker to produce the string to be evaluated for a particular key/url, in which case the loader will evaluate it.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/whatwg/loader/issues/147#issuecomment-229966011, or mute the thread https://github.com/notifications/unsubscribe/AAkiyuOwnvImcJXI1ncmtI2nCIG5CiRHks5qRShAgaJpZM4JCrlP .

[caridy]

@guybedford there is not such things as a module name in loader, but a registry key. My position is that the loader doesn't need to give you such mechanism, just like it doesn't let you create a new reflective module record, you do that via new Reflect.Module(), and then you push the exotic namespace object into the registry. Similarly, we could have a way to create a Source Text Module Record from a source, returning a namespace object to userland. Whether that's via a constructor like Reflect.Module, or via a Realm or any other mechanism, I don't know.

[guybedford]

Right, I mean key instead of module name in the above. Ideally modular eval is a function of the key and source text, returning a promise that acts just like loader.load. Loading and execution can then happen according to the normal spec support, which isn't possible if we want to get back an executed record immediately. I don't know what API is best either - just commenting on the fact that modular eval is now not possible. On Fri, 01 Jul 2016 at 17:09, Caridy Patiño [email protected] wrote:

@guybedford https://github.com/guybedford there is not such things as a module name in loader, but a registry key. My position is that the loader doesn't need to give you such mechanism, just like it doesn't let you create a new reflective module record, you do that via new Reflect.Module(), and then you push the exotic namespace object into the registry. Similarly, we could have a way to create a Source Text Module Record from a source, returning a namespace object to userland. Whether that's via a constructor like Reflect.Module, or via a Realm or any other mechanism, I don't know.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/whatwg/loader/issues/147#issuecomment-229971780, or mute the thread https://github.com/notifications/unsubscribe/AAkiyvaUE5Wo4EXGxjNEaEcEVIjH_Rk3ks5qRS2agaJpZM4JCrlP .

[matthewp]

@caridy I thought ModuleStatus.prototype.resolve was the new loader.define equivalent, is that wrong or are you thinking about removing it?

[caridy]

@matthewp ModuleStatus.prototype.resolve just return the promise of the resolve hook, that's all. What @guybedford is claiming is that by removing the other hooks, you loose the ability to feed the loader with a source text in a form of a string value that the loader will use to call ParseModule abstract operation, which is an accurate claim.

[guybedford]

I have one other concern about this proposal - resolve will now execute as a side-effect for legacy module format cases. But the issue here is that since the loader is available to modules I can write portable code that uses this resolve function, and won't know whether my resolve will happen to result in an execution side-effect or not.

For example:

import {x} from 'y';

import.resolve('q').then(function(resolved) {
  // I really didn't mean to intend execution of q with the above code
  // but if q is a dynamic module, then it would have executed by now!
});

~~The other issue is that it is no longer possible for dynamic modules to execute within their ordering relative to the ES module tree by providing a function output from instantiate. This can be useful for cases like:~~

import 'es-module-1';
import 'cjs-module';
import 'es-module-2';

~~Where we want import ordering to apply so that es-module-1 executes before cjs-module (given no other tree dependencies present).~~

~~If we were to still retain the instantiate function, then we'd still have the ability to provide these kinds of cases.~~

Update: because Module objects have a separate evaluate stage, the above execution ordering can be retained. So it is only the resolve argument above that still stands here.

Let me know if anything in the above is unclear though.

[Alphapage]

I don't understand why metadata is only available after resolve: I think it should be available asap during each hook. Instantiate could help to do some kind of preload. I think for now, it is impossible to resolve, fetch, translate and stop code execution. Instantiate could help to put a module in 'standby' and execute later when needed.

[caridy]

@guybedford

resolve will now execute as a side-effect for legacy module format cases

This is a very good point, yes, resolver hook will have to resolve, inspect and set up something, even when we don't plan to use it, fair enough.

it is no longer possible for dynamic modules to execute within their ordering relative to the ES module tree

I'm not sure this is 100% accurate. For legacy modules, you can still create a reflective module record, passing an evaluator function to apply lazy evaluation, which is going to be triggered just like source text module record evaluation. Also, there is not such thing as a deterministic evaluation order of your dependencies (because they might or might not be already evaluated by another turn), the only thing that is deterministic here is that the dependencies of a module will be evaluated before the module is evaluated, and when circular dependencies are present, this is also deterministic based on the root module to be imported.

[shicks]

Is there a compelling reason to invent the new syntax of having import be indexable and callable? At the very least, this makes it much more difficult to polyfill, and seems gratuitous when it could just as easily be System.import which is valid ES and doesn't require changes to the language.

[matthewp]

System is a global where import.loader presumably refers to the loader that loaded the module.

[guybedford]

@caridy

I'm not sure this is 100% accurate. For legacy modules, you can still create a reflective module record, passing an evaluator function to apply lazy evaluation, which is going to be triggered just like source text module record evaluation. Also, there is not such thing as a deterministic evaluation order of your dependencies (because they might or might not be already evaluated by another turn), the only thing that is deterministic here is that the dependencies of a module will be evaluated before the module is evaluated, and when circular dependencies are present, this is also deterministic based on the root module to be imported.

I'm still not exactly clear on this API. Are you saying I could write something like the following:

loader[Reflect.loader.resolve] = function(key, parent) {
  var exports;
  loader.set('some/cjs/module.js', new Reflect.Module({
    default: { value:  null }
  }, function executor(mutator, module) {
    exports = mutator;
  }, function evaluate() {
    console.log('CJS evaluated');
    exports.default = 'evaluated';
    // CJS loaded from CJS would trigger require execution down the stack here I assume?
  });

  return resolve(key, parent);
};

Where if the module at resolve(key, parent) were to contain:

import './first-es-module.js';
import cjs from 'some/cjs/module.js';
console.log('CJS module from ES module is ' + cjs);

We would get first-es-module.js executing before some/cjs/module.js outputting CJS evaluated before finally getting the output CJS module from ES module is evaluated? If so I guess it does capture the instantiate utility through the module record process?

[caridy]

@guybedford it is not exactly like that (what you set into the registry are instance of ModuleStatus), but you get the idea right. and yes, it does preserve the order of evaluation. this is ideal to support circular dependencies, and all other features of the ES Modules.

[guybedford]

Thanks @caridy for the clarification.

The remaining issue here is then that CommonJS that loads ES modules would need to execute those ES modules within the resolve hook itself (effectively what was the zebra striping).

So even with this execution separation, you'd still need execution within the resolve hook for those cross dependencies, which may still be enough to justify a separate instantiation hook which just separates the phases.

[caridy]

@guybedford correct, it will only work if you already have the CJS module ready to be used before the resolve hook gets invoked (loaded as browserify/webpack bundle of something), or load them sync from disk in node.

[matthewp]

Maybe resolve can be moved out into service workers as well. It doesn't make sense that JavaScript modules will have a resolve but html imports will not. It should be generic. If the "fetch" event added some more information; the document.baseURI, and the identifier (./foo.js) for example then resolve wouldn't be needed window-side and we'd gain the benefit of resolving non-JavaScript stuff.

[domenic]

Having had a few weeks to think about this and talk with other developers, let me outline what I have heard as the desired features:

• Some way of getting a promise for a module instance object by URL (call it window.importModule(url)), usable from classic scripts and module scripts.
• Some way of getting a promise for a module instance object by specifier, usable only in module scripts, resolved relative to the current module script. (Call it import(specifier)).

Some people have asked for the following:

• Some way of customizing the resolution hook, with fallback to the basic one
  • The most basic use case of this is just adding a bunch of mappings for "bare" module specifiers, with no resolving-module dependency. That is, everywhere in the app, "jquery" should point to the URL /js/libs/jquery.js. Everything that is not "jquery" should fall back to the default resolution algorithm.
  • A smaller number of people have asked for the ability to customize based on the resolving module as well, so that inside /node_modules/foo/foo.js, "jquery" points to /node_modules/foo/node_modules/jquery/jquery.js, but inside /node_modules/bar/bar.js, "jquery" points to /node_modules/bar/node_modules/jquery/jquery.js.

However, it's not clear that building in this customization is terribly advantageous, compared to just letting the service worker rewrite the module specifiers. Performance-wise, both are pretty bad; the preload scanner is entirely defeated, which is the biggest hit. I imagine in production people who are not bundling will use tools that rewrite their module import paths to absolute paths to allow the preload scanner to do its work. In any case, this kind of customization certainly seems lower priority.

I am not sure what to do in this area. But in general I am hopeful that trying to solve this problem is not seen as a blocker to solving the previous two.

I also got one ask for the following:

• The ability to map specific URLs to specific source text

However this is clearly more in the territory of service worker.

Nobody has asked for the ability to reflectively construct modules and populate the module map through JavaScript, although some have wanted to be able to do that in the V8 API.

Nobody has asked for the ability to have customized loading behavior in different sections of an application.

[caridy]

@domenic this sounds good, I don't see any red-flags. The only detail that I'm not sure is about this:

Nobody has asked for the ability to reflectively construct modules and populate the module map through JavaScript, although some have wanted to be able to do that in the V8 API.

How are they suppose to interoperate with legacy module systems? If you import jquery, what does that means? is jquery an ES Module? is jquery a global value? is jquery a reflective module record? The only reason why the reflective module record exist is to provide interoperability with existing module systems, how are they going to survive without the ability to create a synthetic namespace exotic object?

[domenic]

@caridy jquery is an ES module, provided by a service worker or more commonly by ahead of time translation/wrapper modules. (That was the motivating use case behind the ability to map specific URLs to specific source text.)

From what I've found people are interested in having ES modules transparently solved in Node.js (through the V8 API), but in the browser they're already using bundling/transpilation/service worker rewrites, and turning to that to solve their problems.