Relative modules break when parentName is a URL

[guybedford]

Consider -

http://some-site.com/app/main.js:

import {dep} from "./dep";
export var app = {};

http://some-site.com/app/dep.js:

export var dep = {};

I now do:

  System.import('http://some-site.com/app/main.js')

This locates to the same path, and the module name is the URL.

When resolving "./dep", we run:

  System.normalize('./dep', 'http://some-site.com/app/main.js', 'http://some-site.com/app/main.js');
  // -> "http://some-site.com/app/dep"

  System.locate('http://some-site.com/app/dep')
  // -> http://some-site.com/app/dep

A module can't be expected to know whether it will be loaded by a module name that is a URL.

What am I missing here?

[jorendorff]

I don't expect people to write System.import('http://some-site.com/app/main.js') though. Instead:

System.import('main', {address: 'http://some-site.com/app/main.js'})

which does not call normalize() or locate() but starts with the fetch() hook.

However this does raise the question of whether to allow the first parameter to import() to be undefined. Right now, we ToString that. Instead I propose using ToString only if the name is not undefined, and if it is undefined, not putting that into the registry. @dherman?

[jorendorff]

@guybedford Hope it's clear what's going on here. We don't expect absolute URLs to be used as module names.

[guybedford]

But this isn't possible within a module -

  import "http://some-site.com/test.js"

won't work with that.

Being able to easily import from a URL can be useful in some scenarios, otherwise the module code has to assume the existence of location rules for external URLs, leading to a configuration problem.

The loader as implemented currently supports this fine treating the URL as a module name, applying no normalization, and passing it through the locate as it is absolute.

A simple way to resolve this relative conflict would be to define a custom normalization rule that checks if the parentName is a URL, and if so, automatically adds a ".js" extension for the dependency name. Perhaps this or another alternative could be considered?

[johnjbarton]

@jorendorff Sry not at all clear to me ;-) What key will be used in the module dictionary for :

System.import('main', {address: 'http://some-site.com/app/main.js'})

Or what I hope is equivalent, what will refererName be if main.js imports other modules? Do these answers differ if the resource containing the line above has the URL 'http://example.org' and includes other imports of modules that may overlap in name with those imported by main.js (which may or may not be intended to mean the same module depending on the design).

[jorendorff]

@johnjbarton The name of the module imported by that is main, and that's what the Loader uses as the refererName to the normalize hook, if main.js imports other modules.

Those answers do not depend on the URL of the script that's calling System.import().

[johnjbarton]

ok so I completely don't understand how this can work :-(

What is the best way to pose examples that seem to break?

[jorendorff]

Yeah, I'm not communicating super well here. Let's start over.

You have a collection of modules stored in a directory on a server. You want to load the main module. How do you do it?

There are a few ways.

1. Just move the files to your scripts directory. Yay everything works.

2. Configure the System loader to tell it where that module is. The System loader isn't specified yet, but I imagine it'd be something like

   System.paths["app/*"] = "http://example.com/app/*.js";
   System.import("app/main");
   

3. Use a custom locate hook. (You normally wouldn't have to resort to this, but if System doesn't exist, you might.)

4. Use a separate custom Loader. (But you really shouldn't have to resort to this.)

[jorendorff]

@guybedford You can specify the URL through the API, but not through import syntax. That's intentional. Modules are libraries; think how weird it is for library source code to contain absolute filenames. Few programming languages go out of their way to support that with special syntax. YAGNI.

When you do need it, the API stands ready.

People often give the impression they want to run code or change the loader's configuration in the middle of loading a dependency graph. There's some danger doing this, because on the Web stuff loads asynchronously, in a nondeterministic order. If loading something affects the loader's semantics for subsequent loads, it's easy to introduce bugs where a little random latency can break your program. We don't want to encourage that.

Still, you can find a way to do it (the Loader hooks are there, the translate hook gets full source code, and eval stands ready for arbitrary mischief); or you can find a way not to do that (...which I would want to try first).

[guybedford]

@jorendorff thanks for clarifying. If this is the case, then URLs should not be allowed at all and it should be clearly communicated when introducing the module system.

I can work with this, but it was just different to my initial expectations.

[johnjbarton]

The scenarios that I am concerned will fail relate to loading two different trees of modules which internally have identical module specifiers. We need to allow these identical module specifiers to resolve to 1) the same module or 2) different modules, depending on load.metadata or equivalent loader configuration. If the Loader's internal map does not distinguish the references created from the two different trees, then locate() will not be able to route the look-up to two different results depending upon the configuration. (Note that we have no ability to change the module specifiers of the two trees, they are given).

If the two different trees of modules are stored internally in a map with keys that specify the module name in a way that includes the roots of the respective trees, then the same module specifier in these two trees can be distinguished. Then we are free to route them to the same or different modules as we choose with locate.metadata. One concrete key satisfying these criteria would be a string structured as a URL, reflecting each module tree's structure. These pseudo-URLs can be converted to addresses in locate() and used to fetch resources.

Please note that I am not talking about URL in module specifiers. I am talking about the output of System.normalize(). It's possible that the spec has no opinion: you'll take whatever System.normalize() gives, put it back in System.locate() and fetch the result.

[jorendorff]

The scenarios that I am concerned will fail relate to loading two different trees of modules which internally have identical module specifiers. We need to allow these identical module specifiers to resolve to 1) the same module or 2) different modules, depending on load.metadata or equivalent loader configuration.

Right. Having the normalize hook return different absolute module names (depending on the refererModuleName) is the way to do this.

If the Loader's internal map does not distinguish the references created from the two different trees, [...]

The keys for the Loader's internal map are normalized names, so it does distinguish.

One concrete key satisfying these criteria would be a string structured as a URL, reflecting each module tree's structure.

True. Nothing in Guy's original post at the top of this issue is impossible. Your normalize and locate hooks just have to cooperate. Either module specifiers do have ".js" in them or they don't; and the same for normalized module names; and the same for the actual URLs produced by the locate hook. As long as you pick consistent answers for those questions, and implement the hooks accordingly, everything should work.

@guybedford What do you think?

[jorendorff]

It's possible that the spec has no opinion: you'll take whatever System.normalize() gives, put it back in System.locate() and fetch the result.

Right, that's what we do. Is that good enough?

[johnjbarton]

@jorendorff great! The missing bits for me concern the root calls. We are trying to push our module-loader towards your API. I've not worked out values are presented to normalize() in these cases. Hence my concern about our earlier example:

     System.import('main', {address: 'http://some-site.com/app/main.js'})

you said the refererName will be 'main' for imported modules. But some how this needs to pass through normalize() or we lose context.

I suppose we could use refererAddress but then we have to do dubious analysis to guess that we have been called from System.import().

I suppose one answer could be:

    var normalizedName = System.normalize('main', null,  'http://some-site.com/app/main.js');

where normalizedName become the key for the 'main' module. That allows my normalize() to unambiguously know that this is the root (!refererName or refererName null) and it allows use to control the refererName used when main imports.

[guybedford]

@johnjbarton, could you perhaps demonstrate an example of how the refererName context is being lost in the first place? It may help in understanding other options.

@jorendorff I have also been suggesting that the contextual mapping would be solved by a custom normalization as you described, but then thought perhaps not as my worry currently is the question of overloading the System loader. Do we encourage custom normalize functions on the System loader? Would this be handled by the framework of the app?

To overload the System loader may lead to conflict and a lack of understanding what rules should and shouldn't be added.

I like all the principles, but I think users need to be steered in the right direction a little more around how to use them, otherwise the spec will introduce such a variety of interpretations that we get further away from module compatibility not closer to it.

Having been playing with a custom loader for a while now, it still feels "proprietary" and in some ways a waste if the first step is to make a custom loader instead of using the System loader. One of the main reasons I needed to create a custom loader was to support AMD, CommonJS and global script loading, surely this should be in the System loader from the start?

[briandipalma]

One of the main reasons I needed to create a custom loader was to support AMD, CommonJS and global script loading, surely this should be in the System loader from the start?

In a few years time when people are writing only ES modules then that might look somewhat odd.

[johnjbarton]

@guybedford Here is one example:

System.import('main', {address: 'http://some-site.com/app/main.js'});
System.import('main', {address: 'http://some-ads.com/bananas/main.js'});

Loading these into the map under the key 'main' will not be useful.

[guybedford]

@johnjbarton what would be the reason for naming both of these modules "main"? What code in turn would need to utilise that reference?

[johnjbarton]

@guybedford I can't give you a reason, that's why I think it does not make sense, but earlier on this issue @jorendorff said: "The name of the module imported by that is main, and that's what the Loader uses as the refererName to the normalize hook, if main.js imports other modules." Subsequent calls to normalize() would use 'main'.

[guybedford]

@johnjbarton but then why not do:

System.import('site-main', {address: 'http://some-site.com/app/main.js'});
System.import('another-main', {address: 'http://some-ads.com/bananas/main.js'});

?

I'm not sure I see the practicality of the example at all though. Rather as far as I can tell the way to do all this would be something like:

  // set up external URLs as paths
  System.paths['some-site:*'] = 'http://some-site.com/*.js';
  System.paths['some-ads:*'] = 'http://some-ads.com/*.js';

  // let the locate function handle the address resolution
  System.import('some-site:app/main');
  System.import('some-ads:bananas/main');

I believe this is what Jason means by loading external URLs with the location configuration, assuming URLs as module names are out.

@jorendorff let me know if you think I'm getting this wrong though.

[jorendorff]

In this use case, 'app/main' and 'bananas/main' should work, and that's necessary anyway to get relative module specifiers in app/main.js and bananas/main.js to work.

When I said import('main', ...) earlier, I was totally confused about what y'all were asking. It was not, therefore, a good answer. My mistake!

[johnjbarton]

The Loader's normalize()/locate() mechanism wonderfully delegates naming to System, but only for names within a single tree. Why don't we extend this mechanism to apply across trees? Then all of the naming will be delegated and @guybedford can use 'some-site:/app/main', @jorendorff can use 'bananas/main', and I can choose to use 'main' because that is the name that the bananas team used and I want them to maintain their code. As far as I can tell, all we need to do is cause all root module names to pass through normalize() with a convention or signal that the name names a root module. Then the entire module map key system is delegated to System.

[guybedford]

@johnjbarton top level naming not running through normalize only affects a System.import call, and nothing else at all. All other imports within modules can be remapped with the normalize function.

So is the issue here that a dynamic require inside of app/module.js might not know how to reference app/main.js?

app/module.js:

  import './local-dep';

  // a dynamic require for content only needed on demand
  // has no way to know the current module context
  System.import('main');

Relative module names, or the normalize function can solve everything else I believe.

[guybedford]

To update on this discussion - the current System implementation DOES break if a module name is a URL.

That is because the http:// is seen as an empty segment in the //, causing a TypeError.