Proposal: Module Loader API

[kitsonk]

With ES Modules, it is effectively impossible to do "hot module reloading" of the modules, as ES Module specification is fairly strict that once the module is resolve and its two pass instantiation is done, that is the end of the road for the module and cannot be replaced.

When you are mocking/testing/etc. though, it is potentially valuable to make changes to a module and "reload" it via dynamic import. In Node.js, they have solved this problem with an experimental loader API. For an example of how this gets integrated to provide a mechanism for module replacements of imports that are still modules, quibble provides ESM support. It basically creates a unique specifier for every time the module is requested to get around the challenge of ES modules being static once resolved.

This would also support solving things like #1739, which we have long wanted to do.

[chrisabrams]

@kitsonk did you ever come across a solution for readTextFile invalidation?

[gherciu]

So any news on this)

Deno team consistently closes every new task opened and links to this And no action on this for 4 years

If you check above at least 10issues closed in favor to this and nothing is happening

cc @lucacasonato just pointing to a big problem, especially, a bug closed with no reason https://github.com/denoland/deno/issues/25742

[bradthomasbrown]

It seems the Deno team not only closes the issues, doesn't offer much in terms of news or progress, but is also minimizing when this is pointed out. That's incredibly disappointing, so far I have found Deno highly preferable as a very lightweight and powerful scripting/programming engine.

Could the Deno team perhaps confirm explicitly that this is something they will not pursue, so this issue can be closed and those watching it and related issues can get closure?

[petamoriken]

This issue is being discussed on the TC39 side. https://github.com/tc39/proposal-esm-phase-imports https://github.com/tc39/proposal-compartments

The following slide from the 8 October 2024 TC39 meeting is helpful. https://docs.google.com/presentation/d/1HF4COMfypVzftilhOIlGxz9Fn1iaiFdwYDZwzqQt1gs/edit?usp=sharing

[gherciu]

The position of Deno team is a bit strange:

• It copies most of the Node API, but says they follow TC39, even tho Node has such module loader api
• They add global Deno.something apis which are not TC39 compliant and never mentioned there but for this feature they do not do that
• They make claims that everything node related is supported using some node layer, and you trust that and when you port your project you find this issue which has 4 years and are totally disappointed because you now are a victim of marketing, and you spent your time to port your project to deno even tho it doesn't fully work

So we follow or not node standards, or we follow only TC39 This is a circle that just slows down the Deno adoption, we need a strong position on what is the path, because if you try to sit on 2 chairs nothing good happens in the end

[petamoriken]

• They make claims that everything node related is supported using some node layer, and you trust that and when you port your project you find this issue which has 4 years and are totally disappointed because you now are a victim of marketing, and you spent your time to port your project to deno even tho it doesn't fully work

I think that statement is incorrect. it wasn't neglected for 4 years, just that the experimental (unstable) API of Node.js is not implemented. https://github.com/denoland/deno/tree/main/ext/node/polyfills#experimental

I understand the frustration of not getting the features you want, but if you want the features you want included, you can contribute in some way by discussing their usefulness for compatibility or actually implementing them to Deno 🙂

[petamoriken]

For example, it's quite useful to report that the widely used npm package does not work properly due to the lack of implementation of the Node.js experimental Loader API.

[gherciu]

We know this is a experimental feature But people come to this because require.cache doesn't work with import in Deno, and that would be an alternative to at least have a custom loader and somehow remove the import cache which is impossible at the moment in Deno.

If you check most of the issues above mentioned, are related to hmr, require.cache, dynamic imports and peoples get redirected to this issue and the ones with the actual bugs are closed by Deno team by mentioning this one for these 4 years, here is the actual issue, if we had the working require.cache with import then no one would even access this 4 years old issue till these days

example: https://github.com/denoland/deno/issues/25742

[petamoriken]

Please calm down. If you want to fix a problem with require.cache, you just provide a repository that shows that you can run it in Node.js but not in Deno.

Sorry but without a full reproduction I can't really say what's going wrong. If you have a repo that can be tested then I can take a look.

https://github.com/denoland/deno/issues/25742#issuecomment-2364587853

(This discussion is obviously not directly related to experimental Module Loader API in Node.js. It might be better to hide their comments as unrelated discussions. It is better to report this problem in a new issue)

[gherciu]

The issue which was closed has a clear description with reproduction steps, please re-read it

How is that related to this issue In most of these issues you are redirected by Deno team to this issue, please re-read the issue and you'll see Even if yes some of them are not related, but they give this as alternative , even if not implemented even the alternative

[gherciu]

Screenshot, just in case And overall closing and redirecting to some 4yr issue is a bad behavior, not even mentioning minimizing a totally fair comment above (Where I just asked for news)

[petamoriken]

Check the timeline. The issue was closed on September 20; a Deno team member requested a repository to reproduce your issue on September 21.

Again, could you please provide a repository to reproduce the issue?

[gherciu]

The reproduction was requested for another thing here https://github.com/denoland/deno/issues/25742#issuecomment-2364099717 Which is another bug in Deno, And yes the issue was closed and then a reproduction was requested which is not how things should work btw Should be the opposite, firstly we discuss and then close if there is no such bug btw And was closed not because of missing reproduction.... the original one they were able to reproduce btw

Thx

[bradthomasbrown]

you can contribute in some way by discussing their usefulness for compatibility or actually implementing them to Deno 🙂

The usefulness was discussed, in many issues. Those discussions were closed and redirected here. The only discussion here was minimized (read: censored). My impression is the exact opposite: no, nobody can contribute in some way by discussion and the Deno team has showed precisely why (no response + censoring).

I had plans to implement something that could help but which would ultimately be a workaround, but seeing that discussion is censored by the Deno team if it is critical, but fair, to the Deno team drives me to not implement the feature in Deno.

The emoji usage comes off as passive aggressive, at least to me.

I also personally don't understand what Node.js or TC39 have to do with considering the functionality for Deno. It should not be required, for some feature, for it to be implemented somewhere else before it is considered for Deno. Otherwise, why use Deno?

[bartlomieju]

Could you folks list which packages you are trying to run that is blocked by not having access to loaders in Deno? Most recently we saw an issue with Playwright that uses TypeScript loader (that shouldn't really be necessary in Deno, but alas), but I'd like to gauge for a wider range of use cases before we undertake this.

Keep in mind that solving this issue will most likely result in worse performance for loading modules as Deno would have to check for registered loaders and potentially hopping multiple times between Rust and JavaScript code to actually load a file before handing it off to V8 for actual execution.

[gherciu]

In my case, we build a new package which will need to load user code, and this code may change at runtime, and since require.cache doesn't work correctly in Deno with import (you can't use even require inside a file that uses import) we can't refresh the cache at runtime (Which is a requirement, and we can't stop the deno process, because it will close server connections), and Deno team redirected me to this alternative Module.Loader In deno you can load a different version using ?query=param but it just loads the first level but the children import again it takes from cache

This package will be used by a big tech company (I can't say the name due to the contract)

For example in Bun the require.cache also shows the modules loaded with import (and you can use these both in same file and no Module related error is thrown) and that is a option we think now, maybe to move to Bun, if in Deno there is no way to achieve it, but Bun has at the moment a small issue to fix, it doesn't return the children imports of a module and is hard to figure out which children's also needs to be deleted from cache and we wait for that

[petamoriken]

Since caching is defined in the language specification when using ES Modules (Cyclic Module Records), this seems to be the only way to accomplish this in CommonJS (Abstract Module Records).

For example in Bun the require.cache also shows the modules loaded with import (and you can use these both in same file and no Module related error is thrown) and that is a option we think now

My understanding is that Bun's implementation deviates from JavaScript...

---

By the way, this issue could be solved by implementing require(esm) feature in Node.js v23 to Deno. That feature seems to work without a flag, but it is still experimental.

I confirmed that the require.cache contains the value of the module:

[gherciu]

At the moment in Deno, import and require just live separate lives. You can't interact between these 2.

In node you at least can compile by yourself .ts to .js and transform all imports to require and you have at the runtime only one way of importing things and you can remove the cache. In Bun this is partially implemented even tho they have a bug at the moment. But they have an alternative to that which is Bun.plugin you may use it as a loader which is an alternative to Module.loader

But in Deno there is no compile step and everything stays as import or require and you just can't uncache something that used import, like literally there is no way. You can't use require.cache with import and the Loader alternative proposed is not even implemented.

[bartlomieju]

At the moment in Deno, import and require just live separate lives. You can't interact between these 2.

You can interact between them just like in Node.js.

In node you at least can compile by yourself .ts to .js and transform all imports to require and you have at the runtime only one way of importing things and you can remove the cache. In Bun this is partially implemented even tho they have a bug at the moment. But they have an alternative to that which is Bun.plugin you may use it as a loader which is an alternative to Module.loader

You can compile these files yourself in Deno as well and you can require them as well. You can only remove from cache if you call require(), you can't remove from cache if you import() something, neither in Node, nor Deno. I can't speak about Bun, because I haven't used that API.

Let's see a short example:

// main.cjs
(async () => {
    const fs = require("fs");
    fs.writeFileSync("./import.mjs", "export default { foo: 'bar' };");
    const foo = await import("./import.mjs");
    console.log(foo);

    console.log(require.cache);
    Object.keys(require.cache).forEach(key => delete require.cache[key]);
    console.log(require.cache);
    
    fs.writeFileSync("./import.mjs", "export default { foo: 'fizz' };");
    const foo1 = await import("./import.mjs");
    console.log(foo1);
})();

$ node main.cjs
[Module: null prototype] { default: { foo: 'bar' } }
[Object: null prototype] {
  '/Users/ib/dev/scratch_node_import/main.cjs': {
    id: '.',
    path: '/Users/ib/dev/scratch_node_import',
    exports: {},
    filename: '/Users/ib/dev/scratch_node_import/main.cjs',
    loaded: true,
    children: [],
    paths: [
      '/Users/ib/dev/scratch_node_import/node_modules',
      '/Users/ib/dev/node_modules',
      '/Users/ib/node_modules',
      '/Users/node_modules',
      '/node_modules'
    ],
    [Symbol(kIsMainSymbol)]: true,
    [Symbol(kIsCachedByESMLoader)]: false,
    [Symbol(kIsExecuting)]: false
  }
}
[Object: null prototype] {}
[Module: null prototype] { default: { foo: 'bar' } }

$ deno -RW main.cjs
[Module: null prototype] { default: { foo: "bar" } }
[Object: null prototype] {
  "/Users/ib/dev/scratch_node_import/main.cjs": Module {
    id: ".",
    path: "/Users/ib/dev/scratch_node_import",
    exports: {},
    filename: "/Users/ib/dev/scratch_node_import/main.cjs",
    loaded: true,
    parent: null,
    children: [],
    paths: [
      "/Users/ib/dev/scratch_node_import/node_modules",
      "/Users/ib/dev/node_modules",
      "/Users/ib/node_modules",
      "/Users/node_modules",
      "/node_modules"
    ]
  }
}
[Object: null prototype] {}
[Module: null prototype] { default: { foo: "bar" } }

But in Deno there is no compile step and everything stays as import or require and you just can't uncache something that used import, like literally there is no way. You can't use require.cache with import and the Loader alternative proposed is not even implemented.

You can have a dedicated compilation/transpilation step like a lot of frameworks do (Vite, Next.js, etc). It's the transpilation output that determines if file uses require() (ie. CommonJS format) or import() (ie. ES modules format).

Altering require.cache doesn't have impact on import in Node either - as per the ES spec, once an ES module is loaded it can't be unloaded.

While I can see why loaders might be a useful feature, there's a whole ecosystem that worked without loaders by handling transpilation it self. It is expected that it will work exactly the same in Deno as it does in Node.js.

If you have some example code that works in Node, but has wrong behavior in Deno I'll be glad to take a look and fix the bug.

By the way, this issue could be solved by implementing require(esm) feature in Node.js v23 to Deno. That feature seems to work https://github.com/nodejs/node/pull/55085, but it is still experimental.

Deno supports require(ESM) as of Deno v2.0.

[petamoriken]

Deno supports require(ESM) as of Deno v2.0.

I didn't know that. Thanks!

[gherciu]

You can compile these files yourself in Deno as well and you can require them as well.

I mean, why then use Deno? If in the end I can run compiled files with Node, that doesn't make sense The whole point of not transpiling files and running TS as is, just gone then

So the idea is just to move back to node and do the compilation to commonJS, that's what I understand

And that's the problem, we apply a Node mindset to Deno

[CyanChanges]

Could you folks list which packages you are trying to run that is blocked by not having access to loaders in Deno? Most recently we saw an issue with Playwright that uses TypeScript loader (that shouldn't really be necessary in Deno, but alas), but I'd like to gauge for a wider range of use cases before we undertake this.

Keep in mind that solving this issue will most likely result in worse performance for loading modules as Deno would have to check for registered loaders and potentially hopping multiple times between Rust and JavaScript code to actually load a file before handing it off to V8 for actual execution.

In @cordisjs/loader (https://github.com/cordiverse/cordis), when a hot module reload triggered https://github.com/cordiverse/cordis/blob/master/packages/hmr/src/index.ts#L252-L254 (this.internal is esmLoader from node:internal/modules/esm/loader) the cache of the imported plugin and all related services is deleted from loadCache, after that, we imported the plugin again and plug the plugin back. Without accessing the loader, we can't invalidate the loadCache, even we can "reload" theplugin entrypoint (with ?query=random-string), the children imports of the plugin can't be reloaded.

[CyanChanges]

We should have something like this

// mod1.ts
console.log("Hello World")

// user.ts
await import("./mod1.ts")
await import("./mod1.ts")

console.log("delete the cache")
delete Module.Loader.cache["./mod1.ts"]

await import("./mod1.ts")

$ deno run user.ts
Hello World
delete the cache
Hello World

Thinking about that user.ts is created by the user, and you can't change it (like add #<random-number> to do a uncached import).

[CyanChanges]

I have created a fork of Deno (which I just decided to called it Done), implements a very simple loader api

here is the demonstrate of invalidating import cache

[CyanChanges]

I have making progress by try provide the ability to mutate the ModuleMap (a struct in deno_core that handles all imports including reuse evaluated module (module cache) and request actual dyn ModuleLoader for loading) If you are willing to push this forward, please checkout the issue and the pr, and show your opinion about this

I will update the progress of this here.

Progress

• [ ] deno_core: feat: enhance ModuleMap https://github.com/denoland/deno_core/issues/1143 https://github.com/denoland/deno_core/issues/1146
• [ ] deno: add ext/loader to provide Deno.loader to users
• [ ] deno: maybe also import hooks support?

We could put this in Deno under a permission flag or unstable flag, require it to be explicit enabled by the user. This enable framework developer to have HMR implemented without pre-processing user code.

This could a possible spec violation?

Since caching is defined in the language specification when using ES Modules (Cyclic Module Records), this seems to be the only way to accomplish this in CommonJS (Abstract Module Records).

Evaluation must be only performed once, as it can cause side effects; it is thus important to remember whether evaluation has already been performed, even if unsuccessfully. (In the error case, it makes sense to also remember the exception because otherwise subsequent Evaluate() calls would have to synthesize a new one.)

Firstly, the module would not be re-evaluated (i guess it's not possible to re-evaluate a v8::Module ?)

So, what we are doing here is not re-evaluating a existing module, but we are creating a new module, and reassign a existing specifier to a new module's id,

So the old module is still there, would not be removed, or evaluated again. we are only making the specifier resolves to the new module.

So, I don't think this is a spec violation. As we are not touching the original Module by anyway.

[Hajime-san]

My understanding is that there is no API to clear a module in V8, so this will be a non-ESM, CJS-like immediately evaluated object. I think it would be more constructive to create parallel PRs to deno(this repository) and discuss them, rather than discussing the API design in an issue due to deno_core is not a public interface.

[CyanChanges]

My understanding is that there is no API to clear a module in V8, so this will be a non-ESM, CJS-like immediately evaluated object. I think it would be more constructive to create parallel PRs to deno(this repository) and discuss them, rather than discussing the API design in an issue due to deno_core is not a public interface.

I would consider create a PR to deno too. If you're willing to see, I have a prototype that demonstrates the Loader API in https://github.com/CyanProjects/done/tree/main/ext/loader

[iacore]

API Proposal from Lume: https://github.com/lumeland/lume/issues/590#issuecomment-3008927969

addEventListener("filechanged", (ev) => {
  const { filename, canHotReload } = ev.details;
  if (canHotReload) {
    console.log(`The file ${filename} was changed and can be hot reloaded`);
    Deno.hotReload(filename); // Hot reload the file
  } else {
    console.log(`The file ${filename} was changed and cannot be hot reloaded. Restarting process...`);
    Deno.restart();
  }
});

Our Use Case

Lume is a static site generator. A page may depend on Deno to execute a Typescript file. When that file changes, we hope to

1. reload that file
2. render the page depending on that file