nodejs
Invalidate cache when using import
How do I invalidate the cache of import?
I have a function that installs missing modules when an import fails, but the import statement seems to preserve the failure while the script is still running.
import('some-module').catch(
// this catch will only be reached the first time the script is run because resolveMissingModule will successfully install the module
() => resolveMissingModule('some-module').then(
// again, this will only be reached once, but it will fail, because the import seems to have cached the previous failure
() => import('some-module')
)
)
The only information I found in regards to import caching was this documentation, which does not tell me where the “separate cache” used by import can be found.
No require.cache
require.cacheis not used byimport. It has a separate cache. — https://nodejs.org/api/esm.html#esm_no_require_cache
the import cache is purposely unexposed. adding a query has been the generally accepted ecosystem practice to re-import something.
however, a failure to import something will not fill the cache.
this trivial program works fine for me (assuming nope.mjs does not exist):
import fs from 'fs';
import('./nope.mjs')
.catch(() => fs.writeFileSync('./nope.mjs'))
.then(() => import('./nope.mjs'))
.then(console.log);
@devsnek, hmm, might this be limited to imports that use node_modules? This similarly trivial program fails for me the first time, but not the second.
import child_process from 'child_process';
import('color-names')
.catch(() => child_process.execSync('npm install --no-save color-names'))
.then(() => import('color-names'))
.then(console.log);
Note that the JS spec requires imports to be deterministic/idempotent on a source text. Exposure of a cache would not allow you to fix the code above.
On Fri, Apr 5, 2019, 12:01 PM Gus Caplan [email protected] wrote:
the import cache is purposely unexposed. adding a query has been the generally accepted ecosystem practice to re-import something.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/nodejs/modules/issues/307#issuecomment-480349244, or mute the thread https://github.com/notifications/unsubscribe-auth/AAOUo5N5tioTq2s_t431OrihH_wH8Qagks5vd4FSgaJpZM4cfV4g .
if its just happening with node_modules it could be https://github.com/nodejs/node/issues/26926
I think a use case like this would hopefully be implemented as a loader. Do we already track this as a use case in that context?
@jkrems we have old documents with that as a feature, but no success criteria examples.
FYI, I'm implementing ESM support in Mocha (https://github.com/mochajs/mocha/pull/4038), and cannot currently implement "watch mode", whereby Mocha watches the test files, and reruns them when they change. So "watch mode" in Mocha, in the first iteration, will probably not support ESM, which is a bummer.
While we could use cache busting query parameters, that would mean that we are always increasing memory usage, and old and never-to-be-used versions of the file will continue staying in memory due to the cache holding on to them.
And I'm not sure a loader would help here, as the loader also has no access to the cache.
An API for unloading modules certainly makes sense.
Usually with a direct registry API there is the tracing issue. An API that handles dependency removal can be useful.
A simple API might be something like -
import { unload } from ‘module’;
unload(import.meta.url); // returns true
Where the unload function would remove that module including all its dependencies from the registry. If in a cycle the whole cycle would be removed.
A subsequent module load would refresh all the loads anew.
Other problems to ensure work out is what if modules in the tree are still in-progress. I’d be tempted to say it should fail for that case and only work when all modules have either errored or completed.
We still have memory leak concerns as v8 doesn’t lend itself easily to module GC still. But Node.js can lead the way here as it should. It will be an ongoing process to get there, but the API can come first.
The main questions then seem to be:
- are we ok exposing this as a module or should it be tied to loaders? If tied to loaders how would userland code request this? Or don’t we want it to - as in Mocha should run a new context with a loader?
- should it be a direct registry API (get/set) with tracing, or should it be a deep API like the example above
- finally, ironing out the partially loaded tree edge cases
On Tue, Nov 26, 2019 at 00:09 Gil Tayar [email protected] wrote:
FYI, I'm implementing ESM support in Mocha (mochajs/mocha#4038 https://github.com/mochajs/mocha/pull/4038), and cannot currently implement "watch mode", whereby Mocha watches the test files, and reruns them when they change. So "watch mode" in Mocha, in the first iteration, will probably not support ESM, which is a bummer.
While we could use cache busting query parameters, that would mean that we are always increasing memory usage, and old and never-to-be-used versions of the file will continue staying in memory due to the cache holding on to them.
And I'm not sure a loader would help here, as the loader also has no access to the cache.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/nodejs/modules/issues/307?email_source=notifications&email_token=AAESFSTBSWVLXPTWDYZOHSDQVSVQRA5CNFSM4HD5LYQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFEXNDI#issuecomment-558462605, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAESFSWTJXGB4J6WWNMOLRDQVSVQRANCNFSM4HD5LYQA .
I'm really not a fan of the idea of our module cache being anything except insert-only. CJS cache modification is bad already, and CJS modules don't even form graphs.
Additionally, other runtimes (like browsers) will never expose this functionality, so some alternative system will have to be used for them regardless of what node does, in which case it seems like that system could just be used for node.
@giltayar have you looked into using Workers or other solutions to have a module cache that you can destroy (such as by killing the Worker)?
@bmeck - interesting. That would mean that the tests themselves run in Workers. While I am theoretically familiar with workers, I haven't yet had any experience with them: is any code that runs in the main process compatible with worker inside a worker? In other words, compatibility-wise, would all test code that works today in the "main process" work inside workers?
I wouldn't want Mocha to have a version (even a semver-major breaking one) where developers will need to tweak their code because now it's running inside a worker. I'm guessing that there's a vast amount of that code running inside Mocha, and any incompatibility would be a deal breaker.
there are differences between workers and the main thread, mostly surrounding the functions on process, like process.exit() in a worker doesn't end the process, just the thread. There's a good list here: https://nodejs.org/api/worker_threads.html#worker_threads_class_worker
Looking at the list, I can see process.chdir() is not available, which is probably a deal breaker in many tests (unit tests probably don't use process.chdir(), but Mocha is used for all sorts of tests), as is breaking some native add-ons (although I'm not sure how big of a problem this is in the real world).
I would hesitate to say this, as my only contribution to Mocha currently is this pull request, but I would guess that the owners would veto this. Or maybe allow this only if we add a --run-in-workers option. In any case, without looking too much at the code, this is probably a significant investment to implement for supporting ES Modules, as this is not a simple refactor, but rather an architectural change in how Mocha works.
If it wasn't apparent from the above, I believe I would still prefer a "module unloading" API, unless the working group is adamant and official about not having one, of course. Which would probably mean going the "subprocess"/"worker" route.
i admittedly don't know much about mocha... is using a separate process not doable either?
Hi, I work on Mocha!! I am trying to see how we can move @giltayar's PR forward.
There are actually two situations in which "module unloading" is needed in Mocha:
- In "watch" mode with CJS scripts, when Mocha detects a file must be reloaded, it is deleted from
require.cacheand re-required, then tests are re-run. Mocha is not the only tool that does this sort of cache-busting. - When developers are writing tests with Mocha (and many other test frameworks), they may want to use module-level mocking--they essentially replace one module with another phony one (I'm going to tag @theKashey here because he knows more about this ). Or even pretend like a module does not exist at all. It is then very important to Mocha that users can consume these sort of mocking frameworks to write their test code.
In the first case, it's possible, though probably at a performance cost, Mocha could leverage workers to handle ESM. I don't know enough about workers to say whether this will provide a sufficient environment for the test cases, but it feels like a misuse of the workers feature. At minimum it seems like a lot of added complexity.
In the second case, I can't see how using workers would be feasible. Test authors need to be able to mock modules on-the-fly and reference them directly from test cases, using mocking frameworks.
I don't know why this sort of behavior was omitted from the official specification. If the reasons involve "browser security", well, it further reinforces that browsers are a hostile environment for testing. I do know that this behavior is a very real need for many, from library and tooling authors down to developers working on production code.
We do need an "unload module" API; until such a thing lands, tools will be limited, implementations will be difficult (if possible), and end users will be frustrated when their tests written in ESM don't work. I will also be frustrated, because those frustrated users will complain in Mocha's issue tracker!
I'm happy to talk in further detail about use-cases, but I'm eager to put an eventual API description in the more-capable hands of people like @guybedford.
@devsnek Given that enabling it also enables tooling, I'm curious why you feel locking this sort of thing down is a better direction?
cc @nodejs/tooling
P.S. I will be at the collab summit, and the tooling group will be hosting a collaboration session; maybe this can be a topic of discussion, or vice-versa if there's a modules group meeting...?
Do you need unloading API for the watch mode? Yes, you need it to update the changed module code.
However, it is enough to handle watch mode? No, as long as the idea is to use changed module, as you have to find parents between you(a test) and changed module, and wipe them to perform a proper reinitialization.
So - an ability to invalidate a cache line is not enough, for the mocking task we also have to know the cache graph, we could traverse and understand which work should be done.
An API for unloading modules certainly makes sense.
In my opinion - this feature is something missing for a proper code splitting. There are already 100Mb bundles, separated into hundreds of pieces, you will never load simultaneously. But you if will - there is no way to unload them. Eventually, the Page or an Application would just crash.
@boneskull - the second case you mentioned, I believe can and should be handled by module "loaders", which are a formal way to do "require hooks" for ESM. These will enable testing frameworks (like sinon and others) to manipulate how ES modules are loaded, and, for example, exchange other modules for theirs.
The spec and implementation for that are actively being discussed and worked on by the modules working group (see https://github.com/nodejs/modules/issues/351).
I also need this. I'm making a template rendering engine. When generating the compiled template, I read from a custom format and output to a .js file (a standard ES Module). In order to use the file, I just import it. Upon file changes, I would like to re-write the file, clear the import cache and then re-import it.
These all sound like use cases for V8's LiveEdit debug api (https://chromedevtools.github.io/devtools-protocol/v8/Debugger#method-setScriptSource). You can call into it using https://nodejs.org/api/inspector.html. cc @giltayar @boneskull
+1 for unloading ES Modules.
It's hard to make Hot Module Reload otherwise. Not for production but for development tools.
And using a ?query=x doesn't seem to work on file node 13.11.0 at least.
Thanks
@devsnek Can you provide a little example or pseudo-code on usage of setScriptSource. I have been researching for an 1hour without progress. Thanks
@georges-gomes you can subscribe to the Debugger.scriptParsed event to track the script id, and then when you need to modify the script you can call Debugger.setScriptSource.
@georges-gomes If you are successful, I would be very grateful if you could post a short description on how you could use setScriptSource to solve this problem. On a blog post or something.
@lulzmachine here is a working prototype https://gist.github.com/georges-gomes/6dc743addb90d2e7c5739bba00cf95ea
Unfinished but working. I have seen a few unexpected issues but let see how far we can get with this. Thanks @devsnek 👍
@devsnek I get segmentation fault if I start using import in the new loaded script. I'm not sure setScriptSource supports ES Modules
The current issues I have:
-
calling
setScriptSourcewith the exact same source =>segmentation fault -
Loading a class from a new import and then extend the existing class =>
#
# Fatal error in , line 0
# Check failed: args[1].IsJSObject().
#
#
#
#FailureMessage Object: 0x7ffeefbf6860
1: 0x1001000d2 node::NodePlatform::GetStackTracePrinter()::$_3::__invoke() [node]
2: 0x100ef507f V8_Fatal(char const*, ...) [node]
3: 0x1006576ee v8::internal::Runtime_LoadFromSuper(int, unsigned long*, v8::internal::Isolate*) [node]
4: 0x1009b0af4 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvInRegister_NoBuiltinExit [node]
5: 0x100a1b0ce Builtins_CallRuntimeHandler [node]
6: 0x10093cabb Builtins_InterpreterEntryTrampoline [node]
7: 0x10093cabb Builtins_InterpreterEntryTrampoline [node]
zsh: illegal hardware instruction
If the new import was previously imported then it works. I can't see any import happening so I can only guess that setScriptSource doesn't trigger module loading if missing.