[Question] - In memory files

[dazinator]

Is it possible to provide the files to madge from an in-memory store rather than it going off to disk?

UPDATE: tracking status of dependencies here before work on madge could begin:

• [x] Precinct - PR https://github.com/dependents/node-precinct/pull/52
• [ ] Cabinet PR - https://github.com/dependents/node-filing-cabinet/pull/68
• [ ] node-dependency-tree

[mrjoelkemp]

Hey! Interesting question. You might be able to use mock-fs and invoke madge programmatically. As in, invoke mockfs then use madge in a node script. I'm not sure how you'd do it via a cli invocation of madge.

[dazinator]

Thanks I will take a look at mock-fs. Other libraries I have integrated with so far, like rollup and requirejs have plugins for this use case. For example: https://github.com/rxaviers/requirejs-memfiles or https://www.npmjs.com/package/rollup-plugin-hypothetical - however if this mock-fs will do the job without any special effort on the part of madge then so much the better :-)

[dazinator]

I tried mock-fs, but the trouble is, once you import mock-fs, it completely overrides all file system access, including to all of node_modules. Therefore when enabling mock-fs, and then invoking madge, madge fails because it can no longer find a bunch of dependencies such as filing-cabinet etc etc. Becasuse all fs access is now pointing at the mocked file system. This would mean I'd have to mock up every dependency..

This probably explains why I haven't seen other libraries use this method :-)

Would you consider extending your API to allow module files to be provided rather than obtained via fs?

[dazinator]

Actually, perhaps there is something that lets me create a virtual filesystem, with in memory files layered on top of physical files. Let me try: https://www.npmjs.com/package/memfs

[dazinator]

@mrjoelkemp Ok so I managed to do some interesting stuff with https://github.com/streamich/unionfs like create a virtual filesystem (object implementing fs API) that is backed by multiple sources such as mem-fs as well as node's physical fs creating a virtual file system. However I have no way to pass this union-fs instance to madge.

mock-fs overrides nodes own fs implementation meaning there is no need to pass an instance to madge - but that has the side effect that all dynamic imports then fail, as node_module dependencies and the like can't be loaded, and this is just too much effort to mock them all up just to be able to supply a couple of module file contents from memory.

I think at this point, the nicest change would be to allow an fs instance to be provided via options and if it is, then madge can use that for file system access rather then node's default fs. This will provide greatest flexibility of using other fs implementations such as union-fs or mem-fs etc etc. What do you think? I can have a go at a PR if this sounds reasonable.

[mrjoelkemp]

Thanks for trying a few solutions. What's the reason for needing the in memory version of the files? Can you provide a bit more context that led to the need?

Curious if it's super custom or a generic use case that we're just encountering. Thanks in advance.

[dazinator]

Ok sure. So for background, I am working on an open source library that asp.net core developers can use (primarily during development) in their web applications, for file pre-processing. It basically can be configured to watch a bunch of files and then process them in some way when they change, by putting them through a configurable pipeline. It runs in proc with the web application, so it's not a seperate tool that you have to start running like gulp etc. Outputs can be generated from the pipeline, and these outputs may in turn trigger further pipelines that are watching for changes to those outputs!

Most of the pipelines I have out of the box so far basically make an RPC style call to an npm process using https://github.com/aspnet/JavaScriptServices to sync any changed files from the asp.net core process, to that npm process (which holds them in memory) and then processes them in npm using the superior libraries available (like rollup) that aren't available for .net core. The results are sent back to the asp.net core process. Some of the files being processed will only ever exist in memory on both the asp.net core side as well as the npm side and the disk may not ever be involved in the equation.

I have done this successfully to process files using things like r.js optimiser, and rollup. The reason I am looking at madge is because I have recently added Hot Module Reload capability. The asp.net core server can detect changed module files, then send a notifications to the browser (via signalR / a websocket connection) where a companion client js library on the page, reloads the module. I can successfully do this with SystemJS to reload a module, because SystemJS registry can be used to compute the dependants of a module, so when reloading the module in the browser, I can also reload its dependants at the same time. However I am now implementing the same thing (trying to) for RequireJS as the module loader on the page. RequireJS isn't as good in this area, i.e it's not possible to compute the dependant modules of the changed module in the browser. So this is where I am looking at madge. The idea is that when the asp.net core server (i.e my library :-)) detects that a module file has changed, the developer can chose to process it with madge on the server side (via JavascriptServices) whcih will compute it's dependants. When madge is in the mix like this, then the signal to reload the module that is sent to the browser will also include a list of the modules dependent modules. This will let the module reloader in the browser, for RequireJS, reload not just the chnaged module, but also its dependants.

I hope that's enough background, sorry if it was rambling a bit. In order for me to achieve this, I need to provide the input to madge from in memory. I can already capture all output from madge in memory as to my knowledge madge doesn't write any files :-)

[mrjoelkemp]

Thanks a ton for the context. That's helpful info to reason about the need.

I think support is required at many layers for this. Madge may have some fs interactions, but the core of it is likely done at deeper layers like node-dependency-tree, node-precinct, and node-filing-cabinet. I'd start reading through dependency-tree and you'll probably need to pass an fs object as an option to the call to precinct.paperwork(). Then submit a patch to node-precinct to use that fs object instead of the default fs import. That may be all that you need in core changes; changes to madge and dependency-tree would be accepting and propagating the fs option down.

Happy to review PRs for dependency-tree all the way down. You'll need @pahen's blessing for the madge proper changes.

As a last part towards your previous workarounds, you can try using https://www.npmjs.com/package/rewire to theoretically replace the fs used all the way down. Maybe worth a try before you have to go deep.

[dazinator]

Thanks a lot for the Rewire suggestion, you are right - if I can get that working it should be a much easier path, I'll give that a go first ;-)

[pahen]

Interesting use case for Madge. I'm gladly accepting a PR to support this option :)

[dazinator]

@mrjoelkemp rewiring the various modules is a bit tricky. Some of the require("fs") calls are themselves in functions, making them hard to replace with rewire (based on my novice skill level anyway!)

So I am starting my journey through the stack like you mentioned, have started with precinct paperwork :-)

[pahen]

Is this something you still work on @dazinator ? Or can we close this issue?

[dazinator]

Still going.. progress stopped for a while whilst waiting on downstream dependency but that's unblocked now so I intend to resume this in the near future.

[pahen]

Ok, cool :)

[iddan]

@dazinator, @pahen is this still being actively worked on? Is there work to be done that can be helped with?

[dazinator]

Hi @iddan - no I am no longer actively working on this. I got quite far with it, including getting a bunch of the underlying dependencies changed to support the feature and then building upon them, with tests.

I was still working on completing node-filing-cabinet however, as per remaining tasks: https://github.com/dependents/node-filing-cabinet/pull/68

Also the file-exists dependency the author never replied. I forked it and made changes to allow a custom fs to be supplied, published my fork to npm and built on that version: https://github.com/dazinator/file-exists

For me, the original use case is not pressing, - would have been a nice to have, but I have now moved on. If someone did want to pick this up, I could provide info, otherwise, it could be dropped.