Allow supplying our own `fs`

[haykam821]

Currently, this module uses the built-in fs module. However, it would be useful to be able to provide our own replacement using an option. We can check if it is compatible enough by checking if all the fs methods that this module uses exist, and throw an error if not all of them are included.

[thom4parisot]

Any hint or reason why?

[haykam821]

@oncletom Sorry for not explaining why. In my project's build process, I webpack the source and generate a manifest. I was attempting to use memory-fs, but since you cannot supply an alternative fs, I must write the files temporarily in order to execute my final step, which is packing the Chrome extension.

[thom4parisot]

Hm, so if I understand well, you would like crx operations (in the context of webpack), to access the resources you previously stored with memory-fs?

Isn't it solved by using the resolve.alias webpack directive? https://webpack.js.org/configuration/resolve/

It looks like you could say fs is aliased as memory-fs, without having to alter crx codebase. I only read the documentation so maybe I misunderstood the intent of this setting.

[haykam821]

@oncletom No, I use this module on the bundle generated by webpack. The build script is regular Node.js.

[thom4parisot]

I'm not sure to understand when you use memory-fs, is it in the build script, or in the bundle generated by webpack?

To confirm, you use crx as part of the build script, in order to package the webpack bundle as a browser extension?

[haykam821]

@oncletom Yes, I am trying to use memory-fs as a custom output filesystem for the bundle generated by webpack. However, since I cannot use memory-fs as the input filesystem for crx, I must temporarily store the bundle so that crx can pack it.

[thom4parisot]

OK, thanks for the link to webpack documentation. The example sounds clear and concise.

By reading it, I understand memory-fs applies to the webpack configuration, but I struggle to understand the impact on the packed bundle. I struggle to understand how they can share the same space of memory-fs. Do you have an example of that in the wild?

Thanks for being patient. I have never used webpack, and I'm aware it may put quite some effort to explain all these stuff to me. Thanks for doing it 👍

[haykam821]

@oncletom The output bundle.js from webpack is exactly the same, except since we are using memory-fs instead of fs, it will be stored in memory and not on the filesystem. The proposal here is that we can continue to use memory-fs and pack bundle.js and a generated manifest.json into a Chrome extension.

I don't know of anything that uses this in the wild, but here is the simplified part of my build script:

const compiler = wp({
	entry: "./src",
	output: {
		filename: "bundle.js",
		path: "./bundled",
	},
});

// set the output of the webpack compiler to the fake/memory filesystem
compiler.outputFileSystem = instanceOfMemoryFS;

compiler.run((err, info) => {
    if (err) {
        process.exit();
    } else {
        // so here's the problem: we have no option to load files from the fake/memory filesystem
        // because of this, we can't use memory-fs
        (new crx({})).load("./bundled").then(loaded => {
            return loaded.pack();
        }).then(buffer => {
            // now, write the extension to the regular filesystem
            fs.writeFile("./dist/chrome.crx", buffer);
        });
    }
});

[thom4parisot]

Okay, I see what you mean. It is clear to me now. Thank you for the code snippet :-)

I dug a bit in the code and most of the I/O operations are performed by archiver, which inflates the files. So the problem would reflect on this module too.

I need to think more about how to design something nice.

[thom4parisot]

A way to achieve this is documented here https://github.com/archiverjs/node-archiver/issues/216