Wrong behavior of fs. * methods / native modules

[raffis]

Is this a BUG or a FEATURE REQUEST?: Bug

What happened: Context: Im trying to include a module with a native . Node dependency.

This only works if I bundle the file with -r node_modules/foobar/build/Release.Node as far as this is documented in issues. (Note that a readme hint would be nice for this since so far every node app I need to bundle has at least one module with native addons).

Good so far, but if I try to write out the module this will end in strange behavior of fs methods. For example, fs.existsSync("node_modules/foobar/build/Release.Node") will return true even if there is no such file on disk. But node internally wants to require this file from disk. Also fs.stat() will return a valid stat which is strange since there is no file on disk. It also fails with the async methods. Note that fs.existSync also returns true for each folder, for example fs.existsSync("node_modules").

While no such file is in the -r list it works.

Im not sure if this is expected behavior but it is incredibly difficult to debug such methods if they act different to the usual node fs module. Is there an option to use the original node fs like electrons original-fs?

What you expected to happen: I would expect that those methods fail/return false since there is no such file on disk. Id like to see a original-fs module. And a big warning in the readme that fs methods may act very strangely (not just fs.readFileSync() and fs.readFile()).

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?: Is it the only way to include native modules and then write them out to the current node_modules folder using fs. readFileSync() and writeFile? Or is there another option? For example Id like to write them to $home/. myapp/nativemod.Node and not to the current directory of the binary. (Besides forking the module and change the require path manually which works perfectly but for sure isnt the best option). Writing native modules to the current directory is quite an unsatisfying option since this needs to be done everytime the binary gets moved. And also can't be done if the binary is located somewhere the user does not have write access to.

Environment

• Platform(OS/Version): Linux Mint 19
• Host Node Version: v8.15.0
• Target Node Version: v8.15.0
• Nexe version: latest, 3.2.0
• Python Version: (not sure im filing this issue from my phone :/)

[mrexodia]

Still cannot reproduce it. I used this color scheme to test.

[mrexodia]

Okay, as noted can be reproduced (with default color settings) with the following settings:

[EricMCornelius]

There's something very funny going on with the build in virtual filesystem. Here's a basic reproduction script:

#!/bin/bash

echo "const find = require('shelljs').find;
console.log(process.cwd());
console.log(find('build').slice(0));" > test.js
npm install shelljs
mkdir build
touch build/test.txt
mkdir build/subdir
touch build/subdir/test.txt
npx nexe -i test.js -o build/test -t node-12.0.0-linux-x64

echo 'Running script via node runtime...'
node test.js

echo 'Running script via nexe build...'
./build/test

Output:

+ [email protected]
added 16 packages from 11 contributors and audited 19 packages in 0.407s
found 0 vulnerabilities

ℹ nexe 3.2.1
✔ Already downloaded...
✔ Compiling result
✔ Entry: 'test.js' written to: build/test
✔ Finished in 0.456s
Running script via node runtime...
/home/ecornelius/Repositories/nexe_tmp
[
  'build',
  'build/subdir',
  'build/subdir/test.txt',
  'build/test',
  'build/test.txt'
]
Running script via nexe build...
/home/ecornelius/Repositories/nexe_tmp
[ 'build', 'build/node_modules', 'build/test.js' ]

Shouldn't we expect these to be the same, given there are no bundled resources?

[kaysersoze]

I believe I'm running into this issue as well. I have a module (https://github.com/kaysersoze/node-printer) that has native .node dependencies to interface with the local machine's printing subsystem.

When trying to bundle into an executable (both on OS X and Windows), however, I kept getting errors like the below:

internal/modules/cjs/loader.js:730
  return process.dlopen(module, path.toNamespacedPath(filename));
                 ^

Error: dlopen(<nexe-executable-directory>/node_modules/printer/lib/precompiled/node_printer_64_darwin_x64.node, 1): image not found
    at Object.Module._extensions..node (internal/modules/cjs/loader.js:730:18)
    at Module.load (internal/modules/cjs/loader.js:600:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:539:12)
    at Function.Module._load (internal/modules/cjs/loader.js:531:3)
    at Module.require (internal/modules/cjs/loader.js:637:17)
    at require (internal/modules/cjs/helpers.js:22:18)
    at Object.<anonymous> (<nexe-executable-directory>/node_modules/printer/lib/printer.js:13:22)
    at Module._compile (internal/modules/cjs/loader.js:701:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:712:10)
    at Module.load (internal/modules/cjs/loader.js:600:32)

I tried adding the .node file as a resource in my nexe build script to no avail. I also tried bundling sets of pre-compiled .node files in the module, but got the same result.

What I ended up doing was rewriting the module to accept an external reference to a .node file on the local filesystem - just needed a quick-and-dirty way to get something working, since I had to fork the module for support anyway.

While that worked, I would love to have a way to build in and reference those .node dependencies into my nexe executable.

[raffis]

I tried adding the .node file as a resource in my nexe build script to no avail. I also tried bundling sets of pre-compiled .node files in the module, but got the same result.

What I ended up doing was rewriting the module to accept an external reference to a .node file on the local filesystem - just needed a quick-and-dirty way to get something working, since I had to fork the module for support anyway.

While that worked, I would love to have a way to build in and reference those .node dependencies into my nexe executable.

I basically do the same now but instead forking the module (which I don't have to in my case) I do a search & replace during the build. In my case I need fuse-bindings which itself has a native requirement . (And on binary start I may write the native dependency to the users home folder):

const nexe = require('nexe');
const fs = require('fs');
const replace = require('replace-in-file');
const promisify = require('util').promisify;
const copyFile = promisify(fs.copyFile);
const replaceOptions = {
  files: "node_modules/@gyselroth/balloon-node-fuse/node_modules/fuse-bindings/index.js",
  from: "var fuse = require('node-gyp-build')(__dirname)",
  to: "var fuse = require(require('path').join(require('os').homedir(), '.mount.balloon', 'fuse_bindings.node'))",
};

(async () => {
  await copyFile(replaceOptions.files, replaceOptions.files+'.orig');
  await replace(replaceOptions);

  var compileOptions = {
    logLevel: 'verbose',
    input: 'build/main.js',
  };

  switch(process.platform) {
    case 'win32':
      compileOptions.target = 'windows-x64-8.15.0';
      compileOptions.output = 'dist/mount.balloon-win-x64';
      compileOptions.resources = [
        'node_modules/@gyselroth/balloon-node-fuse/node_modules/fuse-bindings/build/Release/fuse_bindings.node'
      ];
    break;

    case 'darwin':
    break;

    case 'linux':
    default:
      compileOptions.target = 'linux-x64-8.15.0';
      compileOptions.output = 'dist/mount.balloon-linux-x64';
      compileOptions.resources = [
        'node_modules/@gyselroth/balloon-node-fuse/node_modules/fuse-bindings/prebuilds/linux-x64/node-57.node'
      ];
  }


  nexe.compile(compileOptions);
})();

This is working for now. But still the behaviour of the nexe fs has to be documented better and access to the original fs will be appreciated. (Also a nice built in way to support native modules).

[aheidelberg]

I'm getting the same error as @kaysersoze (with sqlite3 though):

  return process.dlopen(module, path._makeLong(filename));
                 ^

Error: The specified module could not be found.
\\?\C:\Users\Alex\Desktop\node_modules\sqlite3\lib\binding\node-v57-win32-x64\node_sqlite3.node

I've included node_modules\sqlite3\lib\binding\node-v57-win32-x64\node_sqlite3.node with -r. When I put the file next to the binary on the actual file system it's working though.

[nmarus]

I have been fighting this issue for a few weeks. I have attempted some of the workarounds mentioned above as well a in other issue threads (#335, #396, #430, #440, #544, #586). My workaround was to move the native module outside of the binary (and outside of any matches paths referenced in the resources option of nexe build). I do this at runtime, then require from that location, and once the binary exits, delete it.

This may be fixed in nexe@next or in version 4 when it drops, but on 3.x this works very well and does not require forking or find/replace strings in any of the modules. I created a library that does this for you. Hopefully it helps others having similar issues.

https://github.com/nmarus/nexe-natives