Resolve .node native addon on Linux

[andreiglingeanu]

Hi,

It seems that resolving .node native addons is currently broken on Linux. The issue is, *.node file doesn't get properly copied over into /tmp/pkg. I've tried looking into /tmp/pkg but unfortunately this folder doesn't even exist.

Error: /tmp/pkg/dd5efc0c4d629d7af09e14d00e15cb72c99a5d2d0ad653954717b5199cea8f0a/a52501562f36b5439b9e894001086631.node: cannot open shared object file: No such file or directory
    at Object.eval (webpack:///./build/Release/test.node?:5:9)
    at eval (webpack:///./build/Release/test.node?:8:30)
    at Object../build/Release/test.node (/snapshot/project/daemon/daemon/daemon-implementation-bundled.js:0:0)
    at __webpack_require__ (/snapshot/project/daemon/daemon/daemon-implementation-bundled.js:0:0)
    at eval (webpack:///./src/core/features/test/lib-handler.ts?:2:81)
    at Module../src/core/features/test/lib-handler.ts (/snapshot/project/daemon/daemon/daemon-implementation-bundled.js:0:0)
    at __webpack_require__ (/snapshot/project/daemon/daemon/daemon-implementation-bundled.js:0:0)
    at eval (webpack:///./src/core/index.ts?:2:83)
    at Module../src/core/index.ts (/snapshot/project/daemon/daemon/daemon-implementation-bundled.js:0:0)
    at __webpack_require__ (/snapshot/project/daemon/daemon/daemon-implementation-bundled.js:0:0)

The *.node file is properly present in the snapshot filesystem because I can observe it on runtime with DEBUG_PKG=1:

/snapshot                                                                                                                                                                                                                                                                                 
  project.                                     197773                                                                                                                                                                                                                                        
    daemon                                   84088                                                                                                                                                                                                                                        
      daemon                                 84088                                                                                                                                                                                                                                        
        daemon-linux.js                       5072                                                                                                                                                                                                                                        
        daemon-implementation-bundled.js     18456                                                                                                                                                                                                                                        
        a52501562f36b5439b9e894001086631.node     60560                                                                                                                                                                                                                                   
    node_modules                            197773                                                                                                                                                                                                                                        
      optimist                               16896                                                                                                                                                                                                                                        
        package.json                           889                                                                                                                                                                                                                                        
        index.js                              9505                                                                                                                                                                                                                                        
        node_modules                         16896                                                                                                                                                                                                                                        
          minimist                            6502                                                                                                                                                                                                                                        
            package.json                       935                                                                                                                                                                                                                                        
            index.js                          5567                                                                                                                                                                                                                                        

And pkg is able to detect the fact that the .node file is present in the snapshot but for some reason the copying itself into /tmp/pkg/* is failing. For this reason, process.dlopen(module, __dirname + "/a52501562f36b5439b9e894001086631.node") throws the above error.

Any ideas what could have caused the assets copying from snapshot filesystem to /tmp/pkg/* to fail? I'm inclined to think this is a Linux specific issue but I haven't tried yet on a different platform -- but will do so if this can yield any results.

Please don't mind the webpack reference, I'm editing manually webpack's output to ensure we're not dealing here with a cross-tool problem.

Any help is appreciated!

[vitalygashkov]

Similar issue with macOS.

pkg/prelude/bootstrap.js:1740
      throw error;
      ^

Error: dlopen(/var/folders/8t/51sl9j753rgdl0rs25tlk_7r0000gn/T/pkg/5acc190a5370a4e7d0ac5e4f52914907993564555846c68f2c5a1ef6fd9d5be4/addon.node, 0x0001): tried: '/var/folders/8t/51sl9j753rgdl0rs25tlk_7r0000gn/T/pkg/5acc190a5370a4e7d0ac5e4f52914907993564555846c68f2c5a1ef6fd9d5be4/addon.node' (no such file), '/usr/local/lib/addon.node' (no such file), '/usr/lib/addon.node' (no such file)
    at process.dlopen (pkg/prelude/bootstrap.js:2114:28)
    at Object.Module._extensions..node (node:internal/modules/cjs/loader:1183:18)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at Module.require (pkg/prelude/bootstrap.js:1719:31)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/snapshot/app/lib/addon-adapter/lib/addon-adapter.js)
    at Module._compile (pkg/prelude/bootstrap.js:1794:22)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10) {
  code: 'ERR_DLOPEN_FAILED'
}

[robertsLando]

Actually dlopen is patched here. I suggest you to try checking there why the .node file isn't find

[franklevasseur]

As @vitnore said, I'm having the exact same issue on macos.

The error is: dlopen(/var/folders/mb/q2c7_tgs3bd0lls6xzxrw5zm0000gn/T/pkg/791846661d83ae7447035d77e00e14986ad5a3ae23064a97e32d5bf408e18acf/fasttext.node, 1): image not found

I checked and the directory /var/folders/mb/q2c7_tgs3bd0lls6xzxrw5zm0000gn/T/pkg does not exist.

The issue didn't exist in pkg ^4.5.1 and only appeared when switching to ^5.4.1 (I have not checked versions in between).

[franklevasseur]

Reverting to ~5.2.0 solves the issue

[maxpavlov]

Hehe, @franklevasseur thank you. Indeed reverting to a version where .node assets need to be explicitly specified as assets fixes the problem. I now think that since 5.3.3 pkg became starter in a sense that it started to recognize .node deps automatically, but it cast itself incapable of actually embedding them properly. And it's not even the deduped part that is broken with 5.3.3, it doesn't work in neither cases where .node are specified as assets, nor in cases when only auto-detected.

[luckyyyyy]

#1321 have this problem

[robertsLando]

@maxpavlov @franklevasseur I'm using pkg everyday without any problem I always work with .node files, I was having many problems with previous version and everything was fixed after the new releases. What I suggest to you is to create a sample project that reproduces the issue and add some console log to pkg bootstrap to see where the problem is. Also do you actually specify node addons files in assets?

[franklevasseur]

Hey @robertsLando! Sorry for the long delay, I was taking few weeks off coding.

Reverting to ~5.2.0 solves the issue for me. If ever I feel the need to jump to a more recent version and still observe the bug, I'll create a sample project that reproduces the bug and leave it here.

[s-a]

Affects Windows 10 too @5.2.0 works.

[robertsLando]

Does 5.3.0 or 5.3.2 works?

[luckyyyyy]

Does 5.3.0 or 5.3.2 works?

5.2.0 No problem

#1321 have this problem

Unzip the .node file to tmp directory, There is a possibility of losing files.

Mac win linux has this problem

[s-a]

Yep it seemed to work with latest version but temp files randomly are victims of operating systems garbage collectors so it does not work anymore at some point.

[robertsLando]

#1321 have this problem

@luckyyyyy that PR didn't changed the location where .node files are copied, it just add the support to copy the entire package folder instead of just the addon file in order to prevent errors in modules like sharp that has some relative lib inclusion. I'm using it in a project using sharp serialport levelup and many other .node addons without any problem (in linux)

[ghost]

As @vitnore said, I'm having the exact same issue on macos.

The error is: dlopen(/var/folders/mb/q2c7_tgs3bd0lls6xzxrw5zm0000gn/T/pkg/791846661d83ae7447035d77e00e14986ad5a3ae23064a97e32d5bf408e18acf/fasttext.node, 1): image not found

I checked and the directory /var/folders/mb/q2c7_tgs3bd0lls6xzxrw5zm0000gn/T/pkg does not exist.

The issue didn't exist in pkg ^4.5.1 and only appeared when switching to ^5.4.1 (I have not checked versions in between).

Same problem here, I am using Ubuntu 20 and had to downgrade to 5.3.2

[kaichaosun]

Same issues occurs in "pkg": "^5.5.2" on MacOS when running packaged binary a few times, error info,

pkg/prelude/bootstrap.js:1740
      throw error;
      ^

Error: dlopen(/var/folders/bs/v36xf6297qj5zt1h09cj4_c80000gn/T/pkg/e0ffe9880ab98dabdb5b24a7eb052d50bd2f4e92a5ad39f040f190c75e17e0bb/leveldown/prebuilds/d
arwin-x64+arm64/node.napi.node, 0x0001): tried: '/var/folders/bs/v36xf6297qj5zt1h09cj4_c80000gn/T/pkg/e0ffe9880ab98dabdb5b24a7eb052d50bd2f4e92a5ad39f040f
190c75e17e0bb/leveldown/prebuilds/darwin-x64+arm64/node.napi.node' (no such file), '/usr/local/lib/node.napi.node' (no such file), '/usr/lib/node.napi.no
de' (no such file), '/private/var/folders/bs/v36xf6297qj5zt1h09cj4_c80000gn/T/pkg/e0ffe9880ab98dabdb5b24a7eb052d50bd2f4e92a5ad39f040f190c75e17e0bb/leveld
own/prebuilds/darwin-x64+arm64/node.napi.node' (no such file), '/usr/local/lib/node.napi.node' (no such file), '/usr/lib/node.napi.node' (no such file)
    at process.dlopen (pkg/prelude/bootstrap.js:2114:28)
    at Object.Module._extensions..node (node:internal/modules/cjs/loader:1183:18)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at Module.require (pkg/prelude/bootstrap.js:1719:31)
    at require (node:internal/modules/cjs/helpers:102:18)
    at load (/snapshot/ad4m-host/node_modules/node-gyp-build/index.js:21:10)
    at Object.<anonymous> (/snapshot/ad4m-host/node_modules/leveldown/binding.js:1:105)
    at Module._compile (pkg/prelude/bootstrap.js:1794:22) {
  code: 'ERR_DLOPEN_FAILED'
}

When I remove the folder /var/folders/bs/v36xf6297qj5zt1h09cj4_c80000gn/T/pkg/, it works again, but after a few run, error happens again.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove the stale label or comment or this will be closed in 5 days. To ignore this issue entirely you can add the no-stale label

[kaichaosun]

I still encounter this issue..

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove the stale label or comment or this will be closed in 5 days. To ignore this issue entirely you can add the no-stale label

[s-a]

pls dont close

[Waylawww]

我也有这个问题

[tqiqbal]

Same issue on version 5.8.0 on Linux (Windows works fine, Ubuntu 22.04.1 LTS works fine, MacOs didn't try)

While building the executable I was getting this warning: Warning Failed to make bytecode node16-x64 for file /snapshot/tiqbal/mockgee/node_modules/xmllint/xmllint.js

Below is the error while running the executable:

pkg/prelude/bootstrap.js:1876 throw error; ^

Error: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /tmp/pkg/763445c9f165bedfbec25b0c6de2732fd7a64168ed8988af69d46b1075e4c61c/libxmljs2/build/Release/xmljs.node) at process.dlopen (pkg/prelude/bootstrap.js:2255:28) at Object.Module._extensions..node (node:internal/modules/cjs/loader:1189:18) at Module.load (node:internal/modules/cjs/loader:981:32) at Function.Module._load (node:internal/modules/cjs/loader:822:12) at Module.require (node:internal/modules/cjs/loader:1005:19) at Module.require (pkg/prelude/bootstrap.js:1855:31) at require (node:internal/modules/cjs/helpers:102:18) at bindings (/snapshot/tiqbal/mockgee/node_modules/bindings/bindings.js:112:48) at Object. (/snapshot/tiqbal/mockgee/node_modules/libxmljs2/lib/bindings.js:1:99) at Module._compile (pkg/prelude/bootstrap.js:1930:22) { code: 'ERR_DLOPEN_FAILED' }

Note: this works fine on Ubuntu 22.04.1 LTS