How to import/require an "external" (to the pkged archive) node module?

[webhype]

Discussed in https://github.com/vercel/pkg/discussions/1721

^{Originally posted by webhype August 3, 2022} I am trying to pkg a project on my Mac, but for a Linux target.

The project contains the npm module fs-ext ( https://www.npmjs.com/package/fs-ext ) which relies on some low-level C++ code that needs to be compiled into fs-ext.node on the proper target system for it to work.

npm install fs-ext compiles that C++ code, but the resulting fs-ext.node byte code from my Mac won't run on Linux and vice-versa so I want to "externalize" fs-ext, i.e. keep it outside of my pkged Linux project and install it "manually" into the node_modules folder of the pkged Linux project, after I upload the main executable.

I am aware of the general advice given here that the Linux version of a pkg should be compiled on Linux, the Mac OS version should be compiled on Mac OS and so on. Just in this case that's not a practical option. I'd prefer to keep system-specific .node files, and their "owning" node_modules "outside" of my pkged file, if that's possible.

The goal is to have file hierarchy like this on the Linux target:

+ my-project-executable
|
+ node_modules 
    |
    +--+ fs-ext
           |
           +--+ build
                  |
                  +--+ Release
                         |
                         +--- fs-ext.node

Then, the idea is to require fs-ext "dynamically" in my code:

const {flockSync} = require(path.join(process.cwd(), "node_modules", "fs-ext", "fs-ext.js"));

My hope was that this will "reach outside of" the pkg sandbox and execute the fs-ext.js code, which in turn will use is C++ compiled code fs-ext.node as it sees fit.

My pkg command looks like this:

$ pkg . -t node14-macos-x64,node14-linux-x64 --public-packages "fs-ext"

I copy the compiled code from my Mac onto the Linux box to /var/my-pkged-project/ and there I execute npm install fs-ext, which generates the proper fs-ext.node file for Linux.

However, when I execute the pkged file, it says it can't find fs-ext.js:

pkg/prelude/bootstrap.js:1930                                                                                                                                                                                                                                                                    
      return wrapper.apply(this.exports, args);                                                                                                                                                                                                                                                  
                     ^                                                                                                                                                                                                                                                                           
                                                                                                                                                                                                                                                                                                 
Error: Cannot find module '/var/my-pkged-project/node_modules/fs-ext/fs-ext.js' 

Strangely, the file is very much there, it exists at that path.

What am I doing wrong? Can compiled and not-compiled code even be mixed like that? Am I not understanding the pkg file system sandbox correctly? Am I using the --public-packages command-line option incorrectly?