"Cannot find module" when running pkg-created binary of project using yarn workspaces

[lourd]

I'm trying to use pkg with this project, https://github.com/donmccurdy/glTF-Transform.

After setting up the project, running pkg, and then running the created binary, I get an error about not being able to find the first module that's required. I'm using pkg 5.3.3 on macOS 11.4 with Node.js 16.10.0 and Yarn 1.22.10. Here's the repro steps:

git clone [email protected]:donmccurdy/glTF-Transform.git
cd glTF-Transform
yarn && yarn dist
pkg ./packages/cli/bin/cli.js
./cli-macos --help

And here's the error

pkg/prelude/bootstrap.js:1740
      throw error;
      ^

Error: Cannot find module 'gl-matrix/vec3'
Require stack:
- /snapshot/glTF-Transform/packages/core/dist/core.js
- /snapshot/glTF-Transform/packages/cli/dist/cli.js
- /snapshot/glTF-Transform/packages/cli/bin/cli.js
1) If you want to compile the package/file into executable, please pay attention to compilation warnings and specify a literal in 'require' call. 2) If you don't want to compile the package/file into executable and want to 'require' it from filesystem (likely plugin), specify an absolute path in 'require' call using process.cwd() or process.execPath.
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
    at Function._resolveFilename (pkg/prelude/bootstrap.js:1819:46)
    at Function.Module._load (node:internal/modules/cjs/loader:778:27)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at Module.require (pkg/prelude/bootstrap.js:1719:31)
    at require (node:internal/modules/cjs/helpers:94:18)
    at Object.<anonymous> (/snapshot/glTF-Transform/packages/core/dist/core.js:1:77)
    at Module._compile (pkg/prelude/bootstrap.js:1794:22)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/snapshot/glTF-Transform/packages/core/dist/core.js',
    '/snapshot/glTF-Transform/packages/cli/dist/cli.js',
    '/snapshot/glTF-Transform/packages/cli/bin/cli.js'
  ],
  pkg: true
}

But when I run pkg with the debug option, I see that module being included

Partial debug log

...
> [debug] The directory files list was included (2 items)
  /Users/louis/code/glTF-Transform/node_modules/gl-matrix
> [debug] The directory files list was included (2 items)
  /Users/louis/code/glTF-Transform/node_modules/gl-matrix
> [debug] files & folders deduped =
  index.js
  vec3.js
  mat4.js
  common.js
  mat2.js
  mat2d.js
  mat3.js
  quat.js
  quat2.js
  vec2.js
  vec4.js
> [debug] The directory files list was included (11 item)
  /Users/louis/code/glTF-Transform/node_modules/gl-matrix/cjs
> [debug] The directory files list was included (11 item)
  /Users/louis/code/glTF-Transform/node_modules/gl-matrix/cjs
> [debug] files & folders deduped =
  packages
  node_modules
...

Is there something special I need to do for projects using yarn workspaces?

Running the cli.js file with Node directly yields the correct results:

node ./packages/cli/bin/cli.js --help

Thanks for your help!

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove the stale label or comment or this will be closed in 5 days. To ignore this issue entirely you can add the no-stale label

[lourd]

Still open!

[chx-code]

Encounter the same issue

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove the stale label or comment or this will be closed in 5 days. To ignore this issue entirely you can add the no-stale label

[lourd]

Still open

[information-security]

I am not sure if this is the same problem as the one that the OP mentioned. However, I am experiencing a similar issue with Yarn PnP (Plug'n'Play). Using --options didn't help:

yarn pkg -t node16-linux-x64 --options require=./.pnp.cjs dist/app.js

Trying to use --options, I get tons of the following warnings:

> Warning Failed to make bytecode node16-x64 for file /snapshot/.yarn/cache/ansi-regex-npm-5.0.1-c963a48615-2aa4bb54ca.zip/node_modules/ansi-regex/index.js
node:internal/modules/package_json_reader:4
const internalModuleReadJSON = function (f) { return require('fs').internalModuleReadJSON(f); };
                                                                   ^

TypeError: require(...).internalModuleReadJSON is not a function
    at internalModuleReadJSON (node:internal/modules/package_json_reader:4:68)
    at Object.read (node:internal/modules/package_json_reader:21:42)
    at readPackage (node:internal/modules/cjs/loader:296:36)
    at readPackageScope (node:internal/modules/cjs/loader:329:19)
    at trySelf (node:internal/modules/cjs/loader:444:40)
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:910:24)
    at Function.Module._load (node:internal/modules/cjs/loader:778:27)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at Module._preloadModules (node:internal/modules/cjs/loader:1282:12)
    at loadPreloadModules (node:internal/bootstrap/pre_execution:545:5)

The resulting binary file also fails at the first line of code that tries to import/require something from packages downloaded by YarnPnP. I also made sure that the content of the .yarn directory is included in the package using the assets portion of the config (package.json) file.

Here is my config:

    "pkg": {
        "scripts": [
            "dist/server.js",
            "dist/app.js",
            "dist/config/**/*.js",
            "dist/controllers/**/*.js",
            "dist/lib/**/*.js",
            "dist/package.json"
        ],
        "assets": [
            ".pnp.cjs",
            ".pnp.loader.mjs",
            ".yarn/**/*.js"
        ],
        "targets": [
            "node16-linux-x64"
        ],
        "options": [
            "require=./.pnp.cjs"
        ]
    },

No matter whether I use the config file or the pkg arguments, I get the same errors. If I remove the --options, then the library is packaged successfully but it will fail when it tries to import/require any of the packages managed by Yarn.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove the stale label or comment or this will be closed in 5 days. To ignore this issue entirely you can add the no-stale label

[lourd]

Still valid

[pcfreak30]

I can not say if this is 100% the same, but running yarn pkg -c pkg.json build/index.js -t linux --public --no-native-build -C gzip

I get Warning Cannot find module on the build, then Cannot find module on runtime, but they are different.

As I am assuming pkg doesn't support PNP completely (or at all?) I would appreciate this getting some priority. Was hoping to use this to improve my builds :(.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove the stale label or comment or this will be closed in 5 days. To ignore this issue entirely you can add the no-stale label

[lourd]

Still valid

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove the stale label or comment or this will be closed in 5 days. To ignore this issue entirely you can add the no-stale label

[pcfreak30]

Still valid

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove the stale label or comment or this will be closed in 5 days. To ignore this issue entirely you can add the no-stale label

[pcfreak30]

Still valid

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove the stale label or comment or this will be closed in 5 days. To ignore this issue entirely you can add the no-stale label

[lourd]

Still valid