ES modules not supported

[LinusU]

I'm getting the following error as soon as the compiled app boots:

node:internal/modules/cjs/loader:930
  throw err;
  ^

Error: Cannot find module '/snapshot/dhjaks/index.js'
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:927:15)
    at Function._resolveFilename (pkg/prelude/bootstrap.js:1776:46)
    at Function.Module._load (node:internal/modules/cjs/loader:772:27)
    at Function.runMain (pkg/prelude/bootstrap.js:1804:12)
    at node:internal/main/run_main_module:17:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Here is a minimal reproducible example:

package.json

{ "type": "module" }

index.js

import os from 'os'

console.log(os.arch())

Build command:

pkg index.js

[sartaj-sphp]

You didn't post your complete package.json file. So i think you didn't add your sub folders as scripts. For example if you want to add dhjaks folder as script folder then you need to add this in package file like. { "name": "mdm5", "version": "1.0.1", "description": "MDM", "main": "start.js", "bin": "start.js", "scripts": { "start": "node ." }, "pkg": { "scripts": [ "dhjaks/*.js" ], "assets": [], "targets": [ "node12", "linux-x64", "macos-x64", "win-x64" ] }, "author": "demo", "license": "ISC", "dependencies": { } } assume dhjaks is subfolder under pacakge file parent folder. use build command pkg ./package.json

[LinusU]

@sartaj-singh I actually did post my complete package.json file ☺️

The dhjaks folder is the folder of my entire package. My entire package only has two files: package.json & index.js. The goal is for it to just print out one line and then exit.

I did it like this to make a minimal test case that shows the problem.

---

I now tried to use pkg package.json instead:

$ mkdir foobar
$ cd foobar
$ echo '{ "name": "test", "bin": "index.js", "type": "module" }' > package.json
$ echo 'import os from "os"' > index.js
$ echo 'console.log(os.arch())' >> index.js
$ npx pkg package.json
> [email protected]
> Warning Failed to make bytecode node16-arm64 for file /snapshot/foobar/index.js

$ ./test
node:internal/modules/cjs/loader:930
  throw err;
  ^

Error: Cannot find module '/snapshot/foobar/index.js'
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:927:15)
    at Function._resolveFilename (pkg/prelude/bootstrap.js:1776:46)
    at Function.Module._load (node:internal/modules/cjs/loader:772:27)
    at Function.runMain (pkg/prelude/bootstrap.js:1804:12)
    at node:internal/main/run_main_module:17:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

[sartaj-sphp]

Check this line:- Warning Failed to make bytecode node16-arm64 for file /snapshot/foobar/index.js
don't use npm or npx, pkg can run independently. Try to set target in pacakge file like:- "targets": [ "node12", "linux-x64", "macos-x64", "win-x64" ]

[LinusU]

don't use npm or npx, pkg can run independently.

npx is just a way to install & run the package without clobbering your global installs. That is not what's causing problems here since I have tried it with a locally installed version of pkg as well.

Try to set target in pacakge file like

Setting the targets doesn't change anything, I've tried different targets and even running on different platforms...

It does however run if I don't use "type": "module", and use require instead of import, so this issue is clearly related to that.

[sartaj-sphp]

I always create const with require. import statement may be not supported by pkg. I think no need type=module if you compile stand alone executable.

[LinusU]

import statement may be not supported by pkg

If it isn't, then this is a feature request

I think no need type=module if you compile stand alone executable.

I need it because I need to import packages which are ESM-only

[webhype]

Hi, could someone clarify clearly on the home page (README.md) whether or not pkg supports ESMs (ES modules) at all? I know it's a free open-source labor-of-love project so I am not demanding anything. It is what it is and it is appreciated as-is. Just would like a clear positioning so we don't need to waste our time trying to package "type": "module" projects, if that's not supported at all. ESMs are not exactly a new invention so a one-liner positioning in the docs would be helpful. If ESM packaging is hopeless with pkg, does anyone know of a workaround (other than rewriting all your code back into CommonJS)? Cheers!

[CleyFaye]

There is the option of using a barebone webpack config to create a single JS file containing all dependencies and not having any external import. Something like this:

const config = {
  mode: "production",
  entry: "./src/main.ts",
  target: "node",
  output: {
    path: resolve(__dirname, "build", "lib"),
    chunkFormat: "commonjs",
  },
};

The output is then usable with pkg.

It should also be possible to update pkg to support ESM; last time I checked I saw two main issues, the babel configuration used (which can be either completely dropped or updated to support module input with a single change), and bytecode generation that failed. Since I already knew of the webpack option I gave up, but fixing bytecode generation with ESM should be doable since node now have full support for it.

[robertsLando]

For anyone interested I suggest you to firstly use ncc to compile your modules and then use pkg to compile them into executable. There is already an open feature request to include ncc in pkg, maybe with an option

[CleyFaye]

That was what we were doing until a recent update of ncc added compatibility with module-based source. It now produce files that pkg can't use; I could restore the build setup to get the actual error message if needed, but it was something along the line of not handling import statement that were indeed found in the output of ncc.

[robertsLando]

added compatibility with module-based source

Cannot this be disabled with an option?

[CleyFaye]

Not with an option, sadly. But in the end, ncc basically wraps webpack, hence our solution above. I'm not sure which of the two tools should change, but as it is some features of pkg are simply not used (bundling packages, detecting __dirname, etc.). Still the main feature works perfectly, so it's not so bad.

[robertsLando]

By double checking the code seems import statements should be supported: https://github.com/vercel/pkg/blob/59125d1820cdb380f3f592604bcfd7c017495e77/lib/detector.ts#L258

Maybe something isn't working as expected

[robertsLando]

I tried to look into this but haven't find the root cause, the build process seems to work as the import statement is recognized correctly but then the produced binary isn't working 🤷🏼‍♂️

[CleyFaye]

The exact issue, on a very minimalist project:

• have "type":"module" and "bin":"main.js" in package.json
• have import fs from "fs"; in main.js
• run pkg .

It will output this:

> [email protected]
> Targets not specified. Assuming:
  node16-linux-x64, node16-macos-x64, node16-win-x64
> Warning Failed to make bytecode node16-x64 for file /snapshot/t/main.js
> Warning Failed to make bytecode node16-x64 for file /snapshot/t/main.js
> Warning Failed to make bytecode node16-x64 for file C:\snapshot\t\main.js

And the binaries are unusable:

node:internal/validators:119                                                                                                                                               
    throw new ERR_INVALID_ARG_TYPE(name, 'string', value);                                                                                                                 
    ^                                                                                                                                                                      
                                                                                                                                                                           
TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received null                                                                                
    at new NodeError (node:internal/errors:371:5)                                                                                                                          
    at validateString (node:internal/validators:119:11)                                                                                                                    
    at Object.basename (node:path:1309:5)                                                                                                                                  
    at Error.<anonymous> (node:internal/errors:1462:55)                                                                                                                    
    at getMessage (node:internal/errors:421:12)                                                                                                                            
    at new NodeError (node:internal/errors:348:21)                                                                                                                         
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1128:19)                                                                                            
    at Module.load (node:internal/modules/cjs/loader:981:32)                                                                                                               
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)                                                                                                     
    at Function.runMain (pkg/prelude/bootstrap.js:1804:12) {                                                                                                               
  code: 'ERR_INVALID_ARG_TYPE'                                                                                                                                             
}

Removing "type":"module" and altering the file to use require() produce a working build (but is not acceptable on a large codebase). Removing "type":"module" while keeping import statement won't work: error while generating bytecode, and the binary output:

(node:288927) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
(Use `t-linux --trace-warnings ...` to show where the warning was created)
/snapshot/t/main.js:1
import fs from "fs";
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at Object.compileFunction (node:vm:354:18)
    at wrapSafe (node:internal/modules/cjs/loader:1031:15)
    at Module._compile (node:internal/modules/cjs/loader:1065:27)
    at Module._compile (pkg/prelude/bootstrap.js:1758:32)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.runMain (pkg/prelude/bootstrap.js:1804:12)
    at node:internal/main/run_main_module:17:47

as expected.

And since ncc was brought up, using ncc on this minimal example and then using its output with pkg yields:

> [email protected]
> Targets not specified. Assuming:
  node16-linux-x64, node16-macos-x64, node16-win-x64
> Error! import.meta may appear only with 'sourceType: "module"' (5:95)
  /home/cleyfaye/t/dist/index.js

which I traced back to the babel config, and prevent the binary from being build. Quick-fixing this config issue brings us back to the issues described above without using ncc.

[jhmaster2000]

That was what we were doing until a recent update of ncc added compatibility with module-based source. It now produce files that pkg can't use; I could restore the build setup to get the actual error message if needed, but it was something along the line of not handling import statement that were indeed found in the output of ncc.

Would you happen to know what specific version of ncc made this change? I am also facing the issue addressed in this issue and am wondering if we could not just downgrade to an ncc version prior to that change and use that?

[CleyFaye]

The change was introduced in with [email protected]. Since we stopped using it I can't tell if something changed in later releases though.

[ForbiddenEra]

I haven't dug too deep.. but it looks like pkg wraps whatever program/package is compiled?

@ https://github.com/vercel/pkg/blob/main/prelude/bootstrap.js#L1845

  Module.runMain = function runMain() {
    Module._load(ENTRYPOINT, null, true);
    process._tickCallback();
  };

A minimal test using _load shows:

#~/test$ node testloader.js
node:internal/modules/cjs/loader:1146
      throw err;
      ^

Error [ERR_REQUIRE_ESM]: require() of ES Module ~/test/src/test.js not supported.
Instead change the require of test.js in null to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (~/test/testloader.js:2:8) {
  code: 'ERR_REQUIRE_ESM'
}

Node.js v17.2.0
#~/test$ cat testloader.js
const Module = require('module')
Module._load('./src/test.js',null,true)

So, would we not need to detect here if it's a "type":"module" in package.json or a *.mjs and then import it instead?

Is there a reason it's wrapped this way instead of execing node on the main script? Or is it actually even wrapped like that in the final package? Like I said, I haven't picked too deep on this issue yet but I'd like to help solve it if I can.

[robertsLando]

@ForbiddenEra if you check linked pr #1323 you will find the reason while esm are not supported yet

[ForbiddenEra]

@ForbiddenEra if you check linked pr #1323 you will find the reason while esm are not supported yet

I did read all of that, I guess I just (and am still not entirely) don't have full grasp on the process pkg is using. I do plan on possibly pulling the source and digging deeper though.

Now, even if the package was resolved correctly, would we not need a separate runMain for es modules..?

Or, is it the resolver actually generating said runMain function..? or..?

It would be nice if there was a list somewhere of the steps pkg takes exactly, ie:

1. parse files
2. lint/compile/byte code
3. compress

and with which libs/modules any step would involve. pkg seems to work quite differently than I might have guessed, ie, I would've thought that simply it created a self-extracting archive of a node setup and then simply run that node on the script, but there's obviously much more going on here.

[robertsLando]

I would've thought that simply it created a self-extracting archive of a node setup and then simply run that node on the script, but there's obviously much more going on here.

It's much more complicated then that, caxa does that (but doesn't provide source code protection). For more informations about how it work I have write a developer guide here: https://github.com/vercel/pkg/wiki/Developers

Based on what I have understand the only problem is we are using resolve package to resolve modules but it doesn't support es modules, we should use enhanced-resolve instead. Once that is done es modules should work

[ForbiddenEra]

I would've thought that simply it created a self-extracting archive of a node setup and then simply run that node on the script, but there's obviously much more going on here.

For more informations about how it work I have write a developer guide here: https://github.com/vercel/pkg/wiki/Developers

Awesome, I must've missed the link to that, I'll check it out.

[Nisthar]

guys does pkg works if you use dynamic imports?

[jesec]

This is not necessarily a bug.

However, this would be our highest priority feature request.

[lukaskellerstein]

Any progress here?

[robertsLando]

Follow updates on #1323. I know @jesec will try to implement it once he has some free time

[jesec]

[robertsLando]

Well done @jesec 🚀

[robertsLando]

@jesec Did you opened a PR with that?

[jesec]

No. Still long way to go at this point.

We might have to use vm.Module which is in experimental status as of Node 18.2. I am generally against relying on experimental API in this project. Additionally, bytecode generation and walking of async import is still unresolved at the moment.