Add support for node native addons / napi / FFI in `deno compile`

[marvinhagemeister]

Discussed in https://github.com/denoland/deno/discussions/23255

^{Originally posted by babakfp April 6, 2024} Hi 👋

https://github.com/parcel-bundler/lightningcss/discussions/712

Based on this comment https://github.com/parcel-bundler/lightningcss/discussions/712#discussioncomment-9029656 and this:

(I used this library https://deno.land/x/[email protected] and when I compiled, it started to download this ^ file).

It seems like Deno doesn't do the same for lightningcss.

Search keywords:

• TypeError: LoadLibraryExW failed

[bartlomieju]

I don't expect this to work. FFI/native Node extensions can't be included in the the compiled binary.

[AddictArts]

As a data point I ran into similar with electron-builder and now with deno. For elextron-builder I used asar.upacked which essentially put a node_modules and native within. Seems like deno should or maybe could do the same?

[Dmitri-Sintsov]

I don't expect this to work. FFI/native Node extensions can't be included in the the compiled binary.

Why not? These work with non-compiled source runtime. Anyway, in case these cannot be easily included, maybe the compiler could add these native modules as separate files to load them when running resulting binary? That would be much better than nothing.

[kt3k]

I wonder if we can support loading native addons using --include flag of compile command. Did anyone try that?

[funnylookinhat]

@kt3k I was wondering the same thing, but I think that would fail to properly support the --target parameter. I am pretty sure your local node_modules will be built only for your active architecture. As an example, if you were running on an M1 Mac and compiling for x86_64-unknown-linux-gnu, the binaries you include would not run properly in the resulting binary.

[sigmaSd]

@kt3k it doesn't work it can't dlopen embedded files, the workaround is to copy the file to the host system before dl opening it

[sigmaSd]

Bun for example does handle this pretty well, even complex napi projects like slint just works, bun compile produce a single executable that have the library bundled and it does dlopen it at runtime correctly

[tarasglek]

Would be nice to see a suggested workaround for something like #28108

[sigmaSd]

The workaround is to copy the native library to the host system and open it from there

If the library that you're using allow you to set the native library path you can use that or you will have to vendor the library and make the necessary changes

For example I have a couple of apps that use napi.ra so I added this workaround there https://github.com/napi-rs/napi-rs/pull/2471/files

[tarasglek]

I have a hack for same duckdb problem as #28108 :

COMPILED_NPM_DIR=/tmp/deno-compile-app/.deno_compile_node_modules/localhost/
NPM_DIR=.deno/npm/registry.npmjs.org/

mkdir -p $COMPILED_NPM_DIR
ln -svf $(realpath $NPM_DIR)/@duckdb $COMPILED_NPM_DIR
./app

I put above into app.sh, app is a deno compile-d binary. It's horrible, would be nice to override /tmp/deno-compile-app/.deno_compile_node_modules/localhost/ with an env var.

[mommysgoodpuppy]

theres also deno compile vendor, I originally assumed that would enable some workaround like this

--vendor # Toggles local vendor folder usage for remote modules and a node_modules folder for npm packages.

[echoi92]

I have a hack for same duckdb problem as #28108 :

COMPILED_NPM_DIR=/tmp/deno-compile-app/.deno_compile_node_modules/localhost/
NPM_DIR=.deno/npm/registry.npmjs.org/

mkdir -p $COMPILED_NPM_DIR
ln -svf $(realpath $NPM_DIR)/@duckdb $COMPILED_NPM_DIR
./app

I put above into app.sh, app is a deno compile-d binary. It's horrible, would be nice to override /tmp/deno-compile-app/.deno_compile_node_modules/localhost/ with an env var.

I don't quite understand how you got the COMPILED_NPM_DIR and NPM_DIR

this is the error message I get trying to compile with libsql client

error: Uncaught (in promise) TypeError: dlopen(/var/folders/vq/774c9ls12y7f0p01b9lkq9hm0000gn/T/deno-compile-my-app/node_modules/.deno/@[email protected]/node_modules/@libsql/darwin-x64/index.node, 0x0001): tried: '/var/folders/vq/774c9ls12y7f0p01b9lkq9hm0000gn/T/deno-compile-my-app/node_modules/.deno/@[email protected]/node_modules/@libsql/darwin-x64/index.node'

Looks to me like my compiled directory is /var/folders/vq/774c9ls12y7f0p01b9lkq9hm0000gn/T/

[realChakrawarti]

Hi @dsherret, as #28908 has been closed citing duplicate of this issue, could please add a comment about the current status/progress of this. Thank you.

[tarasglek]

I don't quite understand how you got the COMPILED_NPM_DIR and NPM_DIR

I have a hack for same duckdb problem as #28108 :

COMPILED_NPM_DIR=/tmp/deno-compile-app/.deno_compile_node_modules/localhost/

used strace (I'm on linux) to find this :)

NPM_DIR=.deno/npm/registry.npmjs.org/

that's where deno stores npm libs during normal deno run usage

[dsherret]

Hi @dsherret, as https://github.com/denoland/deno/issues/28908 has been closed citing duplicate of this issue, could please add a comment about the current status/progress of this. Thank you.

PR here: https://github.com/denoland/deno/pull/28934 -- should be in Deno 2.3

[johnspurlock]

Was really excited to see this - retested duckdb in 2.3.1 and it still fails at runtime, albeit with a different error: https://github.com/denoland/deno/issues/29203