Releasing v6.0.0

[aminya]

I'd like to explain the project's status and track the remaining issues for v6.0.0.

The latest beta builds without any issues, and prebuilt binaries for many platforms are provided.

The normal unit tests pass locally (pnpm run test.skip_gc_tests)

Some of the GC tests for some protocols fail (pnpm run test). However, I don't think these GC tests are crucial for functionality since the tests try to rely on GC's unpredictable behaviour.

If there is an issue that actually blocks us from releasing v6.0.0 please leave a comment.

[ni-mkrieg]

this is a blocker for us on MacOS: https://github.com/zeromq/zeromq.js/issues/530

[ni-mkrieg]

thanks, @aminya, will this be tagged as beta.11?

[aminya]

@ni-mkrieg This is fixed as of https://github.com/zeromq/zeromq.js/releases/tag/v6.0.0-beta.11

[ni-mkrieg]

thanks, yes would be great if you could prevent the whole rebuild, it is not running for a good while and then I get this error on x64 Mac.

ld: library not found for -lsodium
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Release/zeromq.node] Error 1
rm 7722eaf02fcc073df28ffd1fa831e71544edfd42.intermediate
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/Users/markus.krieg/.nvm/versions/node/v16.13.0/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (node:events:390:28)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:290:12)
gyp ERR! System Darwin 20.6.0
gyp ERR! command "/Users/markus.krieg/.nvm/versions/node/v16.13.0/bin/node" "/Users/markus.krieg/.nvm/versions/node/v16.13.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/markus.krieg/git/native-access/node_modules/zeromq

[ni-mkrieg]

on arm64 / M1 I get this error right away:

-- Configuring incomplete, errors occurred!
See also "/Users/markus.krieg/git/native-access/node_modules/zeromq/build/libzmq-staging/build/CMakeFiles/CMakeOutput.log".
cmake --build ./build --config Release --target install
make[1]: Makefile: No such file or directory
make[1]: *** No rule to make target `Makefile'.  Stop.
  TOUCH Release/obj.target/libzmq.stamp
  CXX(target) Release/obj.target/zeromq/src/context.o
In file included from ../src/context.cc:2:
In file included from ../src/context.h:4:
../src/prefix.h:7:10: fatal error: 'zmq.h' file not found
#include <zmq.h>
         ^~~~~~~
1 error generated.
make: *** [Release/obj.target/zeromq/src/context.o] Error 1
rm 7722eaf02fcc073df28ffd1fa831e71544edfd42.intermediate
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/opt/homebrew/Cellar/node@14/14.20.0/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (events.js:400:28)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:285:12)
gyp ERR! System Darwin 21.5.0
gyp ERR! command "/opt/homebrew/Cellar/node@14/14.20.0/bin/node" "/opt/homebrew/Cellar/node@14/14.20.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/markus.krieg/git/native-access/node_modules/zeromq

[aminya]

This issue can be fixed by installing sodium

brew install libsodium gnutls

Could you install zeromq with npm i --verbose. node-gyp-build doesn't give the reason why the prebuild is not picked up. https://github.com/prebuild/node-gyp-build/issues/54

[ni-mkrieg]

Good news, with sodium installed it did not rebuild on x64 at all any more, testing on arm64 now. I hope sodium is preinstalled on Github servers!

[aminya]

I am working on the node-gyp-build's preinstall logic: https://github.com/prebuild/node-gyp-build/pull/55

Hopefully, that is merged soon, and you will not need to do a rebuild!

[ni-mkrieg]

That would be very cool, even after installing sodium on M1 it is currently still rebuilding indeed. So sodium installed prevented rebuild on x64 but not arm64. Actually I am not 100% sure the brew install for libsodium worked on arm, still getting this line:

ld: library not found for -lsodium

The yarn install --verbose did not contain much useful information either.

verbose 53.652263084 Error: /Users/markus/git/native-access/node_modules/zeromq: Command failed.
Exit code: 1
Command: (shx test -f ./script/build.js || run-s build.js) && node-gyp-build
Arguments: 
Directory: /Users/markus/git/native-access/node_modules/zeromq
Output:
dyld[91022]: missing symbol called

Downloading libzmq source from https://github.com/zeromq/libzmq/releases/download/v4.3.4/zeromq-4.3.4.tar.gz
Building libzmq Release

[ni-mkrieg]

Okay so an x64 this is the reason why it rebuilds, I believe because it cannot find libsodium that is needed by your precompiled binary? That would also explain why it does not recompile after I install libsodium via homebrew. Is there a way for you to compile that lib into your prebuilt binary and not dynamically link to it?

Or could you replace libsodium all together with something that is already preinstalled during CI? Does not make sense for you either to spend all this time brew installing it during every CI build...

verbose 108.080591296 Error: /Users/markus/git/native-access/node_modules/zeromq: Command failed.
Exit code: 1
Command: (shx test -f ./script/build.js || run-s build.js) && node-gyp-build
Arguments: 
Directory: /Users/markus/git/native-access/node_modules/zeromq
Output:
node:internal/modules/cjs/loader:1183
  return process.dlopen(module, path.toNamespacedPath(filename));
                 ^

Error: dlopen(/Users/markus/git/native-access/node_modules/zeromq/prebuilds/darwin-x64/node.napi.glibc.node, 1): Library not loaded: /usr/local/opt/libsodium/lib/libsodium.23.dylib
  Referenced from: /Users/markus/git/native-access/node_modules/zeromq/prebuilds/darwin-x64/node.napi.glibc.node
  Reason: image not found
    at Object.Module._extensions..node (node:internal/modules/cjs/loader:1183:18)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at load (/Users/markus/git/native-access/node_modules/node-gyp-build/index.js:22:10)
    at Object.<anonymous> (/Users/markus/git/native-access/node_modules/node-gyp-build/build-test.js:19:19)
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32) {
  code: 'ERR_DLOPEN_FAILED'
}

[aminya]

libsodium is preinstalled on the GitHub runners, and it is very small. I am not sure where you are running this. GitHub doesn't offer an Arm runner. Please contact your CI engineer.

[ni-mkrieg]

all good if we do not have to rebuild, was more worried about your own CI.yml file having to do this for each build:

- name: Install Mac-OS Dependencies
        if: contains(matrix.os, 'macos')
        run: |
          brew install libsodium gnutls

were you able to avoid the rebuild on Mac x64 and arm64 - ideally without us having to install libsodium? it worked fine without libsodium installed in beta.6 for Mac x64...

[aminya]

That snippet is just to declare the dependencies to be more accurate. In practice it does nothing because it is already installed: https://github.com/zeromq/zeromq.js/actions/runs/3483046346/jobs/5826083947#step:7:11

You will need the libsodium dynamic library in either case during runtime. Without that library, the connections will not be secure.

Regarding the rebuild, I am still handling the external node-gyp-build issues. Please stay tuned.

[bfan1256]

I have done brew install and ran a verbose and can see that libsodium is found But it just happens at the final linking step that it just can't find -lsodium

I'm on M1 Mac ARM (osx64)

ld: library not found for -lsodium
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Release/zeromq.node] Error 1
rm 7722eaf02fcc073df28ffd1fa831e71544edfd42.intermediate
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (node:events:390:28)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:290:12)
gyp ERR! System Darwin 21.6.0
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/weigfan/code/shade-cinema/shade-plugin/node_modules/zeromq

[td-krzysiek]

Hello,

Exactly the same issue here on M1, this is after brew reinstall libsodium gnutls. Full log:

2022-11-22T11_35_47_985Z-debug-0.log

[td-krzysiek]

Hi,

I actually added one line to fix this

link_directories(${SODIUM_LIBRARY_DIRS})

below this line: https://github.com/zeromq/libzmq/blob/edfbb1ced652a93fd39c38338be9eb0f444f6e41/CMakeLists.txt#L269 node_modules/zeromq/build/libzmq/zeromq-4.3.2/CMakeLists.txt

and then to rebuild:

cd node_modules/zeromq
../.bin/node-gyp-build

[aminya]

@td-krzysiek could you make a pull request in that repository?

[td-krzysiek]

Done & looking forward for the V6 release 👍

[DonJayamanne]

Hi, Thank you for looking into releasing a new version. I'm one of the engineers working on the Jupyter extension for VS Code This package is used to communicate with Jupyter kernels (python kernels) over zmq

We found that the package fails in some cases which you may be aware of You have requested this new version to be tested here https://github.com/microsoft/vscode-jupyter/issues/10386#issuecomment-1315042452

I'll test this and get back to you with a confirmation. Again thanks for your contribution to this project

[aminya]

Done & looking forward for the V6 release 👍

The fix is merged! Now on master.

[bfan1256]

Awesome! We want to start testing this integration. Could you push up an NPM release?

[aminya]

v6.0.0-beta.12 is released with the latest changes! P.S: Resolved the fixed comments.

[bfan1256]

Looks like I'm still getting the same issue with the -LSodium

  TOUCH Release/obj.target/libzmq.stamp
  CXX(target) Release/obj.target/zeromq/src/context.o
  CXX(target) Release/obj.target/zeromq/src/incoming_msg.o
  CXX(target) Release/obj.target/zeromq/src/module.o
  CXX(target) Release/obj.target/zeromq/src/observer.o
  CXX(target) Release/obj.target/zeromq/src/outgoing_msg.o
  CXX(target) Release/obj.target/zeromq/src/proxy.o
  CXX(target) Release/obj.target/zeromq/src/socket.o
  SOLINK_MODULE(target) Release/zeromq.node
ld: library not found for -lsodium
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Release/zeromq.node] Error 1
rm 7722eaf02fcc073df28ffd1fa831e71544edfd42.intermediate
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (node:events:390:28)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:290:12)
gyp ERR! System Darwin 21.6.0
gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/weigfan/code/shade-cinema/shade-plugin/node_modules/zeromq

[td-krzysiek]

Yep, I confirm this, the problem is that zeromq.js is compiling node_modules/zeromq/build/libzmq/zeromq-4.3.2 which isn't master but exact 4.3.2 release and link_libraries line is missing there.

EDIT: actually please ignore above, I had a local copy of zeromq which was misleading. Indeed it is compiling latest version and I still get sodium linking error. I'll look into this later today.

[aminya]

Ok, try v6.0.0-beta.13. I forked node-gyp-build, and published my fixes. This should skip rebuilding, and if it does, it will give a clear message on the reason for rebuilding.

[bfan1256]

New Error.

Looks like it's related to something like this

verbose 6.1542075 Error: /Users/weigfan/code/shade-cinema/shade-plugin/node_modules/zeromq: Command failed.
Exit code: 127
Command: (shx test -f ./script/build.js || run-s build.js) && node-gyp-build
Arguments: 
Directory: /Users/weigfan/code/shade-cinema/shade-plugin/node_modules/zeromq
Output:
env: node\r: No such file or directory
    at ProcessTermError.ExtendableBuiltin (/Users/weigfan/.yarn/lib/cli.js:721:66)
    at ProcessTermError.MessageError (/Users/weigfan/.yarn/lib/cli.js:750:123)
    at new ProcessTermError (/Users/weigfan/.yarn/lib/cli.js:790:113)
    at ChildProcess.<anonymous> (/Users/weigfan/.yarn/lib/cli.js:25787:17)
    at ChildProcess.emit (node:events:390:28)
    at maybeClose (node:internal/child_process:1064:16)
    at Socket.<anonymous> (node:internal/child_process:450:11)
    at Socket.emit (node:events:390:28)
    at Pipe.<anonymous> (node:net:687:12)
error /Users/weigfan/code/shade-cinema/shade-plugin/node_modules/zeromq: Command failed.

[td-krzysiek]

https://github.com/zeromq/zeromq.js/blob/master/binding.gyp#L86

I replaced it with '<!@(pkg-config libsodium --libs)',, and it seems to work now. Tested with 6.0.0-beta.12 for now, haven't tried beta.13 yet.

EDIT: beta.13 gives the same issue with sodium ld: library not found for -lsodium, this fixes build for beta.13:

wget $(npm view zeromq dist.tarball)
tar -xzf zeromq-6.0.0-beta.13.tgz
cd package
vim binding.gyp /* modify as mentioned above */
npm i --verbose

[aminya]

Please test v6.0.0-beta.14. It should give more information on the reason for the rebuild.

[Y-qwq]

Please test v6.0.0-beta.14. It should give more information on the reason for the rebuild.

I have the following errors when installing beta.14 in MacOS(12.6 M1)

MacBook Air (M1, 2020)

install$ (shx test -f ./script/build.js || run-s build.js) && cross-env npm_config_build_from_source=true node-gyp-build
│ dyld[72929]: missing symbol called
└─ Failed in 233ms

[aminya]

Maybe we should include the libsodium.dyld in the package too so you don't have to install it manually.