grpc-node
grpc-node copied to clipboard
grpc
missing pre-compiled arm binary
Problem description
Can't install grpc-tools on arm architecture (raspberry pi) because of missing precompiled binaries.
node-pre-gyp ERR! stack Error: 404 status code downloading tarball https://node-precompiled-binaries.grpc.io/grpc-tools/v1.8.1/linux-arm64.tar.gz
Is it possible to build these at all?
Reproduction steps
npm init npm install grpc-tools
Environment
- Ubuntu 20.04 arm64
- 10-12-14
- n
- grpc-tools, possibly more grpc packages - haven't checked yet
Additional context
error /home/repo/grpc/node_modules/grpc-tools: Command failed.
Exit code: 1
Command: node-pre-gyp install
Arguments:
Directory: /home/repo/grpc/node_modules/grpc-tools
Output:
node-pre-gyp info it worked if it ends with ok
node-pre-gyp info using [email protected]
node-pre-gyp info using [email protected] | linux | arm64
node-pre-gyp WARN Using request for node-pre-gyp https download
node-pre-gyp info check checked for "/home/repo/grpc/node_modules/grpc-tools/bin/grpc_tools.node" (not found)
node-pre-gyp http GET https://node-precompiled-binaries.grpc.io/grpc-tools/v1.8.1/linux-arm64.tar.gz
node-pre-gyp http 404 https://node-precompiled-binaries.grpc.io/grpc-tools/v1.8.1/linux-arm64.tar.gz
node-pre-gyp ERR! install error
node-pre-gyp ERR! stack Error: 404 status code downloading tarball https://node-precompiled-binaries.grpc.io/grpc-tools/v1.8.1/linux-arm64.tar.gz
node-pre-gyp ERR! stack at Request.<anonymous> (/home/repo/grpc/node_modules/node-pre-gyp/lib/install.js:149:27)
node-pre-gyp ERR! stack at Request.emit (events.js:203:15)
node-pre-gyp ERR! stack at Request.onRequestResponse (/home/repo/grpc/node_modules/request/request.js:1059:10)
node-pre-gyp ERR! stack at ClientRequest.emit (events.js:198:13)
node-pre-gyp ERR! stack at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:565:21)
node-pre-gyp ERR! stack at HTTPParser.parserOnHeadersComplete (_http_common.js:111:17)
node-pre-gyp ERR! stack at TLSSocket.socketOnData (_http_client.js:451:20)
node-pre-gyp ERR! stack at TLSSocket.emit (events.js:198:13)
node-pre-gyp ERR! stack at addChunk (_stream_readable.js:288:12)
node-pre-gyp ERR! stack at readableAddChunk (_stream_readable.js:269:11)
node-pre-gyp ERR! System Linux 5.4.0-1008-raspi
node-pre-gyp ERR! command "/usr/local/bin/node" "/home/repo/grpc/node_modules/grpc-tools/node_modules/.bin/node-pre-gyp" "install"
node-pre-gyp ERR! cwd /home/repo/grpc/node_modules/grpc-tools
node-pre-gyp ERR! node -v v10.20.1
node-pre-gyp ERR! node-pre-gyp -v v0.12.0
node-pre-gyp ERR! not ok
This is also an issue when running inside Docker on the new MacBook with the M1 Chip.
@Nilos Same issue brought me here 👋
Update: I managed to build and run the container by adding --platform linux/amd64 option to both build and run docker commands. You could alternatively specify platform: linux/amd64 for each docker-compose service.
Given the introduction of Apple Silicon, and the fact that Docker for Mac on these machines use arm64, does it make sense to re-open this case? Building JS apps that depend on grpc (directly or indirectly) is no longer possible in Docker on Mac (or at least not easy; e.g I can't make the platform trick work with docker-compose, and there are many qemu crashes when running in emulated mode on an M1).
@remko is right, I'd reopen this one. Some issues I came across with just can't be sorted out that easily.
Our build process runs on GitHub Actions, so adding this is blocked on actions/virtual-environments#2187.
Can anyone share the instructions on how to build the binaries from the source code?
Can anyone share the instructions on how to build the binaries from the source code?
Basically https://github.com/grpc/grpc-node/blob/master/packages/grpc-tools/build_binaries.sh
This script probably requires changes, but we have no way of testing on any M1 hardware, so you'd be on your own to change it.
@nicolasnoble @murgatroid99 I am actually using trying to install the grpc-tools on AWS Graviton 2 instances. Are there any instructions on how to run the test cases? I have access to both M1 hardware and Graviton instances. I can run the test cases and share my findings/results.
For any on M1 hardware running into this, I was able to workaround it by just forcing the x64 binary and letting Rosetta deal with it:
yarn add grpc-tools --ignore-scripts
pushd node_modules/grpc-tools
node_modules/.bin/node-pre-gyp install --target_arch=x64
popd
Is there any way to include this during normal install?
I think you can pass the --target_arch=x64 argument directly to npm install. I assume it works the same way with yarn, but I don't use yarn.
@shnhrrsn Also, if you use nvm, you can install x86 node versions following the steps under the "Macs with M1 chip" section of https://github.com/nvm-sh/nvm#macos-troubleshooting. This one solved all my M1 related issues.
ARM binaries should be distributed as part of releases. Especially in 2021. Many cloud providers and personal computer manufacturers are beginning to implement ARM as larger offerings. This is also troublesome for raspberry pi users who would like to run node apps. There isn't a good reason not to pre-compile these binaries, otherwise node marketshare will lessen. I'm already pretty frustrated with it and will be doing less node development as a result.
I said that making this change is blocked in https://github.com/grpc/grpc-node/issues/1405#issuecomment-784609436. Nothing in the linked issue appears to have changed, so it's still blocked.
Our build process runs on GitHub Actions, so adding this is blocked on actions/virtual-environments#2187.
lol that issue is blocking SO many people
We do not intend to distribute ARM binaries for
grpc-tools. I suggest generating your code on a different system and then transferring your code to your ARM system instead.
Apple Silicon is here to stay and so is grpc-node. Please reconsider :)
That comment predates the announcement of Apple Silicon. I wasn't talking about Apple Silicon. As I said in my more recent comments https://github.com/grpc/grpc-node/issues/1405#issuecomment-784609436 and https://github.com/grpc/grpc-node/issues/1405#issuecomment-891187791
Our build process runs on GitHub Actions, so adding this is blocked on actions/virtual-environments#2187.
I'm curious why there's not a fallback setting to just compile the binaries if no prebuilt exists. Other packages do something similar.
Edit: it appears node-pre-gyp is being abused to just download files. because we're not actually using a binding to node for those tools?
I disagree with the term "abused". We are using node-pre-gyp to do what it is supposed to do: download binary files for the system the code is running on. It doesn't have a fallback to building locally because it is building protoc, which has an existing build process that uses cmake instead of node-gyp.
Abused might be a strong word, but it's certainly a tad misused. Their docs say it should "stand between" npm and node-gyp. If we can't fallback using the node-gyp to build, i wonder if there's a good way to wrap node-pre-gyp to just auto run ./build_binaries.sh if that fails?
Here's how i got around this without using Rosetta:
- clone this repo
- install your grpc-tools with
yarn install --ignore-scripts - run the script ./build_binaries.sh in the grpc-tools package
- copy your files over, something like this:
#!/bin/bash
cp binaries/arm64/protoc node_modules/grpc-tools/bin/protoc
cp binaries/arm64/grpc_node_plugin node_modules/grpc-tools/bin/grpc_node_plugin
Think of it more as a stop gap, really. The reality being that building protoc from pure source is impossible due to its bootstrapping nature, and so we can't write a node-gyp installer for it, which requires a very flat list of files to compile, with no intermediate compilation process.
Ideally, node-gyp should support cmake, but that's also not happening: https://github.com/nodejs/node-gyp/issues/1590
Additionally, Apple has made it very difficult to build software on their new M1 hardware when using things like github actions, rendering us basically powerless to create working M1 binaries. So we're stuck between a rock and a hard place here, and there's no good solution unfortunately.
This is also an issue when running inside Docker on the new MacBook with the M1 Chip.
Did anyone find a fix when running in Docker?
We have a team that develops in VSCode Remote Containers (ie. docker containers) - some have M1 chips others intel. Any way to support both?
@shnhrrsn Also, if you use nvm, you can install x86 node versions following the steps under the "Macs with M1 chip" section of https://github.com/nvm-sh/nvm#macos-troubleshooting. This one solved all my M1 related issues.
@antonimmo That doesn't work when inside a debian-bullseye docker container (on a Mac M1 host) since arch -x86_64 zsh causes an error.
Looks like we must build grpc-tools ourselves for Apple Silicon? I'm a beginner here. Could anybody kindly share a pointer to how to build it locally?
Looks like we must build grpc-tools ourselves for Apple Silicon? I'm a beginner here. Could anybody kindly share a pointer to how to build it locally?
Clone this repo, then cd grpc-node/packages/grpc-tools; git submodule update --init --recursive; ./build_binaries.sh should build the binaries. Make sure you have XCode/Command Line Tools installed.
@ldx Thanks a lot! I also found that I could resort to the emulated version with a Rosseta-enabled Terminal copy and install the Intel-edition Homebrew.
Kind of off-topic question for the workaround to work.
Is it possible to list the built binary in package.json as fallback when a binary is not available? My issue is that grpc-tools are listed as a dependency which works for all developers except those with M1.
@ldx Thanks a lot! I also found that I could resort to the emulated version with a Rosseta-enabled Terminal copy and install the Intel-edition Homebrew.
could you please give more details on how you did all of that?
Kind of off-topic question for the workaround to work.
Is it possible to list the built binary in package.json as fallback when a binary is not available? My issue is that
grpc-toolsare listed as a dependency which works for all developers except those with M1. For M1 I want to use the compileddarwin-x64.tar.gzinstead.
It is possible and would be nice to be able to build it if a precompiled binary is not available.
I am on an apple m1 chip.
My workaround.....
arch -x86_64 bash
touch ~/.bashrc
curl -sL https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.0/install.sh -o install_nvm.sh
# I wanted to use node 14, so I had to uninstall and reinstall with new arch
nvm uninstall 14
nvm install 14
# navigate to your project dir in the same terminal window running diff arch
cd ~/<my-project-dir>
rm -rf node_modules && rm package-lock.json
npm i
# run start command
npm start
