Provide prebuilt binaries for ARM in GitHub Releases

[hronro]

Since M1 chip becomes more and more popular, it would be great if we can download prebuilt binaries for ARM in GitHub Releases.

[sampajano]

This is a very reasonable ask... We'll keep that in mind and see if we can do it soon (but I don't have a M1 chip laptop yet..) :)

[sampajano]

In the meantime, you could just use make plugin and make install-plugin on Mac from the project root (which in turn uses this Makefile) to build and install the plugin on MacOS.

(You might need to brew install protobuf first to install the protobuf lib. You might also want to remove protobuf's .dylib files to avoid them being dynamically linked in your binary, by: sudo rm /usr/local/lib/libproto*.dylib)

[tiero]

you could just use make client on Mac

In the Makefile you linked there is no client target AFAICT

[sampajano]

you could just use make client on Mac

In the Makefile you linked there is no client target AFAICT

Yes.. Thanks for catching.. i meant make plugin instead.. :)

make plugin from project root would first invoke the top-level Makefile, and will in turn use the Makefile i linked above.

Sorry for the confusion :)

[rafaelmagalhaes]

any updates on a PR for this fix?

[sampajano]

any updates on a PR for this fix?

This is not a code change so no PR is required. Our next step is to find a M1 chip MacBook to build the binary from :)

For now, please use the Makefile to build the plugin as mentioned above.

Thanks :)

[hronro]

@sampajano Cool. And I wonder if Linux ARM binaries and Windows ARM binaries are going to be supported as well?

[sampajano]

@sampajano Cool. And I wonder if Linux ARM binaries and Windows ARM binaries are going to be supported as well?

Not sure since i don't have configurations (or know they exists 😂) .. Might have to rely on the community to help test it out once we have a ARM binary :)

[tiero]

@sampajano I have M1 Apple Silicon, I can build for you and sign&release eventually.

I git cloned the repository, but looks like I need to load some google protobuf somewhere? Didn't find instructions

❯ make plugin          
cd "/Users/tiero/projects/tiero/grpc-web"/javascript/net/grpc/web/generator && make
c++ -std=c++11 -stdlib=libc++ -mmacosx-version-min=10.7  -I/usr/local/include -pthread  -c -o grpc_generator.o grpc_generator.cc
grpc_generator.cc:19:10: fatal error: 'google/protobuf/compiler/code_generator.h' file not found
#include <google/protobuf/compiler/code_generator.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make[1]: *** [grpc_generator.o] Error 1
make: *** [plugin] Error 2

[sampajano]

@sampajano I have M1 Apple Silicon, I can build for you and sign&release eventually.

Thanks a lot for offering to help! 😃 Unfortunately I can't really take and release your binaries (nothing against you personally but just in general).. for obvious reasons... :)

I git cloned the repository, but looks like I need to load some google protobuf somewhere? Didn't find instructions

Yes.. Protobuf lib is required to build. brew install protobuf will probably fix it for you and i've mentioned it above too 😃

I'll probably update our README to add instructions on building the plugin on your own soon.. :)

[tiero]

I can't really take and release your binaries

Makes sense :) Maybe Github Actions offers a way to build on cloud.

Protobuf lib is required to build

Yes I did that already, looks like still not enough (another Apple Silicon quirck?)

[C3sarRCDOU]

having the same problem i faced. same trouble is there any way i can compile the binaries? i already have protobc by running brew install protobuf

https://github.com/grpc/grpc-web/issues/1159#issuecomment-961723898

[sampajano]

I can't really take and release your binaries

Makes sense :) Maybe Github Actions offers a way to build on cloud.

Aha yes that's a great idea! We'll definitely explore that since that's a better way to release our binaries anyways!

Protobuf lib is required to build

Yes I did that already, looks like still not enough (another Apple Silicon quirck?)

Hmmm... sorry I have no idea then...

When i do brew uninstall protobuf i get the exact same error message as you, but then when i run brew install ptorobuf then make works for me with no issue..

I'm no expert on how c++ finds header files (and i doubt M1 changes that..), but here's where i can find the code_generator.h on my Mac (with brew ls protobuf):

Cellar/protobuf/3.17.3/include/google/protobuf/compiler/code_generator.h

Maybe someone more familiar with M1 and/or c++ could help here? 😅

[jdahlq]

I had the same issue running make on my M1 Macbook. Turns out Homebrew uses /opt/homebrew for ARM by default rather than /usr/local. (See https://docs.brew.sh/Installation.) So all you need to do is update the Makefile with the correct paths:

CPPFLAGS += -I/opt/homebrew/include -pthread
...
LDFLAGS += -L/opt/homebrew/lib -lprotoc -lprotobuf -lpthread -ldl

Once I did that, I was able to run make plugin from the project root directory. I verified that the newly compiled binary uses arm64 like so, and finally I copied it into /usr/local/bin:

file javascript/net/grpc/web/generator/protoc-gen-grpc-web
# Output should have "arm64" at the end (rather than "x86_64" or the like)

sudo cp javascript/net/grpc/web/generator/protoc-gen-grpc-web /usr/local/bin/
sudo chmod +x /usr/local/bin/protoc-gen-grpc-web

And FYI, if you don't know where your homebrew bin is, and the protobuf library has been installed, you can just do this:

type -a protoc
# Outputs the location of the protoc binary

[sampajano]

@jdahlq Thanks so much for sharing your solution!! 😃

---

And btw i'm migrating to Github Actions to build release binaries, but so far it seems that Apple M1 is not yet supported: https://github.com/actions/virtual-environments/issues/2187

Hopefully that issue will get resolved soon.. so we'll have M1 binary in future releases :)

[hronro]

After some experiments, now I'm able to cross-compile protoc-gen-grpc-web binaries for x86_64-darwin/aarch64-darwin/x86_64-linux/aarch64-linux/x86_64-windows/aarch64-windows, from a x86_64 Linux machine, by using zig's cross-compile capability.

This means even GitHub Actions does not support some targets (e.g. aarch64 macOS), we can still support all 6 platforms by just using the ubuntu-latest target.

Please let me know if the core team will accept a PR.

[sampajano]

@hronro Thanks so much for the suggestion! I'm not quite familiar with zig but i don't mind using it primarily for the ARM builds.

I wonder if you'd take a look at https://github.com/grpc/grpc-web/pull/1214 and see how your method compare? And potentially make some comments there too?

Thanks! :)

[hronro]

@sampajano

It's a completely different method compared to #1214, so unfortunately I don't have too much to say in that PR.

My method does not require docker, qemu, bazel, or anything else, the only thing required is the zig compiler.

BTW I didn't even notice there are some Dockerfiles. Are they for local testing purposes only? If I'm going to make a PR, do those Dockerfiles need to be changed (to use my method) as well?

[sampajano]

@sampajano

It's a completely different method compared to #1214, so unfortunately I don't have too much to say in that PR.

My method does not require docker, qemu, bazel, or anything else, the only thing required is the zig compiler.

Aha ok! Personally i don't know enough to decide the pros/cons of both approaches. Maybe you could explain how your approaches differs and what's the pros/cons?

CC @strophy for discussion too.

BTW I didn't even notice there are some Dockerfiles. Are they for local testing purposes only? If I'm going to make a PR, do those Dockerfiles need to be changed (to use my method) as well?

They're used for local testing as well as for CI. And yes.. all of them should be updated if we were to make some changes :)

[strophy]

I used a Docker-oriented approach for ARM64 builds (1) because it is what I am familiar with and (2) because it matched the existing build process. Zig looks cool but I'm not familiar with it, it looks like a compiler wrapper to make cross-compiling easier? The Docker approach to this would be to use more $TARGETARCH commands in the Dockerfiles as described here, or the even fancier xx project by lead buildx developer Tõnis Tiigi. I recently made a PR to the docker-protobuf repo to add ARM64 support and the maintainer there ended up going with xx to speed up build times. Since protobuf is a closely related project maybe you could figure out a common approach for consistency and to support each other?

[hronro]

@strophy Yes, zig uses clang internally while compiling C/C++ source code, and zig implements its own mach-o linker so it can target macOS when doing cross-compiling.

As far as I know, Docker is based on cgroup of the Linux kernel, so it's limited to using Linux as the operating system when using Docker, so it can only produce Linux binaries. I think one of the biggest reasons that people are asking for ARM binaries is the popularity of the M1 chips, so ARM binaries for Linux only don't really solve the problem.

As a comparison, zig is a compiler for compiling c/c++/zig source files, and made cross-compiling for c/c++/zig programs as easy as for go programs(with cgo disabled). Not only for Linux/macOS/Windows, if necessary, we can simply add more targets like FreeBSD, by just adding one line of the configuration file.

But like @strophy mentioned, using zig may not match the current build process. There are two approaches:

1. Keep the existing build process, we just use zig to build ARM binaries.
2. Use zig to build binaries for all platforms.

Personally, I would recommend approach 2, the current build process (in .github/workflows) just uses different ways to build for each platform (Docker for Linux, make for macOS, and bazel for Windows 😂), with zig we can manage them in a unified way, in only one GitHub workflow file.

They're used for local testing as well as for CI. And yes.. all of them should be updated if we were to make some changes :)

If they are only for local testing or CI, I don't think they need to be updated, since with zig we don't need to use Docker at all.

[strophy]

Interesting points, I am learning a lot here! It all depends on the target audience of the software I guess. My understanding of GRPC is that it is usually for inter-process communication in datacenters, where there are mainly Linux servers and fewer macOS devices in play. I explicitly need linux/arm64 support (Graviton CPUs on AWS infra and Raspberry Pi devices) and not macOS. However, since a lot of our developers work on macOS M1 devices, it also helps them because they can develop in linux/arm64 Docker containers with no performance penalty.

Not to say that building with Zig isn't the optimal solution here, just as long as one of the resulting target platforms is also linux/arm64 :smile:

[hronro]

My understanding of GRPC is that it is usually for inter-process communication in datacenters, where there are mainly Linux servers and fewer macOS devices in play.

Make sense for grpc, however, this is grpc-web instead of grpc, which is mainly used by front-end developers, and I believe most front-end developers won't use Docker.

BTW, I'm the maintainer of the protoc-gen-grpc-web NPM package. Currently, when you try to install this NPM package on an ARM machine (e.g. the M1 MacBooks), it simply throws an error and kills the install process, which is a really bad experience.

[hronro]

Hi @sampajano, do you have any comments on which approach we should use as I mentioned in https://github.com/grpc/grpc-web/issues/1159#issuecomment-1131358965

[sampajano]

@hronro Thanks for the ping!

Honestly i'm not really knowledgeable enough to make a decision right now whether zig should be adopted across the board.

I tend to lean towards the option of "Keep the existing build process, we just use zig to build ARM binaries.", and after we've seen what the code looks like for Mac/ARM (or maybe this will replace https://github.com/grpc/grpc-web/pull/1214 entirely for both Mac and Linux), then we can decide whether to adopt it across the board -- especially if it's a easy copy/paste for the rest :)

Either way, i'm definitely excited to provide a solution for M1/Mac! So the contribution here would be very much appreciated!

thanks again! :)

[sampajano]

Thanks so much to @hronro's contrib, we now have a workflow able to build the ARM binaries for all 3 platforms (Mac/Windows/Linux)! :)

I have tested building the ARM binaries workflow and the binaries can be downloaded from the link below (in the Artifacts section). https://github.com/grpc/grpc-web/actions/runs/2490449724

I cannot verify the binaries since i don't have ARM machines. If someone can help verify them on any of the 3 platforms, it would be helpful!

I'll make sure to include these binaries in the next release we cut 😃

Thanks again for the great contrib, @hronro!

[strophy]

Thanks @hronro for providing this effective build solution! @sampajano will the release also include a multi-arch docker image published to docker hub?

[sampajano]

Thanks @hronro for providing this effective build solution! @sampajano will the release also include a multi-arch docker image published to docker hub?

I'm actually not sure how to do multi-arch docker image (i supposed it's unrelated to the zig binaries?).. could look into that potentially if it's helpful to many :) (For now we don't really keep our docker hub images super up to date so i'd recommend just to build locally.. and it shouldn't take very long either.. :))

[strophy]

Yeah, it's unrelated to Zig binaries. I have a Dockerfile repo here that builds multi-arch grpc-web images, would you be interested in a PR to merge this Dockerfile and a GitHub Action to build a multiarch Docker image on a workflow_dispatch or release trigger? I'd really appreciate this, my goal in the previous PR was also to stop maintaining a separate repo just to build ARM, so I'd like to stop maintaining a separate docker image as well. :wink:

[sampajano]

Yeah, it's unrelated to Zig binaries. I have a Dockerfile repo here that builds multi-arch grpc-web images, would you be interested in a PR to merge this Dockerfile and a GitHub Action to build a multiarch Docker image on a workflow_dispatch or release trigger? I'd really appreciate this, my goal in the previous PR was also to stop maintaining a separate repo just to build ARM, so I'd like to stop maintaining a separate docker image as well. 😉

Thanks for the initiative! I think it'll be helpful to maintain the docker repo automatically on release!

Just to confirm, it'll be uploading to our dockerhub here right? https://hub.docker.com/r/grpcweb/

I'm not sure how the credentials stuff work but if you have a good solution i'd be happy to see it! :)

(Maybe send a simple POC PR first so we can iterate on it rather than you spending too much time on it and we ended up deciding against the general approach :))

---

Separately, curious why do you rely much on the docker images? AFAIU they're only used for people who want to test out our demo server/clients. And will not be used for any production, right? Also, it doesn't take that long to build locally too? 😃