Ruby google-protobuf-4.26.1-aarch64-linux precompiled gem is not compatible with linux-musl

[ntkme]

What version of protobuf and what language are you using? Version: 4.26.1 Language: Ruby

What operating system (Linux, Windows, ...) and version? Alpine Linux 3.19.1

What runtime / compiler are you using (e.g., python version or gcc version) ruby 3.3.1 (2024-04-23 revision c56cd86388) [aarch64-linux-musl]

What did you do? Steps to reproduce the behavior:

/ # gem install google-protobuf
Successfully installed google-protobuf-4.26.1-aarch64-linux
1 gem installed
/ # ruby -e "require 'google/protobuf'"
<internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require': cannot load such file -- google/protobuf_c (LoadError)
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf_native.rb:15:in `rescue in <top (required)>'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf_native.rb:12:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:57:in `<module:Protobuf>'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:15:in `<module:Google>'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:14:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `rescue in require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:135:in `require'
	from -e:1:in `<main>'
<internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require': Error loading shared library ld-linux-aarch64.so.1: No such file or directory (needed by /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/3.3/protobuf_c.so) - /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/3.3/protobuf_c.so (LoadError)
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf_native.rb:13:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:57:in `<module:Protobuf>'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:15:in `<module:Google>'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:14:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `rescue in require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:135:in `require'
	from -e:1:in `<main>'
<internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require': cannot load such file -- google/protobuf (LoadError)
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from -e:1:in `<main>'

What did you expect to see google-protobuf should work on aarch64-linux-musl just like how it works on x86_64-linux-musl

What did you see instead? google-protobuf prebuilt library is strictly linked to glibc and it's not compatible with musl-libc

Anything else we should know about your project / environment Please see rake-compiler's documentation regarding recent changes in rubygems in regarding to -linux, -linux-gnu, and -linux-musl platforms: https://github.com/rake-compiler/rake-compiler-dock?tab=readme-ov-file#linux-gnu-and-musl-important-details

[ntkme]

As a workaround, uninstalling the prebuilt gem and reinstall platform ruby to build from source fixes the issue:

/ # gem uninstall google-protobuf
Successfully uninstalled google-protobuf-4.26.1-aarch64-linux
/ # apk add alpine-sdk
...
/ # gem install --platform ruby google-protobuf
Fetching google-protobuf-4.26.1.gem
Building native extensions. This could take a while...
Successfully installed google-protobuf-4.26.1
1 gem installed
/ # ruby -e "require 'google/protobuf'"

For bundler, the following in Gemfile would work:

gem 'google-protobuf', force_ruby_platform: true if RUBY_PLATFORM.include?('linux-musl')

[ntkme]

There are a few options here:

1. Do not ship aarch64-linux gem, and let both aarch64-linux-gnu and aarch64-linux-musl platforms fallback to platform ruby and compile from source.
2. Ship the current flavor, but label it as aarch64-linux-gnu, so that it does not get installed incorrectly on aarch64-linux-musl.
3. Upgrade rake-compiler, and ship separate aarch64-linux-gnu and aarch64-linux-musl gems.

Again, please refer to rake-compiler's documentation on details: https://github.com/rake-compiler/rake-compiler-dock?tab=readme-ov-file#linux-gnu-and-musl-important-details

[JasonLunn]

@dazuma - what's the expected platform support for linux-musl?

[ntkme]

As said early, even if we don't "officially" support linux-musl, we can avoid it from breaking out of box. - Just ship the gem with label -linux-gnu instead of -linux using latest rake-compiler.

[henrahmagix]

I'd really appreciate if @ntkme's comment could be actioned pls ☺️

ship the gem with label -linux-gnu instead of -linux using latest rake-compiler

[ntkme]

This issue used to be aarch64-linux-musl only.

However, with 4.29.3 release, that x86_64-linux prebuilt now triggers Segmentation fault on x86_64-linux-musl:

• google-protobuf 4.29.2 (x86_64-linux) worked on musl: https://github.com/sass-contrib/sass-embedded-host-ruby/actions/runs/12606460485/job/35136617853
• google-protobuf 4.29.3 (x86_64-linux) crashed on musl with Segmentation fault: https://github.com/sass-contrib/sass-embedded-host-ruby/actions/runs/12712732880/job/35438995154

This issue can be worked around by force compiling from source by adding the following in Gemfile:

gem 'google-protobuf', force_ruby_platform: true if RUBY_PLATFORM.include?('linux-musl')

@JasonLunn I highly suggest that we take action now to update rake-compiler and split linux-gnu and linux-musl gems as soon as possible.

[ntkme]

The statistics from my own gem that has 10M+ total download shows that roughly 10% of x86_64-linux are from linux-musl, which is definitely not a small number. E.g. As of writing, the download counts for the latest version show:

• https://rubygems.org/gems/sass-embedded/versions/1.83.1-x86_64-linux-gnu - 24,734
• https://rubygems.org/gems/sass-embedded/versions/1.83.1-x86_64-linux-musl - 2,983

The statistics for google-protobuf might be different, but the total download for google-protobuf is way higher that even only 5% would affect tons of users. I know lots of users in ruby community uses docker.io/library/ruby:alpine image, some even use alpine for production, thus it's critical to not break linux-musl out of box even if we decide to not ship linux-musl prebuilt - an error from bundler asking user to install a compiler is way better than seeing Segmentation fault.

[ntkme]

I was probably the earliest adaptor of splitting linux-gnu and linux-musl prebuilt, as I introduced it way before rake-compiler added support, using my own custom cross-platform build scripts. I have been through lots of trouble with rubygems/bundler's bug around this area in terms of platform detection and even helped fixing a few bugs in rubygems/bundler.

From my own experience with it, I can say that as of now it's reliable with ruby >=3.1 and the default rubygems/bundler or higher version from them. In fact, nokogiri, one of the most downloaded gem of all time, introduced split -linux-gnu and -linux-musl with 1.18.0 release this month: https://rubygems.org/gems/nokogiri/versions https://github.com/sparklemotion/nokogiri/commit/4c15c5c590eeca4a91eb4cf7fe13f117fedcfe37

Now it's time to adopt -linux-gnu and -linux-musl gems.

[dazuma]

I share the desire to split the builds into -linux-gnu and -linux-musl as soon as possible. However, we cannot until April 2025 because our support policy doesn't drop Ruby 3.0 until then. Once we're able to drop support for Ruby 3.0, we'll definitely split the builds with the next protobuf release.

[ntkme]

@dazuma In fact shipping separate -gnu and -musl does not require dropping Ruby 3.0.

The way it works is that you can set different ruby version and rubygems version requirements on the platform ruby gem and native prebuilt gems.

For example:

• https://rubygems.org/gems/sass-embedded/versions/1.83.4 - platform ruby requires ruby 3.1

• https://rubygems.org/gems/sass-embedded/versions/1.83.4-aarch64-linux-gnu - native gem requires ruby 3.2

• https://rubygems.org/gems/nokogiri/versions/1.18.1 -platform ruby gem requires ruby 3.1

• https://rubygems.org/gems/nokogiri/versions/1.18.1-x86_64-linux-gnu - native gem requires ruby 3.1 AND rubygems 3.3.22

This allow older version of ruby and rubygems/bundler to install the gem by compiling from source. For google-protobuf, we can require 3.0 on the platform ruby gem, and require ruby 3.0 AND rubygems 3.3.22.

You can learn more here about in this issue: https://github.com/rake-compiler/rake-compiler-dock/issues/117 and why ruby 3.2 instead of rubygems 3.3.22 on native gem might be a better idea in this comment https://github.com/rake-compiler/rake-compiler-dock/issues/117#issuecomment-2101338327

[dazuma]

@ntkme It sounds like the above would force ALL users of Ruby 3.0 on Linux to install the gem from source. That's a regression we're trying to avoid.

(The news about Debian-sid and Ruby 3.1 is also disappointing. But I don't think we can wait another year for this.)

[ntkme]

Yes, I agree not able to use prebuilt is a “regression”. However, I’d argue it’s way better than segmentation fault on musl-libc, especially that 4.29.3 literally just regressed with segmentation fault on x86_64-linux-musl. It used to have segmentation fault only on aarch64-linux-musl, which has a very tiny user base, but now impacting many more users (about 10x more based on download stats on sass-embedded and nokogiri).

The 4.29.3 segmentation fault regression on all ruby versions on x86_64-linux-musl is much worse than asking ruby 3.0 on any linux to compile from source. I think it’s worth trading the compiling from source “regression” to fix the segmentation fault, especially given that the compiling from source “regression” will likely break much smaller number of installation on ruby 3.0 on Linux without a working compiler. In addition, the installation time error will be more trivial for users to deal with than a runtime segmentation fault.

After all, I don’t have statistics to proof which one has a border impact on the user base, but here are the two things we are trading:

• ruby 3.0-3.4 *-linux-musl crashing with segmentation fault
• ruby 3.0 *-linux-* without a working compiler failing to install, or with a working compiler to have slightly slower installation

I would recommend that we don’t wait.

[jeffque]

The current suggested workaround had some problems to me, as in my CI the Gemfile.lock was frozen.

For me, this was the workaround:

gem 'google-protobuf', force_ruby_platform: RUBY_PLATFORM.include?('linux-musl')

Pretty similar to the one suggested by @ntkme , but the gem is added regardless of platform. In my case, the gem was added as a transitive dependency from sass-embedded gem.

Running with my workaround: https://gitlab.com/computaria/blog/-/jobs/9012424269

Running with the workaround from @ntkme: https://gitlab.com/computaria/blog/-/jobs/9012354715

Notice the error:

Installing bundler 2.5.11
Bundler is unlocking ruby, but the lockfile can't be updated because frozen mode
is set
You have added to the Gemfile:
* google-protobuf
Run `bundle install` elsewhere and add the updated Gemfile to version control.

[ntkme]

@dazuma April is coming soon. Can we formally put this on the roadmap of next v31 release?

[JasonLunn]

Work to drop Ruby 3.0 and update to a newer bundler version is in progress, stay tuned for further updates.

[ntkme]

I see that we merged the commit to drop Ruby 3.0. Next step would be update rake-compiler / rake-compiler-dock, so that we can relabel the -linux gem to -linux-gnu, and optionally publish a separate -linux-musl gem.

[ball-hayden]

Could I please add a request that -linux-musl be published. Using Alpine as a base image is reasonably common (Alpine using MUSL rather than GNU).

[JasonLunn]

Please try out the latest release, 4.31.0-rc1, and confirm that it has resolved this issue for you, @ntkme. The release includes binary packages for -gnu and -musl.

[ntkme]

@JasonLunn Worked as expected!

[henrahmagix]

@JasonLunn could we kindly get a v3.x.x release of the separate -musl binary if possible please, when it's out of release-candidate? Thanks in advance! 🤞

[ntkme]

@henrahmagix Unfortunately, based on https://github.com/protocolbuffers/protobuf/issues/16853#issuecomment-2591772277 back-porting is probably not going to happen, because they cannot drop support of ruby <= 3.0 in a patch release.