ruby: wrap_memcpy causes error with LTO

[sorah]

What version of protobuf and what language are you using?

Version: d447b2924f695ffa55f6bdfc3537b97bc12d9504 and 3.21.12 Language: Ruby

What operating system (Linux, Windows, ...) and version?

Linux, Fedora 36

What runtime / compiler are you using (e.g., python version or gcc version)

• Fedora 36, GCC 12.2
• Fedora 38, GCC 13.0

What did you do?

The following script under docker run -ti fedora:36 bash (and fedora:38) fails:

dnf install ruby-devel
dnf group install "C Development Tools and Libraries" "Development Tools"
gem install google-protobuf --platform ruby

What did you expect to see

• installation succeeds

What did you see instead?

Fedora 38 (GCC 13.0): Installation failure:

gcc -shared -o protobuf_c.so protobuf.o convert.o defs.o message.o repeated_field.o map.o ruby-upb.o wrap_memcpy.o naive.o range2-neon.o range2-sse.o -L. -L/usr/lib64 -L. -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -fstack-protector-strong -rdynamic -Wl,-export-dynamic -Wl,--no-as-needed -Wl,-wrap,memcpy -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1   -m64  -lruby  -lm -lpthread  -lc
/usr/bin/ld: /tmp/ccU44S3j.ltrans1.ltrans.o: in function `_upb_EnumReservedNames_New':
/usr/include/bits/string_fortified.h:29: undefined reference to `__wrap_memcpy'
/usr/bin/ld: /tmp/ccU44S3j.ltrans1.ltrans.o: in function `jsondec_string':
/usr/include/bits/string_fortified.h:29: undefined reference to `__wrap_memcpy'
/usr/bin/ld: /usr/include/bits/string_fortified.h:29: undefined reference to `__wrap_memcpy'
/usr/bin/ld: /tmp/ccU44S3j.ltrans1.ltrans.o: in function `jsondec_wellknown':
/usr/include/bits/string_fortified.h:29: undefined reference to `__wrap_memcpy'
/usr/bin/ld: /tmp/ccU44S3j.ltrans1.ltrans.o: in function `jsonenc_putbytes.lto_priv.0':
/usr/include/bits/string_fortified.h:29: undefined reference to `__wrap_memcpy'
/usr/bin/ld: /tmp/ccU44S3j.ltrans1.ltrans.o:/usr/include/bits/string_fortified.h:29: more undefined references to `__wrap_memcpy' follow
collect2: error: ld returned 1 exit status
make: *** [Makefile:266: protobuf_c.so] Error 1

Fedora has enabled LTO when building Ruby, so its RbConfig::CONFIG["CFLAGS"] includes -flto as well.

Removing -Wl,-wrap,memcpy resolves this compilation error.

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88643 mentions use of -wrap is not supported when LTO enabled, so it might be worth to do either of:

1. as the wrap_memcpy exists for binary distrribution, avoid using this trick when building not for binary distribution
2. always add -fno-lto at extconf.rb

Anything else we should know about your project / environment

[sorah]

I also personally maintains the ruby binary packages for Ubuntu and Debian https://github.com/sorah-rbpkg

Under Ubuntu 22.04 and Ruby 3.2.1 from my archive, installation succeeds but hangs when loaded:

(docker run --rm -ti public.ecr.aws/sorah/ruby:3.2-dev-jammy)

gem i google-protobuf --platform ruby
ruby -rgoogle/protobuf -ve0

I suspect this is also caused by the same issue explained above, while not sure why compilation succeeds in this case.

[sorah]

Submitted https://github.com/protocolbuffers/protobuf/pull/11938 for option (2).

[Transfusion]

I am encountering this problem too with truffleruby 22.3.1 on aarch64. Version: https://github.com/sorah/protobuf/commit/d447b2924f695ffa55f6bdfc3537b97bc12d9504 and 3.21.12 from RubyGems.

What operating system (Linux, Windows, ...) and version? Oracle Linux 9 aarch64 as distributed by https://github.com/graalvm/container/pkgs/container/graalvm-ce/65493297?tag=22.3.1

GraalVM ships its own LLVM runtime. https://www.graalvm.org/latest/reference-manual/llvm/

What did you do?

docker run --rm -it ghcr.io/graalvm/graalvm-ce:ol9-java11-22.3.1 /bin/bash
# install the truffleruby runtime
gu install ruby && /opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/truffle/post_install_hook.sh
ruby -v
# truffleruby 22.3.1, like ruby 3.0.3, GraalVM CE Native [aarch64-linux]
gem install google-protobuf

What did you see instead?

...
13 warnings generated.
compiling wrap_memcpy.c
compiling ./third_party/utf8_range/naive.c
compiling ./third_party/utf8_range/range2-neon.c
compiling ./third_party/utf8_range/range2-sse.c
linking shared-object google/protobuf_c.so
ld.lld: error: undefined hidden symbol: __wrap_memcpy
>>> referenced by convert.c:49 (/opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/gems/gems/google-protobuf-3.21.12/ext/google/protobuf_c/convert.c:49)
>>>               lto.tmp:(Convert_StringData)
>>> referenced by convert.c:301 (/opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/gems/gems/google-protobuf-3.21.12/ext/google/protobuf_c/convert.c:301)
>>>               lto.tmp:(Msgval_DeepCopy)
>>> referenced by message.c:687 (/opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/gems/gems/google-protobuf-3.21.12/ext/google/protobuf_c/message.c:687)
>>>               lto.tmp:(Message_dup)
>>> referenced 91 more times
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Makefile:752: protobuf_c.so] Error 1

make failed, exit code 2

I then tried building https://github.com/protocolbuffers/protobuf/pull/11938 from source with rake but unfortunately the error was still the same, even with $LDFLAGS += " -fno-lto -Wl,-wrap,memcpy".

Next, I tried gem install google-protobuf -- --with-cflags="-fno-lto". This time, the installation succeeded, however I ran into runtime errors.

# ruby -rgoogle/protobuf -ve0
truffleruby 22.3.1, like ruby 3.0.3, GraalVM CE Native [aarch64-linux]
[To redirect Truffle log output to a file use one of the following options:
* '--log.file=<path>' if the option is passed using a guest language launcher.
* '-Dpolyglot.log.file=<path>' if the option is passed using the host Java launcher.
* Configure logging using the polyglot embedding API.]
[ruby] WARNING: Encoding.find('locale') is US-ASCII, this often indicates that the system locale is not set properly. Set LANG=en_US.UTF-8 and see https://www.graalvm.org/dev/reference-manual/ruby/UTF8Locale/ for details.
<internal:core> core/kernel.rb:234:in `gem_original_require': libgraalvm-llvm.so.1: cannot open shared object file: No such file or directory (RuntimeError)
        from <internal:/opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/mri/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/gems/gems/google-protobuf-3.21.12/lib/google/protobuf.rb:51:in `<top (required)>'
        from <internal:core> core/kernel.rb:234:in `gem_original_require'
        from <internal:/opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/mri/rubygems/core_ext/kernel_require.rb>:160:in `require'
        from <internal:core> core/unbound_method.rb:18:in `bind_call'
        from <internal:core> core/kernel.rb:272:in `require'
<internal:core> core/kernel.rb:236:in `gem_original_require': cannot load such file -- google/3.0/protobuf_c (LoadError)
        from <internal:/opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/mri/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/gems/gems/google-protobuf-3.21.12/lib/google/protobuf.rb:49:in `<top (required)>'
        from <internal:core> core/kernel.rb:234:in `gem_original_require'
        from <internal:/opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/mri/rubygems/core_ext/kernel_require.rb>:160:in `require'
        from <internal:core> core/unbound_method.rb:18:in `bind_call'
        from <internal:core> core/kernel.rb:272:in `require'
<internal:core> core/kernel.rb:236:in `gem_original_require': cannot load such file -- google/protobuf (LoadError)
        from <internal:/opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/mri/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:core> core/unbound_method.rb:18:in `bind_call'
        from <internal:core> core/kernel.rb:272:in `require'

I thought this might be a GraalVM quirk, so I tried LD_LIBRARY_PATH=/opt/graalvm-ce-java11-22.3.1/languages/llvm/native/lib/ ruby -rgoogle/protobuf -ve0.

# LD_LIBRARY_PATH=/opt/graalvm-ce-java11-22.3.1/languages/llvm/native/lib/ ruby -rgoogle/protobuf -ve0
truffleruby 22.3.1, like ruby 3.0.3, GraalVM CE Native [aarch64-linux]
[To redirect Truffle log output to a file use one of the following options:
* '--log.file=<path>' if the option is passed using a guest language launcher.
* '-Dpolyglot.log.file=<path>' if the option is passed using the host Java launcher.
* Configure logging using the polyglot embedding API.]
[ruby] WARNING: Encoding.find('locale') is US-ASCII, this often indicates that the system locale is not set properly. Set LANG=en_US.UTF-8 and see https://www.graalvm.org/dev/reference-manual/ruby/UTF8Locale/ for details.
<internal:core> core/kernel.rb:234:in `gem_original_require': /opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/gems/gems/google-protobuf-3.21.12/lib/google/protobuf_c.so: undefined symbol: rb_eRuntimeError (RuntimeError)
        from <internal:/opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/mri/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/gems/gems/google-protobuf-3.21.12/lib/google/protobuf.rb:51:in `<top (required)>'
        from <internal:core> core/kernel.rb:234:in `gem_original_require'
        from <internal:/opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/mri/rubygems/core_ext/kernel_require.rb>:160:in `require'
        from <internal:core> core/unbound_method.rb:18:in `bind_call'
        from <internal:core> core/kernel.rb:272:in `require'
<internal:core> core/kernel.rb:236:in `gem_original_require': cannot load such file -- google/3.0/protobuf_c (LoadError)
        from <internal:/opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/mri/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/gems/gems/google-protobuf-3.21.12/lib/google/protobuf.rb:49:in `<top (required)>'
        from <internal:core> core/kernel.rb:234:in `gem_original_require'
        from <internal:/opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/mri/rubygems/core_ext/kernel_require.rb>:160:in `require'
        from <internal:core> core/unbound_method.rb:18:in `bind_call'
        from <internal:core> core/kernel.rb:272:in `require'
<internal:core> core/kernel.rb:236:in `gem_original_require': cannot load such file -- google/protobuf (LoadError)
        from <internal:/opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/mri/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:core> core/unbound_method.rb:18:in `bind_call'
        from <internal:core> core/kernel.rb:272:in `require'

Only option 1 would work for me; I had to remove this line. https://github.com/protocolbuffers/protobuf/blob/c4ed9483146ac633a9f725f307f9ac60074d0f4d/ruby/ext/google/protobuf_c/extconf.rb#L17

Then

bundle && rake && rake clobber_package gem && gem install `ls pkg/google-protobuf-*.gem`

succeeds.

Anything else we should know about your project / environment

 ruby -rrbconfig -e 'puts RbConfig::CONFIG["CFLAGS"]'
...
  -Werror=implicit-function-declaration -Wno-int-conversion -Wno-int-to-pointer-cast -Wno-incompatible-pointer-types -Wno-format-invalid-specifier -Wno-format-extra-args -ferror-limit=500

-flto=full is somehow always added when compiling native extensions, however, unlike @sorah , it is not in my RbConfig::CONFIG["CFLAGS"]'.

/opt/graalvm-ce-java11-22.3.1/lib/llvm/bin/clang -I/opt/graalvm-ce-java11-22.3.1/languages/llvm/native/include -I/opt/graalvm-ce-java11-22.3.1/languages/llvm/include -stdlib=libc++ -Wno-unused-command-line-argument -flto=full -O1 -gdwarf-5 -I. -I/opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/cext/include -I/opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/cext/include/ruby/backward -I/opt/graalvm-ce-java11-22.3.1/languages/ruby/lib/cext/include -I../third_party/utf8_range -DTRUFFLERUBY_ABI_VERSION=3.0.3.22.3.1 -fPIC -Werror=implicit-function-declaration -Wno-int-conversion -Wno-int-to-pointer-cast -Wno-incompatible-pointer-types -Wno-format-invalid-specifier -Wno-format-extra-args -ferror-limit=500 -std=gnu99 -O3 -DNDEBUG -fvisibility=hidden -Wall -Wsign-compare -Wno-declaration-after-statement -o message.o -c message.c                                                                            

I did not encounter this issue or have to do anything special with the x86_64 version of GraalVM, truffleruby, docker container, etc. -flto=full is also being used in x86_64. It seems there is some strange interaction between enabling LTO and the dynamic linking process on my specific aarch64 environment, but I cannot explain why.

In addition to option 1, maybe adding a build flag like --disable-memcpy-compat to gem install would be better, seeing as the motive was backwards compatibility https://github.com/protocolbuffers/protobuf/commit/9fa40314fcfd19d2e09459faeb480d213af56324.

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago.

[sorah]

this issue is still active

[haberman]

@dazuma could you take a look? This relates to the binary package.

It may be that this wrap_memcpy() issue is obsolete, and the wrapper can simply be removed. There was a binary compatibility issue a while ago where binaries built against a new glibc would fail to load on systems with an older glibc.

If we still need to keep the wrapper, we could do one of the two things suggested in the issue report.

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago.

[joeyparis]

This issue is still active.

OS: Amazon Linux 2023 Ruby: 3.2.2 Bundler: 2.4.10 Gem: 3.4.10

It seems to work fine when installing directly via gem install, but this error occurs when running bundle install.

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago. This issue will be closed and archived after 14 additional days without activity.

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please reopen it.

This issue was closed and archived because there has been no new activity in the 14 days since the inactive label was added.

[sorah]

this issue is still active.