Optionally enable ASAN for Ruby extension builds

[eightbitraptor]

When building the Ruby native extension, one can provide the environment variable PROTOBUF_CONFIG with the value dbg to conditionally enable debug symbols and disable compiler optimisations. However, there’s no way to configure or enable Protobuf’s address sanitization support from Ruby.

This PR adds asan as a valid PROTOBUF_CONFIG option. Building with PROTOBUF_CONFIG=asan will enable the same debug options as dbg, but will also add -fsanitize=address to both cflags and ldflags in order to link against AddressSanitizer.

Running with ASAN

When a native extension is loaded using require, Ruby uses dlopen to load the shared object.

When a native extension is loaded using require, Ruby uses dlopen to load the shared objects. The consequence of this, is that extensions are loaded after `libc`. For extensions compiled with ASAN this will result in an error because ASAN needs to be loaded before libc in order to correctly install the interceptors:

❯ ~/workspaces/protobuf-ruby/folders/ruby/target/bin/ruby -Ilib ~/workspaces/protobuf-ruby/folders/resources/test.rb
==78689==ERROR: Interceptors are not working. This may be because AddressSanitizer is loaded too late (e.g. via dlopen). Please launch the executable with:
DYLD_INSERT_LIBRARIES=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/17/lib/darwin/libclang_rt.asan_osx_dynamic.dylib
"interceptors not installed" && 0
fish: Job 2, '~/workspaces/protobuf-ruby/fold…' terminated by signal SIGABRT (Abort)

In order to correctly initialise the interceptors, it is necessary to set either LD_PRELOAD, or DYLD_INSERT_LIBRARIES as instructed.

This limitation has been documented in the README

[google-cla[bot]]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[eightbitraptor]

@JasonLunn Apologies, but I originally pushed this PR with the incorrect email address associated with the commits. I've updated to use my corporate address and force pushed, but the CLA bot hasn't restarted - Is there anything special I need to do?

[JasonLunn]

@JasonLunn Apologies, but I originally pushed this PR with the incorrect email address associated with the commits. I've updated to use my corporate address and force pushed, but the CLA bot hasn't restarted - Is there anything special I need to do?

I retriggered the CLA check on the latest commit, but it still fails because it still sees an author on the commit that hasn't signed the CLA:

https://github.com/protocolbuffers/protobuf/pull/22176/checks?check_run_id=43792226887

[eightbitraptor]

Ok. Problem addressed on my end, sorry for the confusion. It looks like I messed up re-authoring one of the commits. I've fixed that, signed the CLA and verified that the check is passing now. Thanks for your help.

[JasonLunn]

I've done some work on getting Ruby tests to run under ASAN in the past, and to the best of my understanding, one needs to compile the Ruby interpreter with ASAN first. Without that, tests will fail loudly in interpreter code. However, once the Ruby interpreter is built for ASAN, I am not sure you still need the environment variable manipulation in the PR.

Have you tried running the test suite locally under ASAN?

[eightbitraptor]

Hey Jason - Yes, I have tried running the test suite under ASAN and you're correct. Running the test suite will require a Ruby built with ASAN support at which point injecting the ASAN library becomes unnecessary because Ruby's already done it.

The environment variables are useful when running more focussed, smaller test scripts when one is interested purely in the native extension code - this was the case I had in mind when implementing this, although I appreciate I didn't clarify that in the documentation.

What do you think the best approach to resolve this is? I'm happy to expand the documentation to explain the different use cases and mention the need for a custom Ruby build when running the full test suite (Ruby has good documentation already on how to build with ASAN)

[JasonLunn]

Can you help me better understand an example of one of the "smaller test scripts when one is interested purely in the native extension code" where running under ASAN is helpful without building Ruby under ASAN?