Need updated Linux profiler build process that works for Alpine Linux

[nr-ahemsath]

See #918 for more context.

PR #877 attempted to update our Linux profiler build process to use a more modern base container image and llvm toolchain. However, we forgot to test the results on Alpine Linux. When using the profiler built with that new build process on Alpine, a profiled app would crash immediately.

Looking at ldd output on libNewRelicProfiler.so on Alpine between the old build process and the new one from PR #877, it's clear there's something wrong:

ldd output from profiler built with old build process

~/9.4.0/newrelic-netcore20-agent # ldd libNewRelicProfiler.so
        /lib/ld-musl-x86_64.so.1 (0x7f52560ca000)
        libm.so.6 => /lib/ld-musl-x86_64.so.1 (0x7f52560ca000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7f5255ca4000)
        libpthread.so.0 => /lib/ld-musl-x86_64.so.1 (0x7f52560ca000)
        libc.so.6 => /lib/ld-musl-x86_64.so.1 (0x7f52560ca000)
Error relocating libNewRelicProfiler.so: RaiseException: symbol not found
Error relocating libNewRelicProfiler.so: strtoll_l: symbol not found
Error relocating libNewRelicProfiler.so: strtoull_l: symbol not found

ldd output from profiler built with new build process

~/newrelic-netcore20-agent # ldd libNewRelicProfiler.so
        /lib/ld-musl-x86_64.so.1 (0x7f8b4f159000)
        libm.so.6 => /lib/ld-musl-x86_64.so.1 (0x7f8b4f159000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7f8b4ed63000)
        libpthread.so.0 => /lib/ld-musl-x86_64.so.1 (0x7f8b4f159000)
        libc.so.6 => /lib/ld-musl-x86_64.so.1 (0x7f8b4f159000)
Error relocating libNewRelicProfiler.so: __cxa_rethrow_primary_exception: symbol not found
Error relocating libNewRelicProfiler.so: __cxa_get_exception_ptr: symbol not found
Error relocating libNewRelicProfiler.so: __cxa_uncaught_exception: symbol not found
Error relocating libNewRelicProfiler.so: __cxa_free_exception: symbol not found
Error relocating libNewRelicProfiler.so: __cxa_begin_catch: symbol not found
Error relocating libNewRelicProfiler.so: _ZNSt8bad_castC1Ev: symbol not found
Error relocating libNewRelicProfiler.so: __cxa_current_primary_exception: symbol not found
Error relocating libNewRelicProfiler.so: _ZNSt13runtime_errorD2Ev: symbol not found
Error relocating libNewRelicProfiler.so: _ZSt15get_new_handlerv: symbol not found
Error relocating libNewRelicProfiler.so: RaiseException: symbol not found
Error relocating libNewRelicProfiler.so: __cxa_guard_release: symbol not found
Error relocating libNewRelicProfiler.so: __vsnprintf_chk: symbol not found
Error relocating libNewRelicProfiler.so: __cxa_call_unexpected: symbol not found
Error relocating libNewRelicProfiler.so: strtoll_l: symbol not found
Error relocating libNewRelicProfiler.so: __cxa_allocate_exception: symbol not found
Error relocating libNewRelicProfiler.so: __cxa_increment_exception_refcount: symbol not found
Error relocating libNewRelicProfiler.so: __cxa_throw: symbol not found
Error relocating libNewRelicProfiler.so: __cxa_guard_abort: symbol not found
Error relocating libNewRelicProfiler.so: _ZNSt9bad_allocC1Ev: symbol not found
Error relocating libNewRelicProfiler.so: __cxa_end_catch: symbol not found
Error relocating libNewRelicProfiler.so: __cxa_guard_acquire: symbol not found
Error relocating libNewRelicProfiler.so: strtoull_l: symbol not found
Error relocating libNewRelicProfiler.so: __dynamic_cast: symbol not found
Error relocating libNewRelicProfiler.so: __cxa_decrement_exception_refcount: symbol not found
Error relocating libNewRelicProfiler.so: __cxa_rethrow: symbol not found
Error relocating libNewRelicProfiler.so: _ZNSt11logic_errorD2Ev: symbol not found
Error relocating libNewRelicProfiler.so: __vasprintf_chk: symbol not found
Error relocating libNewRelicProfiler.so: _ZSt9terminatev: symbol not found
Error relocating libNewRelicProfiler.so: _ZTVN10__cxxabiv117__class_type_infoE: symbol not found
Error relocating libNewRelicProfiler.so: _ZTVN10__cxxabiv117__class_type_infoE: symbol not found
Error relocating libNewRelicProfiler.so: _ZTVN10__cxxabiv117__class_type_infoE: symbol not found
Error relocating libNewRelicProfiler.so: _ZTVN10__cxxabiv117__class_type_infoE: symbol not found
Error relocating libNewRelicProfiler.so: _ZTVN10__cxxabiv117__class_type_infoE: symbol not found
Error relocating libNewRelicProfiler.so: _ZTVN10__cxxabiv117__class_type_infoE: symbol not found
Error relocating libNewRelicProfiler.so: _ZTVN10__cxxabiv117__class_type_infoE: symbol not found
Error relocating libNewRelicProfiler.so: _ZTVN10__cxxabiv117__class_type_infoE: symbol not found
....
way too many lines to fit in an issue

Searching online for "error relocating" and "Alpine Linux" indicates that the problem is caused by glibc vs musl compatibility problems: https://www.musl-libc.org/faq.html

It would be annoying to have a separate profiler build to target musl-based systems, but we might have to think about doing that. However, for what it is worth, it doesn't look like Microsoft's own process for building the .NET runtime for Linux from source has an option to specific Alpine or musl specifically: https://github.com/dotnet/runtime

On the other hand, Microsoft does have separate .NET SDKs for Alpine, and there's a separate runtime identifier (RID) used for publishing .NET applications for musl-based Linux systems (linux-musl-x64 as opposed to linux-x64 for glibc-based Linux: https://docs.microsoft.com/en-us/dotnet/core/rid-catalog#linux-rids

I gathered some data from building the old way vs. the new way (the build process output and the auto-generated makefiles) and am attaching it here: linux-alpine-profiler-build-problems.zip However, while there are some differences, nothing jumps out as an obvious smoking gun.

[workato-integration[bot]]

https://issues.newrelic.com/browse/NEWRELIC-3675

[tippmar-nr]

If we come back to this at some point, there's preliminary work related to updating the Linux build process to support Arm64 and to use more recent base images in https://github.com/newrelic/newrelic-dotnet-agent/tree/tippmar-nr/arm64-docker-build - the changes in that branch successfully build both Arm64 and x64 versions using very a recent Ubuntu base image and recent llvm (clang / lldb) packages.

[workato-integration[bot]]

This issue requires more information and/or work.