Update llvmdev to build compiler-rt

[oliverhu]

As part of https://github.com/numba/llvmlite/pull/976 discussion, split the llvmdev build into a separate PR for easier testing.

Validated on Ubuntu, it builds llvm with compiler-rt properly.

❯ tar xvf llvmdev-14.0.6-h84b9e52_3.tar.bz2 
❯ find . -name "libclang*"
./lib/clang/14.0.6/lib/linux/libclang_rt.builtins-x86_64.a

[gmarkall]

I'm experimenting with building the package locally on Linux to begin with - will post any feedback after some experimentation.

[gmarkall]

Something odd about the build on macOS AArch64 - a snippet of the output:

ld: warning: ignoring file /Users/gmarkall/mambaforge/conda-bld/llvmdev_1691648028380/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib/libc++.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
[ 65%] Building ASM object projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins_arm64_ios.dir/outline_atomic_helpers.dir/outline_atomic_ldadd1_3.S.o
[ 65%] Building ASM object projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.builtins_arm64e_osx.dir/outline_atomic_helpers.dir/outline_atomic_ldadd2_2.S.o
cd $SRC_DIR/build/projects/compiler-rt/lib/builtins && $BUILD_PREFIX/bin/arm64-apple-darwin20.0.0-clang -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -DL_ldadd -DMODEL=3 -DSIZE=1 -I$SRC_DIR/compiler-rt/lib/builtins -I$SRC_DIR/build/projects/compiler-rt/lib/builtins -I$SRC_DIR/compiler-rt/lib/builtins -I$SRC_DIR/build/include -I$SRC_DIR/llvm/include -O3 -DNDEBUG -arch arm64  -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS16.4.sdk -miphoneos-version-min=6.0 -fPIC -O3 -fvisibility=hidden -DVISIBILITY_HIDDEN -Wall -fomit-frame-pointer  -DHAS_ASM_LSE -arch arm64 -o CMakeFiles/clang_rt.builtins_arm64_ios.dir/outline_atomic_helpers.dir/outline_atomic_ldadd1_3.S.o -c $SRC_DIR/build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_ldadd1_3.S
cd $SRC_DIR/build/projects/compiler-rt/lib/builtins && $BUILD_PREFIX/bin/arm64-apple-darwin20.0.0-clang -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -DL_ldadd -DMODEL=2 -DSIZE=2 -I$SRC_DIR/compiler-rt/lib/builtins -I$SRC_DIR/build/projects/compiler-rt/lib/builtins -I$SRC_DIR/compiler-rt/lib/builtins -I$SRC_DIR/build/include -I$SRC_DIR/llvm/include -O3 -DNDEBUG -arch arm64e  -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk -mmacosx-version-min=10.5 -fPIC -O3 -fvisibility=hidden -DVISIBILITY_HIDDEN -Wall -fomit-frame-pointer  -DHAS_ASM_LSE -arch arm64e -o CMakeFiles/clang_rt.builtins_arm64e_osx.dir/outline_atomic_helpers.dir/outline_atomic_ldadd2_2.S.o -c $SRC_DIR/build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_ldadd2_2.S
[ 65%] Building ASM object projects/compiler-rt/lib/builtins/CMakeFiles/clang_rt.cc_kext_arm64_osx.dir/outline_atomic_helpers.dir/outline_atomic_ldadd2_1.S.o
cd $SRC_DIR/build/projects/compiler-rt/lib/builtins && $BUILD_PREFIX/bin/arm64-apple-darwin20.0.0-clang -DKERNEL_USE -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -DL_ldadd -DMODEL=1 -DSIZE=2 -I$SRC_DIR/compiler-rt/lib/builtins -I$SRC_DIR/build/projects/compiler-rt/lib/builtins -I$SRC_DIR/compiler-rt/lib/builtins -I$SRC_DIR/build/include -I$SRC_DIR/llvm/include -I$SRC_DIR/compiler-rt/lib/builtins/../../include -O3 -DNDEBUG -arch arm64  -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk -mmacosx-version-min=10.5 -fPIC -O3 -fvisibility=hidden -DVISIBILITY_HIDDEN -Wall -fomit-frame-pointer  -DHAS_ASM_LSE -arch arm64 -mkernel -o CMakeFiles/clang_rt.cc_kext_arm64_osx.dir/outline_atomic_helpers.dir/outline_atomic_ldadd2_1.S.o -c $SRC_DIR/build/projects/compiler-rt/lib/builtins/outline_atomic_helpers.dir/outline_atomic_ldadd2_1.S
Undefined symbols for architecture x86_64:
  "std::__1::__call_once(unsigned long volatile&, void*, void (*)(void*))", referenced from:
      _wrap__ZNSt3__111__call_onceERVmPvPFvS2_E in tsan_interceptors_mac.cpp.o
      __tsan::substitution__ZNSt3__111__call_onceERVmPvPFvS2_E in tsan_interceptors_mac.cpp.o
     (maybe you meant: _wrap__ZNSt3__111__call_onceERVmPvPFvS2_E)
  "std::__1::__shared_count::__release_shared()", referenced from:
      _wrap__ZNSt3__114__shared_count16__release_sharedEv in tsan_interceptors_mac.cpp.o
      __tsan::substitution__ZNSt3__114__shared_count16__release_sharedEv in tsan_interceptors_mac.cpp.o
     (maybe you meant: _wrap__ZNSt3__114__shared_count16__release_sharedEv)
  "std::__1::__shared_weak_count::__release_shared()", referenced from:
      _wrap__ZNSt3__119__shared_weak_count16__release_sharedEv in tsan_interceptors_mac.cpp.o
      __tsan::substitution__ZNSt3__119__shared_weak_count16__release_sharedEv in tsan_interceptors_mac.cpp.o
     (maybe you meant: _wrap__ZNSt3__119__shared_weak_count16__release_sharedEv)
ld: symbol(s) not found for architecture x86_64
clang-16: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [projects/compiler-rt/lib/tsan/rtl/CMakeFiles/clang_rt.tsan_osx_dynamic.dir/build.make:787: lib/clang/14.0.6/lib/darwin/libclang_rt.tsan_osx_dynamic.dylib] Error 1
make[2]: Leaving directory '$SRC_DIR/build'
make[1]: *** [CMakeFiles/Makefile2:31299: projects/compiler-rt/lib/tsan/rtl/CMakeFiles/clang_rt.tsan_osx_dynamic.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

I'll have to check whether the package builds on main next, or whether this is an issue introduced with this PR.

[gmarkall]

The package built OK on main on macOS AArch64, so I guess the issue is with the addition of building compiler-rt.

[gmarkall]

The error seems to be coming from the ThreadSanitizer build, which should be supported even in 14 on macOS AArch64: https://releases.llvm.org/14.0.0/tools/clang/docs/ThreadSanitizer.html

[gmarkall]

I found a hardcoded x86_64 in the build.sh - will post more info if changing that fixes the issue.

[gmarkall]

The following (or something a little more elegant) also needs to be applied because it influences the build of compiler-rt:

diff --git a/conda-recipes/llvmdev/build.sh b/conda-recipes/llvmdev/build.sh
index ef21f54..6a5558d 100644
--- a/conda-recipes/llvmdev/build.sh
+++ b/conda-recipes/llvmdev/build.sh
@@ -10,8 +10,10 @@ LLVM_TARGETS_TO_BUILD=${LLVM_TARGETS_TO_BUILD:-"all"}
 # This is the clang compiler prefix
 if [[ $build_platform == osx-arm64 ]]; then
     DARWIN_TARGET=arm64-apple-darwin20.0.0
+    DARWIN_ARCH=arm64
 else
     DARWIN_TARGET=x86_64-apple-darwin13.4.0
+    DARWIN_ARCH=x86_64
 fi
 
 mv llvm-*.src llvm
@@ -74,7 +76,7 @@ if [[ $(uname) == Darwin ]]; then
   # Once we are using our libc++ (not until llvm_build_final), it will be single-arch only and not setting
   # this causes link failures building the santizers since they respect DARWIN_osx_ARCHS. We may as well
   # save some compilation time by setting this for all of our llvm builds.
-  _cmake_config+=(-DDARWIN_osx_ARCHS=x86_64)
+  _cmake_config+=(-DDARWIN_osx_ARCHS=${DARWIN_ARCH})
 elif [[ $(uname) == Linux ]]; then
   _cmake_config+=(-DLLVM_USE_INTEL_JITEVENTS=ON)
 #  _cmake_config+=(-DLLVM_BINUTILS_INCDIR=${PREFIX}/lib/gcc/${cpu_arch}-${vendor}-linux-gnu/${compiler_ver}/plugin/include)

[oliverhu]

Got this error on my macos (M1), looking

Testing compiler for supporting FLAGS-fxray-instrument:
Command: /Users/khu/miniconda3/envs/rt/conda-bld/llvmdev_1691732858678/_build_env/bin/arm64-apple-darwin20.0.0-clang;;;-ftree-vectorize;-fPIC;-fPIE;-fstack-protector-strong;-O2;-pipe;-isystem;/Users/khu/miniconda3/envs/rt/conda-bld/llvmdev_1691732858678/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/include;-fdebug-prefix-map=/Users/khu/miniconda3/envs/rt/conda-bld/llvmdev_1691732858678/work=/usr/local/src/conda/llvmdev-14.0.6;-fdebug-prefix-map=/Users/khu/miniconda3/envs/rt/conda-bld/llvmdev_1691732858678/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh=/usr/local/src/conda-prefix;-fPIC;-Werror=date-time;-Werror=unguarded-availability-new;-Wall;-Wextra;-Wno-unused-parameter;-Wwrite-strings;-Wmissing-field-initializers;-pedantic;-Wno-long-long;-Wc++98-compat-extra-semi;-Wimplicit-fallthrough;-Wcovered-switch-default;-Wstring-conversion;-Wmisleading-indentation;-Wall;-Wno-unused-parameter;-fxray-instrument;-o;/Users/khu/miniconda3/envs/rt/conda-bld/llvmdev_1691732858678/work/build/CMakeFiles/test.o;-c;/Users/khu/miniconda3/envs/rt/conda-bld/llvmdev_1691732858678/work/build/CMakeFiles/src.c

clang-14: error: the clang compiler does not support '-fxray-instrument on arm64-apple-darwin20.0.0'

[oliverhu]

Skipping those binaries for now:

_cmake_config+=(-DCOMPILER_RT_BUILD_BUILTINS:BOOL=ON)
_cmake_config+=(-DCOMPILER_RT_BUILD_LIBFUZZER:BOOL=OFF)
_cmake_config+=(-DCOMPILER_RT_BUILD_CRT:BOOL=OFF)
_cmake_config+=(-DCOMPILER_RT_BUILD_MEMPROF:BOOL=OFF)
_cmake_config+=(-DCOMPILER_RT_BUILD_PROFILE:BOOL=OFF)
_cmake_config+=(-DCOMPILER_RT_BUILD_SANITIZERS:BOOL=OFF)
_cmake_config+=(-DCOMPILER_RT_BUILD_XRAY:BOOL=OFF)
_cmake_config+=(-DCOMPILER_RT_BUILD_GWP_ASAN:BOOL=OFF)
_cmake_config+=(-DCOMPILER_RT_BUILD_ORC:BOOL=OFF)
_cmake_config+=(-DCOMPILER_RT_INCLUDE_TESTS:BOOL=OFF)

[oliverhu]

Validated in macOS & Linux

❯ nm -a ./lib/clang/14.0.6/lib/darwin/libclang_rt.osx.a | grep f2h
000000000000011c T ___gnu_f2h_ieee

./lib/clang/14.0.6/lib/linux/libclang_rt.builtins-x86_64.a
❯ readelf -s -A ./lib/clang/14.0.6/lib/linux/libclang_rt.builtins-x86_64.a | grep f2h
     7: 0000000000000000     0 SECTION LOCAL  DEFAULT    5 .text.__gnu_f2h_ieee
    12: 0000000000000000     5 FUNC    GLOBAL DEFAULT    5 __gnu_f2h_ieee

[oliverhu]

Windows:

C:\Users\test\source\repos>dumpbin /symbols C:\Users\khu\llvm\build\llvmdev-14.0.6-4\Library\lib\windows\clang_rt.builtins-x86_64.lib
...
00B 00000000 SECT4  notype ()    External     | __truncsfhf2
00C 00000000 SECT3  notype ()    External     | __gnu_f2h_ieee

[oliverhu]

Validated llvmdev_manylinux2014 via ./buildscripts/manylinux_2_17/docker_run_x64.sh build_llvmdev.sh, instead of conda-build llvmdev_manylinux2014

+ exit 0

Resource usage statistics from testing llvmdev:
   Process count: 2
   CPU time: Sys=-, User=0:00:00.0
   Memory: 12.0M
   Disk usage: 16B
   Time elapsed: 0:00:02.0


TEST END: /root/miniconda3/envs/buildenv/conda-bld/linux-64/llvmdev-14.0.6-manylinux2014.tar.bz2

[oliverhu]

Added a patch for macOS build in latest commit to expose builtin symbols