Statically link compiler-rt

[oliverhu]

This PR addesses issues in https://github.com/numba/llvmlite/issues/834 and https://github.com/tinygrad/tinygrad/issues/1367, in which libllvmlite.so fails to resolve symbols in libclang_rt.builtin.a since compiler rt is not linked. This PR statically links the compiler_rt library.

(address the comments from https://github.com/numba/llvmlite/pull/976)

[oliverhu]

@gmarkall found the problem with the test...the issue is related to cdll loading logic, left a comment in the code: https://github.com/numba/llvmlite/pull/986/files#diff-a23ac0d3c4c6d8093d709eb3c5d28fc4dadae4e58e648ba306485a7f4eddd776R119

[oliverhu]

After more investigation... I reverted the build change for macOS & Windows, and added a test for Linux.

Issues for macOS:

1. For M1 Macbook, there is native fp16 register so I can't reproduce the missing symbol issue for __gnu_f2h_ieee.
2. The clang linker in macOS behaves different than GNU. For example: libclang_rt.osx.a has visible symbols ___divtc3, verified via otool as well and there is no hidden flag.

nm /Users/khu/miniconda3/envs/rt/lib/clang/14.0.6/lib/darwin/libclang_rt.osx.a | grep ___divtc3
0000000000000000 T ___divtc3

However, linker complains the ___divtc3 symbol is hidden if I try to export that symbol..

❯ clang++ -dynamiclib -o dummy.dylib  -L/Users/khu/miniconda3/envs/rt/lib -Wl,-search_paths_first  /Users/khu/miniconda3/envs/rt/lib/clang/14.0.6/lib/darwin/libclang_rt.osx.a  -Wl,-exported_symbol,___divtc3
ld: warning: cannot export hidden symbol ___divtc3 from /Users/khu/miniconda3/envs/rt/lib/clang/14.0.6/lib/darwin/libclang_rt.osx.a(divtc3.c.o)

Issues for Windows

I'm mainly struggling with dev tooling to figure out what is missing, applied the following diff ( /WHOLEARCHIVE is the Windows equivalent of --no-whole-archive):

diff --git a/ffi/CMakeLists.txt b/ffi/CMakeLists.txt
index 07ab5ec..c6b843e 100755
--- a/ffi/CMakeLists.txt
+++ b/ffi/CMakeLists.txt
@@ -23,7 +23,7 @@ else()
 endif()

 if (${CMAKE_SYSTEM_NAME} MATCHES "Windows")
-    set(COMPILER_RT_LOCATION ${LLVM_LIBRARY_DIR}/windows/clang_rt.builtins-${ARCH}.lib)
+    set(COMPILER_RT_LOCATION ${LLVM_LIBRARY_DIR}/clang/${LLVM_PACKAGE_VERSION}/lib/windows/clang_rt.builtins-${ARCH}.lib)
 else()
     if (EXISTS "$(LLVM_LIBDIR)/clang")
         set(COMPILER_RT_LOCATION ${LLVM_LIBRARY_DIR}/linux/libclang_rt.builtins-${ARCH}.a)
@@ -65,8 +65,12 @@ add_library(llvmlite SHARED assembly.cpp bitcode.cpp core.cpp initfini.cpp
             custom_passes.cpp orcjit.cpp)

 # Link compiler-rt whole archive
+#set_target_properties(llvmlite PROPERTIES
+ #   LINK_FLAGS " -Wl,--whole-archive,${COMPILER_RT_LOCATION},--no-whole-archive ")
+
+target_link_libraries(llvmlite ${COMPILER_RT_LOCATION})
 set_target_properties(llvmlite PROPERTIES
-    LINK_FLAGS "-Lfoobar -Wl,--whole-archive,${COMPILER_RT_LOCATION},--no-whole-archive ")
+    LINK_FLAGS " /WHOLEARCHIVE:${COMPILER_RT_LOCATION} ")

However, when I dump the symbols of the generated libllvmlite.lib and libllvmlite.dll, there is no symbol:

Dump of file C:\Users\khu\llvm\llvmlite\ffi\build\Release\llvmlite.dll

File Type: DLL

  Summary

       BE000 .data
      149000 .pdata
     217C000 .rdata
       A0000 .reloc
        1000 .rsrc
     2A8C000 .text

C:\Program Files (x86)\Microsoft Visual Studio\2019\Community>dumpbin /symbols   C:/Users/khu/llvm/llvmlite/ffi/build/Release/llvmlite.lib
Microsoft (R) COFF/PE Dumper Version 14.29.30151.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file C:\Users\khu\llvm\llvmlite\ffi\build\Release\llvmlite.lib

File Type: LIBRARY

COFF SYMBOL TABLE
000 010175C7 ABS    notype       Static       | @comp.id
001 00000000 SECT2  notype       External     | __IMPORT_DESCRIPTOR_llvmlite
002 C0000040 SECT2  notype       Section      | .idata$2
003 00000000 SECT3  notype       Static       | .idata$6
004 C0000040 UNDEF  notype       Section      | .idata$4
005 C0000040 UNDEF  notype       Section      | .idata$5
006 00000000 UNDEF  notype       External     | __NULL_IMPORT_DESCRIPTOR
007 00000000 UNDEF  notype       External     | ⌂llvmlite_NULL_THUNK_DATA

String Table Size = 0x54 bytes

COFF SYMBOL TABLE
000 010175C7 ABS    notype       Static       | @comp.id
001 00000000 SECT2  notype       External     | __NULL_IMPORT_DESCRIPTOR

String Table Size = 0x1D bytes

COFF SYMBOL TABLE
000 010175C7 ABS    notype       Static       | @comp.id
001 00000000 SECT2  notype       External     | ⌂llvmlite_NULL_THUNK_DATA

String Table Size = 0x1E bytes

  Summary

          C6 .debug$S
          14 .idata$2
          14 .idata$3
           8 .idata$4
           8 .idata$5
           E .idata$6

Shall we fix the Linux version first?

[oliverhu]

Additional debugging for macOS, got some help from llvm discourse: https://discourse.llvm.org/t/lld-automatically-hide-symbols-with-prefix/73192, nm -m shows the symbols are private external.

/Users/khu/miniconda3/envs/rt/lib/clang/14.0.6/lib/darwin/libclang_rt.osx.a(divtc3.c.o):
0000000000000000 (__TEXT,__text) private external ___divtc3
                 (undefined) external _logbl
                 (undefined) external _scalbnl

[oliverhu]

After checking out the compiler-rt source, It seems we need to turn off COMPILER_RT_BUILTINS_HIDE_SYMBOLS flag, and also need to fix an issue that symbol visibility was hard coded to be off in the Darwin cmake module (compiler-rt-14.0.6.src/cmake/Modules/CompilerRTDarwinUtils.cmake), something like this:

macro(darwin_add_builtin_libraries)
  set(DARWIN_EXCLUDE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/Darwin-excludes)

-set(CFLAGS "-fPIC -O3 -fvisibility=hidden -DVISIBILITY_HIDDEN -Wall -fomit-frame-pointer")
+set(CFLAGS "-fPIC -O3  -Wall -fomit-frame-pointer")
+append_list_if(COMPILER_RT_BUILTINS_HIDE_SYMBOLS -fvisibility=hidden -DVISIBILITY_HIDDEN CFLAGS)
  set(CMAKE_C_FLAGS "")
  set(CMAKE_CXX_FLAGS "")
  set(CMAKE_ASM_FLAGS "")

After the change:

❯ nm -m ./lib/darwin/libclang_rt.osx.a | grep ___divtc3
0000000000000000 (__TEXT,__text) external ___divtc3

[esc]

@gmarkall @oliverhu -- wanted to add a label to this. What state is it in?

[gmarkall]

@esc It's waiting on me - I've been struggling to find the time to get back to progressing with it.

[gmarkall]

(It's waiting for #979, which is also waiting for me)

[esc]

@gmarkall ok, thank you for the info. I have updated the labels accordingly.