llvm-mingw Issues building all-static runtimes

Sometimes I encounter these errors:

ld.lld: error: undefined symbol: __dynamic_cast
>>> referenced by /xc2/work_test/llvm-mingw/llvm-project/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cpp:245
>>>               lib/ubsan/CMakeFiles/RTUbsan_cxx.x86_64.dir/ubsan_type_hash_itanium.cpp.obj:(__ubsan::checkDynamicType(void*, void*, unsigned long long))
>>> referenced by /xc2/work_test/llvm-mingw/llvm-project/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cpp:137
>>>               lib/ubsan/CMakeFiles/RTUbsan_cxx.x86_64.dir/ubsan_type_hash_itanium.cpp.obj:(isDerivedFromAtOffset(__cxxabiv1::__class_type_info const*, __cxxabiv1::__class_type_info const*, long long))
>>> referenced by /xc2/work_test/llvm-mingw/llvm-project/compiler-rt/lib/ubsan/ubsan_type_hash_itanium.cpp:141
>>>               lib/ubsan/CMakeFiles/RTUbsan_cxx.x86_64.dir/ubsan_type_hash_itanium.cpp.obj:(isDerivedFromAtOffset(__cxxabiv1::__class_type_info const*, __cxxabiv1::__class_type_info const*, long long))
>>> referenced 2 more times

ld.lld: error: undefined symbol: vtable for __cxxabiv1::__class_type_info
>>> referenced by lib/ubsan/CMakeFiles/RTUbsan_cxx.x86_64.dir/ubsan_type_hash_itanium.cpp.obj:(typeinfo for std::type_info)

ld.lld: error: undefined symbol: vtable for __cxxabiv1::__si_class_type_info
>>> referenced by lib/ubsan/CMakeFiles/RTUbsan_cxx.x86_64.dir/ubsan_type_hash_itanium.cpp.obj:(typeinfo for __cxxabiv1::__class_type_info)
>>> referenced by lib/ubsan/CMakeFiles/RTUbsan_cxx.x86_64.dir/ubsan_type_hash_itanium.cpp.obj:(typeinfo for __cxxabiv1::__si_class_type_info)
>>> referenced by lib/ubsan/CMakeFiles/RTUbsan_cxx.x86_64.dir/ubsan_type_hash_itanium.cpp.obj:(typeinfo for __cxxabiv1::__vmi_class_type_info)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[315/374] Building C object lib/profile/CMakeFiles/clang_rt.profile-x86_64.dir/InstrProfiling.c.obj
ninja: build stopped: subcommand failed

Other times it segfaults.

Any idea? EDIT: I tried looking more into it, might be related to trying to build statically.

On that note, what is the correct way to build the entire toolchain statically? Including omp and others.

Aug 22 '23 00:08 DeadSix27

This sounds weird - it sounds like there's some nondeterminism involved if it happens only occasionally, and if it crashes occasionally. I build this toolchain continuously in a number of configurations daily (both on GitHub Actions and on non-cloud machines), and haven't run into anything like it.

The error you're showing looks like what the sanitizer build shows when it fails to link the libc++abi bits (which are required since https://github.com/llvm/llvm-project/commit/076d351e8bdd98fdcc7291c437be82d1a8480d11, and linking them in is enabled with fcafcacc5850a840eb887386b611e4f5be61a291 in this repo).

EDIT: I tried looking more into it, might be related to trying to build statically.

Right, if you're tweaking the build process, those errors you showed are quite possibly caused by that. But it sounds weird if you're experiencing nondeterminism nevertheless.

On that note, what is the correct way to build the entire toolchain statically? Including omp and others.

I assume you mean making the toolchain only link runtime libraries statically (if we're looking at a cross compiler), not whether the toolchain itself is linked statically (not relying on the host glibc/libstdc++ or similar).

For libcxx/libunwind, it should work out by just passing --disable-shared to the build-libcxx.sh script. I guess something similar should be possible for winpthreads, but I haven't added such a flag to build-mingw-w64-libraries.sh yet. For OpenMP, I think it upstream only supports being built as a DLL. Not sure about if there are any specific technical reasons for it, or if it would work just as well if the cmake files were to be tweaked to allow it to be built statically as well. The sanitizers (in particular ASAN) is a case where this often is requested though, but as far as I know, those are only built as a DLL, and I think there are reasonable technical arguments for it being like that.

When trying out a build with --disable-shared passed to build-libcxx.sh I can reproduce the error message you quoted though.

(Originally, libcxx/libunwind were only built statically, but since 957c34372c1ce8765b1184d3ff7160f7642a2de9 in 2018 they're built both shared and static, and I've primarily tested the configuration with the shared version enabled. But since the change in late 2020 to make it use the libc++abi bits in ubsan, I probably haven't tested that configuration much since.)

The root cause seems to be that during the sanitizers build, it looks for the libc++abi bits. Those tests normally run like this:

-- Looking for __cxa_throw in c++
-- Looking for __cxa_throw in c++ - found

But in the case of a static build, the check fails - with errors like this within the cmake check:

        : && /home/martin/clang-static-test/bin/i686-w64-mingw32-clang --target=i686-w64-windows-gnu -DCHECK_FUNCTION_EXISTS=__cxa_throw  -nodefaultlibs  CMakeFiles/cmTC_4155c.dir/CheckFunctionExists.c.obj -o cmTC_4155c.exe -Wl,--out-implib,libcmTC_4155c.dll.a -Wl,--major-image-version,0,--minor-image-version,0  -lc++  -lmingw32  /home/martin/clang-static-test/lib/clang/17/lib/windows/libclang_rt.builtins-i386.a  -lmoldname  -lmingwex  -lmsvcrt  -ladvapi32  -lshell32  -luser32  -lkernel32  -lmingw32  /home/martin/clang-static-test/lib/clang/17/lib/windows/libclang_rt.builtins-i386.a  -lmoldname  -lmingwex  -lmsvcrt  -ladvapi32  -lshell32  -luser32  -lkernel32  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && :
        ld.lld: error: undefined symbol: __Unwind_RaiseException
        >>> referenced by libc++.a(cxa_exception.cpp.obj):(___cxa_throw)
        >>> referenced by libc++.a(cxa_exception.cpp.obj):(___cxa_rethrow)
        >>> referenced by libc++.a(cxa_exception.cpp.obj):(___cxa_rethrow_primary_exception)

It fails since it tests linking with -nodefaultlibs and then manually tries to link against -lc++ and other needed libraries, but it also would need to include -lunwind. (Before e35ef3b7d796a174eeb35fc2595281a3285a38ea in this repo, in late 2021, libunwind.a would be merged into libc++.a, so this issue wouldn't appear as libc++.a contained everything that's needed.)

I guess potential ways forward to fix this would be:

Add a way to disable the libc++abi bits when building ubsan. (As https://github.com/llvm/llvm-project/commit/076d351e8bdd98fdcc7291c437be82d1a8480d11 shows - this is kinda hardcoded in the ubsan source right now.) Or more automatically - if no checks for the libc++abi symbols succeed, disable this codepath.
Improve the libc++ detection in compiler-rt and make it test if -lunwind is available before checking for -lc++

Aug 31 '23 07:08 mstorsjo

I can't reliably reproduce the segfaults either, might possibly be hardware related (my linux test system is rather old), so never mind that for the time. But the static build is definitely something I'd want.

The 2nd method sounds more robust and less hacky.

EDIT: I haven't fully tested it but omp builds just fine if you remove the windows check.

EDIT2: with some more modifications, e.g fixing the lib naming of omp I managed to build it statically just fine. No more external libs/dlls needed. May I suggest that we/you add a switch/code for handling the needed arch's, in my case for example I have no use for aarch/etc all I need is x86-64. Having to manually remove the other archs from the loop is not pretty. If needed I could work on it and add a PR. As for static builds that should also be a switch which automatically adds the disable-shared options and co to every needed build conf.

As for the workarounds, we could either ask upstream for the fixes (e.g for omp) or manually patch the llvm clone within this repo.

EDIT3: that said; I just omitted the sanitizer build, because I have no idea how to fix the above mentioned issue.

as for omp again, this is all that's needed to get it to build just fine, so quite simple (can't remember wether LIBOMP_GENERATED_IMP_LIB_FILENAME needed a change as well):

diff --git a/openmp/runtime/CMakeLists.txt b/openmp/runtime/CMakeLists.txt
index 2b7a3eb5b..81a99d546 100644
--- a/openmp/runtime/CMakeLists.txt
+++ b/openmp/runtime/CMakeLists.txt
@@ -296,9 +296,9 @@ endif()
 set(LIBOMP_ENABLE_SHARED TRUE CACHE BOOL
   "Shared library instead of static library?")

-if(WIN32 AND NOT LIBOMP_ENABLE_SHARED)
-  libomp_error_say("Static libraries requested but not available on Windows")
-endif()
+# if(WIN32 AND NOT LIBOMP_ENABLE_SHARED)
+  # libomp_error_say("Static libraries requested but not available on Windows")
+# endif()

 if(LIBOMP_USE_ITT_NOTIFY AND NOT LIBOMP_ENABLE_SHARED)
   message(STATUS "ITT Notify not supported for static libraries - forcing ITT Notify off")
diff --git a/openmp/runtime/src/CMakeLists.txt b/openmp/runtime/src/CMakeLists.txt
index bb5822264..9e2cb005f 100644
--- a/openmp/runtime/src/CMakeLists.txt
+++ b/openmp/runtime/src/CMakeLists.txt
@@ -194,7 +194,7 @@ if(OPENMP_MSVC_NAME_SCHEME)
  )
 else()
   set_target_properties(omp PROPERTIES
-    PREFIX "" SUFFIX "" OUTPUT_NAME "${LIBOMP_LIB_FILE}"
+    PREFIX "" SUFFIX "" OUTPUT_NAME "${LIBOMP_IMP_LIB_FILE}"
     LINK_FLAGS "${LIBOMP_CONFIGURED_LDFLAGS}"
     LINKER_LANGUAGE ${LIBOMP_LINKER_LANGUAGE}
   )
@@ -259,7 +259,7 @@ if(WIN32)
     endif()
   else()
     set(LIBOMP_IMP_LIB_FILE ${LIBOMP_LIB_NAME}${CMAKE_IMPORT_LIBRARY_SUFFIX})
-    set(LIBOMP_GENERATED_IMP_LIB_FILENAME ${LIBOMP_LIB_FILE}${CMAKE_STATIC_LIBRARY_SUFFIX})
+    set(LIBOMP_GENERATED_IMP_LIB_FILENAME ${LIBOMP_LIB_FILE}) #${CMAKE_STATIC_LIBRARY_SUFFIX}
   endif()
   set_target_properties(omp PROPERTIES

Aug 31 '23 12:08 DeadSix27

May I suggest that we/you add a switch/code for handling the needed arch's, in my case for example I have no use for aarch/etc all I need is x86-64. Having to manually remove the other archs from the loop is not pretty. If needed I could work on it and add a PR.

There is already a way of controlling which architectures you want included; set ARCHS="x86_64" when building and it will only add wrapper frontends for that arch and only build runtimes for that arch.

As for the workarounds, we could either ask upstream for the fixes (e.g for omp) or manually patch the llvm clone within this repo.

No, this is a rather firm policy of this project. We solve things upstream (for the benefit for other users as well). If upstream doesn't agree with a direction, we should work with them about figuring out a reasonable way forward. But we don't carry local patches that won't be upstreamed.

Aug 31 '23 21:08 mstorsjo

FYI, the fixes to make compiler-rt compileable when libunwind/libc++ are only available in static form, should have landed in https://github.com/llvm/llvm-project/commit/7c5e4e5fa3a948fc662be3a6bf057021d32f72e6 now.

For building static openmp, larger changes seem to be needed, to get the library names right throughout (and for testing etc).

Oct 05 '23 10:10 mstorsjo

That's awesome, I'll test it later, this means sanitizer builds will build statically now?

Oct 05 '23 10:10 DeadSix27

That's awesome, I'll test it later, this means sanitizer builds will build statically now?

No, the sanitizers themselves will still require being built as DLL as they were.

The only thing that has changed is that the build doesn't fail, if libunwind/libc++ is only available in static form.

Oct 05 '23 10:10 mstorsjo