shared link llvm (this is a discussion not a issue)

[SquallATF]

Current trunk version lld not support -Wl,--version-script on mingw target but support -Wl,--export-all-symbols, after modify llvm-shlib's (CmakeLists.txt), I can build shared version LLVM.dll. Current build linked static llvm lib, will cost a lot of space(I tried build latest trunk and full install without strip the bin directory cost 1733 MB). then I tried build with LLVM_LINK_LLVM_DYLIB=ON, have 4 file link failed(failed.log), and can fix by modify CMakeLists.txt, after fix and install the result bin directory only 503 MB. I tried use shared link version toolchain build some example code, can generate executables and run normally, but the ld.lld hang on exit. but a strange phenomenon is that when I use the Sysinternals process explorer to view ld.lld threads, the ld.lld will automatically quit. Has anyone tried shared link llvm?

[mstorsjo]

Current trunk version lld not support -Wl,--version-script on mingw target

Correct. So far, I have ran into very few projects that use this, and one that did, had a build system that checked whether the option worked before using it.

but support -Wl,--export-all-symbols, after modify llvm-shlib's (CmakeLists.txt)

FWIW, it doesn't look like this version script is very useful for COFF targets at all. Version scripts can be used for two things; for setting symbol versions (which isn't supported for COFF), and for limiting what symbols are exported when exporting everything. In this case, it still exports everything, but just using a symbol version, so for COFF this is pointless, it seems.

, I can build shared version LLVM.dll.

Current build linked static llvm lib, will cost a lot of space(I tried build latest trunk and full install without strip the bin directory cost 1733 MB). then I tried build with LLVM_LINK_LLVM_DYLIB=ON, have 4 file link failed(failed.log), and can fix by modify CMakeLists.txt, after fix and install the result bin directory only 503 MB. I tried use shared link version toolchain build some example code, can generate executables and run normally, but the ld.lld hang on exit.

Does the ld.lld built statically work correctly, or does that hang on exit? There was an issue in August where ld.lld built with this toolchain didn't exit correctly - after taking __cxa_thread_atexit into use. This was fixed in 6edb1a60e95e8308b6ae2ac3a9f863c056a2f7ee by updating mingw-w64.

but a strange phenomenon is that when I use the Sysinternals process explorer to view ld.lld threads, the ld.lld will automatically quit. Has anyone tried shared link llvm?

Sorry, no, I haven't. But I have made some effort into providing a windows version of this toolchain lately, I pushed most of that work now, and you can have a look at a test version of it at https://martin.st/temp/llvm-mingw-x86_64.zip.

[SquallATF]

Does the ld.lld built statically work correctly, or does that hang on exit? There was an issue in August where ld.lld built with this toolchain didn't exit correctly - after taking __cxa_thread_atexit into use. This was fixed in 6edb1a6 by updating mingw-w64.

I used https://martin.st/temp/llvm-mingw-x86_64.zip build trunk 343560 version llvm with 1b29d1b version mingw-w64 first, and use output toolchain build shared version. I also tried to compile directly with the version you provided, have same issue. here is the ld.lld.exe frame stack on hang

#0  0x00007ffa9f3bd7e4 in ntdll!ZwWaitForAlertByThreadId ()
   from C:\WINDOWS\SYSTEM32\ntdll.dll
#1  0x00007ffa9f33ef19 in ntdll!RtlSleepConditionVariableSRW ()
   from C:\WINDOWS\SYSTEM32\ntdll.dll
#2  0x00007ffa9bd02c72 in SleepConditionVariableSRW ()
   from C:\WINDOWS\System32\KernelBase.dll
#3  0x0000000180026b53 in std::__1::__libcpp_condvar_wait(void**, void**) ()
   from F:\llvm8\bin\libc++.dll
#4  0x0000000180006472 in std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) () from F:\llvm8\bin\libc++.dll
#5  0x00000000046d771b in (anonymous namespace)::ThreadPoolExecutor::~ThreadPoolExecutor() () from F:\llvm8\bin\LLVM.dll
#6  0x00007ffa9bb31243 in ucrtbase!_execute_onexit_table ()
   from C:\WINDOWS\System32\ucrtbase.dll
#7  0x00007ffa9bb31017 in ucrtbase!_execute_onexit_table ()
   from C:\WINDOWS\System32\ucrtbase.dll
#8  0x00007ffa9bb30fd4 in ucrtbase!_execute_onexit_table ()
   from C:\WINDOWS\System32\ucrtbase.dll
#9  0x000000000462109a in _CRT_INIT (
    hDllHandle=0x70ad050 <__native_dllmain_reason>, dwReason=<optimized out>,
    lpreserved=0x1) at ../crt/crtdll.c:140
#10 0x0000000004621341 in __DllMainCRTStartup (hDllHandle=0x4620000,
    dwReason=0, lpreserved=0x1) at ../crt/crtdll.c:204
#11 0x00007ffa9f354053 in ntdll!RtlActivateActivationContextUnsafeFast ()
   from C:\WINDOWS\SYSTEM32\ntdll.dll
#12 0x00007ffa9f360a05 in ntdll!LdrShutdownProcess ()
   from C:\WINDOWS\SYSTEM32\ntdll.dll
#13 0x00007ffa9f3608c8 in ntdll!RtlExitUserProcess ()
   from C:\WINDOWS\SYSTEM32\ntdll.dll
#14 0x00007ffa9edab19a in KERNEL32!FatalExit ()
   from C:\WINDOWS\System32\kernel32.dll
#15 0x00007ffa9bb33828 in ucrtbase!exit ()
   from C:\WINDOWS\System32\ucrtbase.dll
#16 0x00007ffa9bb337db in ucrtbase!exit ()
   from C:\WINDOWS\System32\ucrtbase.dll
#17 0x000000014002d824 in lld::exitLld(int) ()
#18 0x000000014000202f in lld::coff::link(llvm::ArrayRef<char const*>, bool, llvm::raw_ostream&) ()
#19 0x0000000140027b7e in lld::mingw::link(llvm::ArrayRef<char const*>, llvm::raw_ostream&) ()
#20 0x0000000140001ace in main ()

I try to use Process Explorer let ld.lld.exe process continue run, and get exit code from gdb

(gdb) cont
Continuing.
[Thread 4924.0x3244 exited with code 0]
[Inferior 1 (process 4924) exited normally]
(gdb) print $_exitcode
$2 = 0

it looks like exit normally

[mstorsjo]

So I presume that my prebuilt ld.lld.exe exits correctly, while the one you built with shared libraries hangs?

I think I understand why this happens.

LLD exits in a pretty special way. Have a look at https://github.com/llvm-mirror/lld/blob/master/Common/ErrorHandler.cpp#L49.

It does some minimal cleanup first, then it calls _exit(), which should exit the process immediately without running the destructors. Due to the minimal cleanup it did before, the destructors, if they were to be executed, would actually hang on windows.

If you exit a process with either _exit() or ExitProcess(), it doesn't run the destructors of the exe itself, but it still does run all destructors for all loaded DLLs before stopping. (The same also applies for MSVC, so it's not a mingw specific issue.) In your case, some destructors are in LLVM.dll, and they try to run (see ~ThreadPoolExecutor), and they fail since we already have done some cleanup. It's strange how the backtrace shows ucrtbase!exit, when the called function should be _exit.

So in any case, I would consider this a conflict/bug between LLD's cleanup mechanism and DLL destructor handling on windows. If you really want to run a LLD which is linked shared, try modifying the exitLld function, removing the llvm_shutdown and maybe the flush calls, and have it call exit instead of _exit. Then I think it might work...

A different solution could be to set the environment variable LLD_IN_TEST=1 before running it. Then it should try to do a normal exit. But this doesn't help if lld chooses to exit early due to an error in the link.

[SquallATF]

I found the hang reason and create a patch D53968

[mati865]

FYI https://reviews.llvm.org/D70447 has landed in the master.

[mati865]

MSYS2 has been packaging dynamically linked LLVM but this is no longer the case since it was causing LLD to hang: https://github.com/msys2/MINGW-packages/issues/6126 Probably libstdc++/winpthreads bug.

[adrpo]

I will give it another try with dynamically linked LLVM but statically linked lld. The only "problem" is basically time (CPU resources). In PKGBUILD we can do a shared build first for all the tools then a static build for lld only via -DLLVM_ENABLE_PROJECTS="lld" then put them together. Would that be OK?

[mati865]

Hmm, we should try using libc++ instead of libstdc++. It doesn't use winpthreads so one less possible point of failure.

[mstorsjo]

On Thu, 9 Jul 2020, Mateusz Mikuła wrote:

Now that MSYS2 is getting working libc++ I tried to link LLVM in shared mode but it fails with both LLD and LD:

[ 74%] Linking CXX executable ../../bin/llvm-cfi-verify.exe clang++: warning: argument unused during compilation: '-Wa,-mbig-obj' [-Wunu sed-command-line-argument] D:\msys64\mingw64\bin\ld: ../../lib/libLLVMSymbolize.a(Symbolize.cpp.obj):Sy mbolize.cpp:(.text+0x1200): multiple definition of `llvm::symbolize::LLVMSym bolizer::flush()'; ../../lib/libLLVM.dll.a(d037264.o):(.text+0x0): first def ined here

Attempting to reply via mail, as this reply doesn't seem to show up on the web version at all...

This looks like an issue I've seen as well, if trying to build with -DLLVM_BUILD_LLVM_DYLIB=TRUE -DLLVM_LINK_LLVM_DYLIB=TRUE; that configuration doesn't seem to really work properly here, as it doesn't inhibit linking to the individual libraries in addition to the combined LLVM.dll.

Building with -DBUILD_SHARED_LIBS=TRUE should work though (at least in current master, there might be a couple build errors in the 10 release), but that results in a separate DLL for each of the invidiual LLVM internal libraries.

// Martin

[mati865]

I've backported the patches that fix the build it and I'm testing the toolchain (with -DLLVM_LINK_LLVM_DYLIB=TRUE) with libc++ instead of libstdc++. Expectedly I haven't encountered winpthreads bug that was causing LLD to hang (https://github.com/msys2/MINGW-packages/issues/6126).

[mati865]

We had to revert libc++ change because of ABI incompatibility: https://github.com/msys2/MINGW-packages/issues/7170

There is new patch https://github.com/msys2/MINGW-packages/pull/7226 that makes it work with libstdc++.

That is one thing that one should be aware of when linking dynamically.

---

@mstorsjo in case you want to try dynamic linking again with LLVM 11 you will have to backport https://github.com/llvm/llvm-project/blob/04ea680a8ccc4f9a4d7333cd712333960348c35b/clang/examples/Attribute/CMakeLists.txt