mingw-w64-x86_64-zig: `zig` fails to start, `libLLVM.dll` is missing

[sskras]

I updated MSYS2 an hour ago, and installed zig package:

$ pacman -S mingw64/mingw-w64-x86_64-zig
   ...
$ pacman -Q mingw-w64-x86_64-zig
mingw-w64-x86_64-zig 0.7.1-2

$ zig
bash: zig: command not found

$ PATH=$PATH:/mingw64/bin

$ zig
C:/msys64/mingw64/bin/zig.exe: error while loading shared libraries: libwinpthread-1.dll: cannot open shared object file: No such file or directory

[sskras]

Investigation steps:

$ ldd $(which zig)
        ntdll.dll => /c/Windows/SYSTEM32/ntdll.dll (0x7ffcc4bf0000)
        KERNEL32.DLL => /c/Windows/System32/KERNEL32.DLL (0x7ffcc36e0000)
        KERNELBASE.dll => /c/Windows/System32/KERNELBASE.dll (0x7ffcc2770000)
        ADVAPI32.dll => /c/Windows/System32/ADVAPI32.dll (0x7ffcc48b0000)
        msvcrt.dll => /c/Windows/System32/msvcrt.dll (0x7ffcc2d70000)
        libgcc_s_seh-1.dll => /mingw64/bin/libgcc_s_seh-1.dll (0x7ffca17b0000)
        libstdc++-6.dll => /mingw64/bin/libstdc++-6.dll (0x7ffc80fd0000)
        sechost.dll => /c/Windows/System32/sechost.dll (0x7ffcc4960000)
        RPCRT4.dll => /c/Windows/System32/RPCRT4.dll (0x7ffcc3400000)
        libwinpthread-1.dll => /mingw64/bin/libwinpthread-1.dll (0x7ffca1790000)
        ole32.dll => /c/Windows/System32/ole32.dll (0x7ffcc3860000)
        libclang-cpp.dll => /mingw64/bin/libclang-cpp.dll (0x7ffc48e10000)
        ucrtbase.dll => /c/Windows/System32/ucrtbase.dll (0x7ffcc2b30000)
        combase.dll => /c/Windows/System32/combase.dll (0x7ffcc2fc0000)
        GDI32.dll => /c/Windows/System32/GDI32.dll (0x7ffcc47a0000)
        win32u.dll => /c/Windows/System32/win32u.dll (0x7ffcc2520000)
        gdi32full.dll => /c/Windows/System32/gdi32full.dll (0x7ffcc2550000)
        msvcp_win.dll => /c/Windows/System32/msvcp_win.dll (0x7ffcc2a40000)
        USER32.dll => /c/Windows/System32/USER32.dll (0x7ffcc3530000)
        SHELL32.dll => /c/Windows/System32/SHELL32.dll (0x7ffcc3fd0000)
        libLLVM.dll => not found

The similarly named DLL seems to be around:

$ find / -iname libLLVM*dll
/mingw64/bin/libLLVM-13.dll
find: failed to read file names from file system at or below ‘/’: No such file or directory

Symlinking the missing name to one I found results in exception and the message box during zig run from Explorer:

---------------------------
zig.exe - Entry Point Not Found
---------------------------
The procedure entry point _ZN5clang16CompilerInstance16createOutputFileEN4llvm9StringRefEbbS2_S2_bb
 could not be located in the dynamic link library C:\msys64\mingw64\bin\zig.exe. 
---------------------------
OK   
---------------------------

[jeremyd2019]

#9791 - zig was not rebuilt against LLVM 13, because it wasn't possible to do so.

[sskras]

Yes, but now the stable version is 0.9.0. Which seems to use LLVM 13 already:

This release of Zig upgrades to LLVM 13.

Good news: we noticed something like a 12% improvement in compilation speed when we upgraded Zig from LLVM 12 to 13.

I guess the PKGBUILD just needs an update. Ping @kassane.

[kassane]

Yes, but now the stable version is 0.9.0. Which seems to use LLVM 13 already:

This release of Zig upgrades to LLVM 13. Good news: we noticed something like a 12% improvement in compilation speed when we upgraded Zig from LLVM 12 to 13.

I guess the PKGBUILD just needs an update. Ping @kassane.

Hi @sskras

So far I stopped updating the mingw package waiting to fix zig0 and its high memory consumption during build causing the error process to stop in some cases. And also because there was a bug during the binary linking generating conflict between mingw/msvc.

[ 98%] Linking CXX executable zig0.exe
D:/M/msys64/mingw64/lib\liblldELF.a(InputFiles.cpp.obj): duplicate section `.rdata$_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits[_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits]' has different size
D:/M/msys64/mingw64/lib\liblldELF.a(InputSection.cpp.obj): duplicate section `.rdata$_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits[_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits]' has different size
D:/M/msys64/mingw64/lib\liblldCOFF.a(InputFiles.cpp.obj): duplicate section `.rdata$_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits[_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits]' has different size
D:/M/msys64/mingw64/lib\liblldWasm.a(Writer.cpp.obj): duplicate section `.rdata$_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits[_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits]' has different size
D:/M/msys64/mingw64/lib\liblldWasm.a(WriterUtils.cpp.obj): duplicate section `.rdata$_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits[_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits]' has different size
[ 98%] Built target zig0
[ 99%] Building self-hosted component C:/_/mingw-w64-zig/src/zig-0.9.0/build-x86_64-w64-mingw32/zig1.o
allocation failed
make[2]: *** [CMakeFiles/zig.dir/build.make:347: zig1.o] Error 3

[sskras]

Thanks for the pointer. I guess one way is just to sit and wait, as

@andrewrk commented on Oct 4, 2020 in ziglang/zig/issues/6485:

The current plan is to [...] (2) focus the main fforts of the next release cycle on finishing the self-hosted compiler, which has much better memory utilization. At this point the plan is #6378.

But I guess I will also try rebuilding the PKGBUILD file and see myself. (Some time after my exams:)

[sskras]

OK, I did it just now, same errors and then some more:

$ cd src/MINGW-packages/mingw-w64-zig
$ makepkg-mingw -sCLf
  ...
[ 95%] Building CXX object CMakeFiles/zigstage1.dir/src/stage1/stage1.cpp.obj
[ 95%] Building CXX object CMakeFiles/zigstage1.dir/src/stage1/tokenizer.cpp.obj
[ 96%] Building CXX object CMakeFiles/zigstage1.dir/src/stage1/util.cpp.obj
[ 97%] Linking CXX static library zigcpp/libzigstage1.a
[ 97%] Built target zigstage1
[ 98%] Building CXX object CMakeFiles/zig0.dir/src/stage1/zig0.cpp.obj
[ 98%] Linking CXX executable zig0.exe
C:/msys64/mingw64/lib\liblldELF.a(InputFiles.cpp.obj): duplicate section `.rdata$_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits[_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits]' has different size
C:/msys64/mingw64/lib\liblldELF.a(InputSection.cpp.obj): duplicate section `.rdata$_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits[_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits]' has different size
C:/msys64/mingw64/lib\liblldCOFF.a(InputFiles.cpp.obj): duplicate section `.rdata$_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits[_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits]' has different size
C:/msys64/mingw64/lib\liblldWasm.a(Writer.cpp.obj): duplicate section `.rdata$_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits[_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits]' has different size
C:/msys64/mingw64/lib\liblldWasm.a(WriterUtils.cpp.obj): duplicate section `.rdata$_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits[_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits]' has different size
[ 98%] Built target zig0
[ 99%] Building self-hosted component C:/msys64/home/saukrs/src/MINGW-packages/mingw-w64-zig/src/zig-0.9.0/build-x86_64-w64-mingw32/zig1.o
[ 99%] Building CXX object CMakeFiles/zig.dir/src/stage1/empty.cpp.obj
[100%] Linking CXX executable zig.exe
C:/msys64/mingw64/lib\liblldELF.a(Driver.cpp.obj): duplicate section `.rdata$_ZTVN4llvm3opt7ArgListE[_ZTVN4llvm3opt7ArgListE]' has different size
C:/msys64/mingw64/lib\liblldELF.a(DriverUtils.cpp.obj): duplicate section `.rdata$_ZTVN4llvm3opt7ArgListE[_ZTVN4llvm3opt7ArgListE]' has different size
C:/msys64/mingw64/lib\liblldELF.a(InputFiles.cpp.obj): duplicate section `.rdata$_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits[_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits]' has different size
C:/msys64/mingw64/lib\liblldELF.a(InputSection.cpp.obj): duplicate section `.rdata$_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits[_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits]' has different size
C:/msys64/mingw64/lib\liblldCOFF.a(Driver.cpp.obj): duplicate section `.rdata$_ZTVN4llvm3opt7ArgListE[_ZTVN4llvm3opt7ArgListE]' has different size
C:/msys64/mingw64/lib\liblldCOFF.a(DriverUtils.cpp.obj): duplicate section `.rdata$_ZTVN4llvm3opt7ArgListE[_ZTVN4llvm3opt7ArgListE]' has different size
C:/msys64/mingw64/lib\liblldCOFF.a(InputFiles.cpp.obj): duplicate section `.rdata$_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits[_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits]' has different size
C:/msys64/mingw64/lib\liblldWasm.a(Driver.cpp.obj): duplicate section `.rdata$_ZTVN4llvm3opt7ArgListE[_ZTVN4llvm3opt7ArgListE]' has different size
C:/msys64/mingw64/lib\liblldWasm.a(Writer.cpp.obj): duplicate section `.rdata$_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits[_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits]' has different size
C:/msys64/mingw64/lib\liblldWasm.a(WriterUtils.cpp.obj): duplicate section `.rdata$_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits[_ZZNSt8__detail18__to_chars_10_implIjEEvPcjT_E8__digits]' has different size
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x849b1): undefined reference to `LLVMInitializePowerPCTarget'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x849b6): undefined reference to `LLVMInitializePowerPCTargetInfo'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x849bb): undefined reference to `LLVMInitializePowerPCTargetMC'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x849c0): undefined reference to `LLVMInitializePowerPCAsmPrinter'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x849c5): undefined reference to `LLVMInitializePowerPCAsmParser'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84a80): undefined reference to `LLVMInitializeSparcTarget'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84a85): undefined reference to `LLVMInitializeSparcTargetInfo'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84a8a): undefined reference to `LLVMInitializeSparcTargetMC'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84a8f): undefined reference to `LLVMInitializeSparcAsmPrinter'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84a94): undefined reference to `LLVMInitializeSparcAsmParser'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84b9f): undefined reference to `LLVMInitializeHexagonTarget'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84ba4): undefined reference to `LLVMInitializeHexagonTargetInfo'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84ba9): undefined reference to `LLVMInitializeHexagonTargetMC'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84bae): undefined reference to `LLVMInitializeHexagonAsmPrinter'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84bb3): undefined reference to `LLVMInitializeHexagonAsmParser'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84bf9): undefined reference to `LLVMInitializeSystemZTarget'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84bfe): undefined reference to `LLVMInitializeSystemZTargetInfo'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84c03): undefined reference to `LLVMInitializeSystemZTargetMC'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84c08): undefined reference to `LLVMInitializeSystemZAsmPrinter'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84c0d): undefined reference to `LLVMInitializeSystemZAsmParser'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84c17): undefined reference to `LLVMInitializeXCoreTarget'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84c1c): undefined reference to `LLVMInitializeXCoreTargetInfo'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84c21): undefined reference to `LLVMInitializeXCoreTargetMC'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84c26): undefined reference to `LLVMInitializeXCoreAsmPrinter'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84c30): undefined reference to `LLVMInitializeLanaiTarget'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84c35): undefined reference to `LLVMInitializeLanaiTargetInfo'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84c3a): undefined reference to `LLVMInitializeLanaiTargetMC'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84c3f): undefined reference to `LLVMInitializeLanaiAsmPrinter'
C:/msys64/mingw64/bin/ld: CMakeFiles/zig.dir/objects.a(zig1.o):zig1:(.text+0x84c44): undefined reference to `LLVMInitializeLanaiAsmParser'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [CMakeFiles/zig.dir/build.make:392: zig.exe] Error 1
make[1]: *** [CMakeFiles/Makefile2:232: CMakeFiles/zig.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
==> ERROR: A failure occurred in build().
    Aborting...

@kassane, what do you think:

is this bug in Zig itself, in the used gcc suite (linker or compiler) or maybe in the upstream LLVM code?

[kassane]

@sskras This case the bug is only in zig because it has no link to libgcc or compiler-rt or any mingw-specific lib. Perhaps the only viable alternative would be to try compiling with the official zig build itself instead of mingw's clang/gcc. Because the old-version available in msys2/mingw is broken to compile under windows-gnu.

[jeremyd2019]

The new undefined references are due to us needing to disable some targets in LLVM 13 to avoid hitting an intrinsic limit of 65535 exported symbols from a DLL. Zig does not support building against an LLVM that was not built with all targets enabled. I tried hacking out targets in msys2/MINGW-packages-dev#16 but still hit the out-of-memory or __emutls_get_address errors

[sskras]

Thank you both.

@jeremyd2019, as you suggested in #8272, should we try adding mingw-w64-x86_64-compiler-rt as build dependency?

Not sure what else would be needed, like adding -lcompiler-rt flag somewhere.

[kassane]

Thank you both.

@jeremyd2019, as you suggested in #8272, should we try adding mingw-w64-x86_64-compiler-rt as build dependency?

Not sure what else would be needed, like adding -lcompiler-rt flag somewhere.

By the way, in https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html Andrew says:

Zig's compiler-rt is not yet complete. However, completing it is a prerequisite for releasing Zig version 1.0.0.

[sskras]

@kassane: thanks for the pointers. Looks like their progress with missing compiler-rt functions is blocked by:

• deciding between the the amount of hand-rolled asm & the speed of functions.
• Zig signal handling port missing a strategy.
• missing benchmarking against realistic workloads to prevent regressions.
• missing efforts to debug using posix_spawn instead of fork.

A lot of stuff here!

[sskras]

@jeremyd2019 commented on Jan 22:

The new undefined references are due to us needing to disable some targets in LLVM 13 to avoid hitting an intrinsic limit of 65535 exported symbols from a DLL.

Wow, that's a big discovery to me! The search query for future reference: https://www.qwant.com/?q=DLL+limit+of+65535+exported+symbols

Zig does not support building against an LLVM that was not built with all targets enabled.

Without knowing what symbol names (related to any target) make it into the export table, this sounds fair enough.

But after reading pages from the search above, my guess is that maybe the export table is being filled with all possible symbols from particular lib, not just the necessary ones. Eg: https://cygwin.com/pipermail/cygwin/2016-May/227888.html

This leaves me interested in whether --export-all-symbols is used during this Zig build.

I tried hacking out targets in msys2/MINGW-packages-dev#16 but still hit the out-of-memory or __emutls_get_address errors

Thanks. Should we transfer further investigation and discussion here?

[lazka]

#12550 might help here

[lazka]

I guess the next step would be make a clang13/llvm13 or 14 package like in Arch: https://archlinux.org/packages/community/x86_64/zig/

We would need something similar for python-llvmlite too

[sskras]

@lazka commented 20 days ago:

I guess the next step would be make a clang13/llvm13 or 14 package like in Arch: https://archlinux.org/packages/community/x86_64/zig/

I guess so. Tried rebuilding Zig-0.9.1, it fails for LLVM being too new:

$ time makepkg-mingw -sCLf
==>  MINGW_ARCH: mingw64
  -> Building mingw64...
==> Making package: mingw-w64-zig 0.9.1-1 (08 Sep 2022 15:20:42)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
  -> Found zig-0.9.1.tar.gz
==> Validating source files with sha512sums...
    zig-0.9.1.tar.gz ... Passed
==> Removing existing $srcdir/ directory...
==> Extracting sources...
  -> Extracting zig-0.9.1.tar.gz with bsdtar
==> Starting build()...
-- The C compiler identification is Clang 14.0.6
-- The CXX compiler identification is Clang 14.0.6
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/msys64/mingw64/bin/clang.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/msys64/mingw64/bin/clang++.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Configuring zig version 0.9.1
CMake Error at cmake/Findllvm.cmake:68 (message):
  expected LLVM 13.x but found 14.0.6 using
  C:/msys64/mingw64/bin/llvm-config.exe
Call Stack (most recent call first):
  CMakeLists.txt:107 (find_package)


-- Configuring incomplete, errors occurred!
See also "C:/msys64/home/saukrs/src/MINGW-packages/mingw-w64-zig/src/zig-0.9.1/build-x86_64-w64-mingw32/CMakeFiles/CMakeOutput.log".
==> ERROR: A failure occurred in build().
    Aborting...

real    1m20.781s
user    0m12.388s
sys     0m45.844s

[mati865]

FWIW upcoming Zig version will support LLVM 14 but we are moving to LLVM 15 already... So yeah, we need special version for it.

[andrewrk]

master branch of Zig depends on LLVM 14, however I am actively working to merge the llvm15 branch of Zig as soon as upgrade issues are solved. This will be done within a few days.

Upcoming Zig version (0.10.0) has a release date of 2022-11-01 and will link against LLVM 15.

[mati865]

Cool, feel free to ping me if you have problems with mingw-w64 targets.