building from source no longer links llvm correctly

[nektro]

Zig Version

0.11.0-dev.3653+1253d591b

Steps to Reproduce and Observed Behavior

[meghan@nixos:~]$ uname -a
Linux nixos 5.15.102 #1-NixOS SMP Mon Mar 13 09:20:37 UTC 2023 x86_64 GNU/Linux

git clone https://github.com/ziglang/zig
cd zig
nix-shell -p cmake gnumake zlib
ccd build
cmake .. -DCMAKE_PREFIX_PATH=/home/meghan/src/llvm-16/out -DZIG_NO_LIB=ON -DZIG_STATIC_LLVM=OFF
make install

[nix-shell:~/src/zig2/build]$ ldd ./stage3/bin/zig 
	linux-vdso.so.1 (0x00007ffc357fb000)
	libclang-cpp.so.16 => not found
	libstdc++.so.6 => not found
	libLLVM-16.so => not found
	libz.so.1 => /nix/store/4xqki8nyafl26w2p71w988djvfmfz4x1-zlib-1.2.13/lib/libz.so.1 (0x00007fbfaa2a6000)
	libc.so.6 => /nix/store/2j8jqmnd9l7plihhf713yf291c9vyqjm-glibc-2.35-224/lib/libc.so.6 (0x00007fbfaa09d000)
	/nix/store/2j8jqmnd9l7plihhf713yf291c9vyqjm-glibc-2.35-224/lib/ld-linux-x86-64.so.2 => /nix/store/2j8jqmnd9l7plihhf713yf291c9vyqjm-glibc-2.35-224/lib64/ld-linux-x86-64.so.2 (0x00007fbfaa2c6000)

/home/meghan/src/zig/build/stage3/bin/zig: error while loading shared libraries: libclang-cpp.so.16: cannot open shared object file: No such file or directory

https://github.com/ziglang/zig/commit/529ef75101e21bd45402c516138343b4770238eb was good

https://github.com/ziglang/zig/commit/1253d591be2aeea49c0b67ce0416237060e78a57 is bad

https://github.com/ziglang/zig/commit/d7b73af8f65bb891c8700ed47777144bb6f35fe1 looks sus since it calls out nixos, linux CI is passing so its not a universal issue

git bisect leads to this commit as well

Expected Behavior

[nix-shell:~/src/zig2/build]$ ldd ./stage3/bin/zig 
	linux-vdso.so.1 (0x00007ffd2bff4000)
	libclang-cpp.so.16 => /home/meghan/src/llvm-16/out/lib/libclang-cpp.so.16 (0x00007f4667cf0000)
	libstdc++.so.6 => /nix/store/v8481v3a686aa0pa4d016c0lnz2j0wwb-gcc-11.3.0-lib/lib/libstdc++.so.6 (0x00007f4667ada000)
	libLLVM-16.so => /home/meghan/src/llvm-16/out/lib/libLLVM-16.so (0x00007f465e84a000)
	libz.so.1 => /nix/store/4xqki8nyafl26w2p71w988djvfmfz4x1-zlib-1.2.13/lib/libz.so.1 (0x00007f465e82c000)
	libc.so.6 => /nix/store/2j8jqmnd9l7plihhf713yf291c9vyqjm-glibc-2.35-224/lib/libc.so.6 (0x00007f465e621000)
	/nix/store/2j8jqmnd9l7plihhf713yf291c9vyqjm-glibc-2.35-224/lib/ld-linux-x86-64.so.2 => /nix/store/2j8jqmnd9l7plihhf713yf291c9vyqjm-glibc-2.35-224/lib64/ld-linux-x86-64.so.2 (0x00007f466c485000)
	libm.so.6 => /nix/store/2j8jqmnd9l7plihhf713yf291c9vyqjm-glibc-2.35-224/lib/libm.so.6 (0x00007f465e541000)
	libgcc_s.so.1 => /nix/store/2j8jqmnd9l7plihhf713yf291c9vyqjm-glibc-2.35-224/lib/libgcc_s.so.1 (0x00007f465e527000)
	librt.so.1 => /nix/store/2j8jqmnd9l7plihhf713yf291c9vyqjm-glibc-2.35-224/lib/librt.so.1 (0x00007f465e522000)
	libdl.so.2 => /nix/store/2j8jqmnd9l7plihhf713yf291c9vyqjm-glibc-2.35-224/lib/libdl.so.2 (0x00007f465e51d000)

[nektro]

cc @Jan200101

[Jan200101]

What rpaths does the binary have?

[nektro]

[nix-shell:~]$ readelf -d ./src/zig/build/stage3/bin/zig

Dynamic section at offset 0x29f86c0 contains 31 entries:
  Tag        Type                         Name/Value
 0x000000000000001d (RUNPATH)            Library runpath: [
    /nix/store/as3hc6s4ljma9kibvpq9cqbmm33pd6zy-nix-shell/lib64:
    /nix/store/as3hc6s4ljma9kibvpq9cqbmm33pd6zy-nix-shell/lib:
    /home/meghan/src/llvm-16/out/lib:
    /nix/store/4xqki8nyafl26w2p71w988djvfmfz4x1-zlib-1.2.13/lib:
    /nix/store/v8481v3a686aa0pa4d016c0lnz2j0wwb-gcc-11.3.0-lib/lib
]

[nix-shell:~]$ readelf -d ./src/zig2/build/stage3/bin/zig

Dynamic section at offset 0x297aa80 contains 31 entries:
  Tag        Type                         Name/Value
 0x000000000000001d (RUNPATH)            Library runpath: [
    /nix/store/dxqqhs8z247vzgyk7h12qq1ayp4dkmw5-shell/lib64:
    /nix/store/dxqqhs8z247vzgyk7h12qq1ayp4dkmw5-shell/lib:
    /nix/store/4xqki8nyafl26w2p71w988djvfmfz4x1-zlib-1.2.13/lib
]

[Jan200101]

Okay, so this isn't an issue about dealing with the Nix constructs (e.g. NIXLDFLAGS) but with disabling each_lib_rpath on native systems.

I don't know a good way forward, perhaps storing all system-wide lib path in a map to look at when adding rpaths?

Every solution thus far has managed to regress something because there are so many different ways people run their system.

[Jan200101]

should be fixed by #16062

alternative solutions are also welcome, getting rid of obstructive rpaths has now caused problems on a specific setup for the second time.

[nektro]

building today only libstdc++ fails

[meghan@nixos:~]$ ldd src/zig2/build/stage3/bin/zig
       linux-vdso.so.1 (0x00007ffce838d000)
       libclang-cpp.so.16 => /nix/store/p0gn97dky1qs972c7xbmn33p44h81na8-clang-16.0.1-lib/lib/libclang-cpp.so.16 (0x00007fb3c1600000)
       libstdc++.so.6 => not found
       libLLVM-16.so => /nix/store/33zbbqa7n6qfbliy77d4c53fcvd2bym9-llvm-16.0.1-lib/lib/libLLVM-16.so (0x00007fb3b8c00000)
       libz.so.1 => /nix/store/fxq6yn4mfnsnh418l2k218j7sp365sa9-zlib-1.2.13/lib/libz.so.1 (0x00007fb3c5668000)
       libc.so.6 => /nix/store/dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8/lib/libc.so.6 (0x00007fb3b8a1a000)
       /nix/store/dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8/lib/ld-linux-x86-64.so.2 => /nix/store/dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8/lib64/ld-linux-x86-64.so.2 (0x00007fb3c5689000)
       libstdc++.so.6 => /nix/store/sm14bmd3l61p5m0q7wa5g7rz2bl6azqf-gcc-12.2.0-lib/lib/libstdc++.so.6 (0x00007fb3b8600000)
       libm.so.6 => /nix/store/dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8/lib/libm.so.6 (0x00007fb3c5586000)
       libgcc_s.so.1 => /nix/store/sm14bmd3l61p5m0q7wa5g7rz2bl6azqf-gcc-12.2.0-lib/lib/libgcc_s.so.1 (0x00007fb3c5565000)
       libffi.so.8 => /nix/store/pq9vbv9q0q1vsv6yamqyrfv4qkzz0487-libffi-3.4.4/lib/libffi.so.8 (0x00007fb3c5556000)
       librt.so.1 => /nix/store/dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8/lib/librt.so.1 (0x00007fb3c5551000)
       libdl.so.2 => /nix/store/dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8/lib/libdl.so.2 (0x00007fb3c554c000)
       libncursesw.so.6 => /nix/store/4yx5cv12dyswnaiyfbafymmhl3h26dgi-ncurses-6.4/lib/libncursesw.so.6 (0x00007fb3c158c000)
       libxml2.so.2 => /nix/store/hyn81nswiyrkgd2365zjgyml05zf58vf-libxml2-2.10.4/lib/libxml2.so.2 (0x00007fb3b88b4000)

[Jan200101]

libstdc++ appears to be a special case since that appears to be handled by the build.zig

[nektro]

oh another thing is that it is in there twice, and one of them resolved to my system

       libstdc++.so.6 => not found
       libstdc++.so.6 => /nix/store/sm14bmd3l61p5m0q7wa5g7rz2bl6azqf-gcc-12.2.0-lib/lib/libstdc++.so.6 (0x00007fb3b8600000)

[Jan200101]

Its possible that one of those is a direct dependency and one is an indirect dependency of somethinge else.

[nektro]

hmm yeah you're probably right, the new list is much longer than the one in my original description. it should only be the direct dependencies

[andrewrk]

Regression reverted in ae2cd5fe263e9d8ddd5719b4795f9cd39b5c0324.

[nektro]

can confirm this is working again

[meghan@nixos:~/src/zig]$ ldd build/stage3/bin/zig 
	linux-vdso.so.1 (0x00007ffcc9939000)
	libclang-cpp.so.17 => /home/meghan/src/llvm-17/out/lib/libclang-cpp.so.17 (0x00007fc45da00000)
	libstdc++.so.6 => /nix/store/sm14bmd3l61p5m0q7wa5g7rz2bl6azqf-gcc-12.2.0-lib/lib/libstdc++.so.6 (0x00007fc45d600000)
	libLLVM-17.so => /home/meghan/src/llvm-17/out/lib/libLLVM-17.so (0x00007fc454600000)
	libz.so.1 => /nix/store/fxq6yn4mfnsnh418l2k218j7sp365sa9-zlib-1.2.13/lib/libz.so.1 (0x00007fc462021000)
	libc.so.6 => /nix/store/dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8/lib/libc.so.6 (0x00007fc45441a000)
	/nix/store/dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8/lib/ld-linux-x86-64.so.2 => /nix/store/dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8/lib64/ld-linux-x86-64.so.2 (0x00007fc462042000)
	libm.so.6 => /nix/store/dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8/lib/libm.so.6 (0x00007fc461f3f000)
	libgcc_s.so.1 => /nix/store/sm14bmd3l61p5m0q7wa5g7rz2bl6azqf-gcc-12.2.0-lib/lib/libgcc_s.so.1 (0x00007fc45d9df000)
	librt.so.1 => /nix/store/dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8/lib/librt.so.1 (0x00007fc461f3a000)
	libdl.so.2 => /nix/store/dg8mpqqykmw9c7l0bgzzb5znkymlbfjw-glibc-2.37-8/lib/libdl.so.2 (0x00007fc461f35000)