Broken rpath output when building Zig with Nix

[erikarvstedt]

trafficstars

Since #17917, when building Zig within a Nix environment, the rpath of the resulting Zig binary is faulty/broken:

1. libgcc, the lib providing the direct dependency libstdc++.so, is missing from rpath. This causes an error when running zig:

   zig: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory
   

   Previously, libgcc was correctly included in rpath.

2. Include paths from NIX_CFLAGS_COMPILE are present in rpath. These paths contain no shared objects and should not be present in rpath. Examples:

   /nix/store/nhv7xmxzi63lp3x1j7zrwb7f2248sgl2-libxml2-2.11.5-dev/lib
   /nix/store/fak09pxs42fqcgngf2yznw83f88kmw7s-zlib-1.3-dev/lib
   ...
   

Details

Zig rpath before #17917:

/home/user/src/outputs/out/lib
/nix/store/rzx1szlgxck2wj4dqhq1q3p4330nj47j-llvm-17.0.6-lib/lib
/nix/store/lprrvg44cyn9ajpq77lh7rlnp4kzdqzs-zlib-1.3/lib
/nix/store/8g3myidcxw02mv9pzyvwq9cv0y9nayi7-libxml2-2.11.5/lib
/nix/store/fmkdf1b03w5v9yzk2mm3b4yqg1gwhrf5-ncurses-6.4/lib
/nix/store/lqrynl0j2ph8ggsfcc41nqcw5di0va50-clang-17.0.6-lib/lib
/nix/store/dghjv6hfz0s0z4kffa5ahyw2mhp79215-gcc-12.3.0-lib/lib     (Missing after #17917)
/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27/lib

Zig rpath after #17917:

/nix/store/rzx1szlgxck2wj4dqhq1q3p4330nj47j-llvm-17.0.6-lib/lib
/nix/store/nhv7xmxzi63lp3x1j7zrwb7f2248sgl2-libxml2-2.11.5-dev/lib (Include path)
/nix/store/fak09pxs42fqcgngf2yznw83f88kmw7s-zlib-1.3-dev/lib       (Include path)
/nix/store/lprrvg44cyn9ajpq77lh7rlnp4kzdqzs-zlib-1.3/lib
/nix/store/8g3myidcxw02mv9pzyvwq9cv0y9nayi7-libxml2-2.11.5/lib
/nix/store/v1wj8jz1ap5rf5yqh1wn5ykqg9fr5vds-clang-17.0.6-dev/lib   (Include path)
/nix/store/lqrynl0j2ph8ggsfcc41nqcw5di0va50-clang-17.0.6-lib/lib
/nix/store/q6af8zwgwpr7lkfk8rcxc9pyvmmrclhd-lld-17.0.6-dev/lib     (Include path)
/nix/store/pakl6wfmh8vkir27kjlfdb6677pix001-lld-17.0.6-lib/lib
/nix/store/zpxb1xsx483qdzxf9ps3vjyqija5pms0-llvm-17.0.6-dev/lib    (Include path)
/nix/store/ifx5q9x3vcck3b01wb9h47x7va00snkd-ncurses-6.4-dev/lib    (Include path)
/nix/store/fmkdf1b03w5v9yzk2mm3b4yqg1gwhrf5-ncurses-6.4/lib
/home/main/d/zig-dev/zig/outputs/out/lib

When the Zig src is located in /home/user/src/zig, the path to the nonexisting dir /home/user/src/outputs/out/lib is added to rpath. This is probably also an error, but this was already present before #17917.

Reproduce

# Enter Nix dev env for Zig.
# This sets up a build environment, setting env vars like `NIX_CFLAGS_COMPILE`, `NIX_LDFLAGS`, ...
nix develop github:erikarvstedt/nix-zig-build/f3ef222402b1e75e691c9122e17ac4f96ffd8af4
                                                                                                                                        
# Enter Zig src dir
cd /path/to/zig-src
                                                                                                                                        
cmake . -DCMAKE_BUILD_TYPE=Release -DZIG_NO_LIB=ON -GNinja
                                                                                                                                        
## Build Zig. option 1: bootstrap (slow)
ninja install
                                                                                                                                        
## Build Zig, option 2: use prebuilt Zig (fast)
ninja zigcpp
# Grab a Zig build that includes #17917 (852e7e2)
version=0.12.0-dev.2236+32e88251e
url=https://ziglang.org/builds/zig-linux-x86_64-$version.tar.xz
# Extract Zig binary to ./zig
curl -fL "$url" | tar xJ --strip-components=1 --wildcards --no-wildcards-match-slash '*/zig'
./zig build -Denable-llvm -Dno-lib --prefix stage3
                                                                                                                                        
./stage3/bin/zig
# => error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory
                                                                                                                                        
ldd ./stage3/bin/zig
readelf -d ./stage3/bin/zig

cc @Jan200101, @kubkon, @andrewrk

[andrewrk]

why are you saying those lib directories are include paths? What files are in there?

[erikarvstedt]

Ah, sorry, these *-dev paths are not only present in NIX_CFLAGS_COMPILE, but also in other env vars.

Let's take /nix/store/nhv7xmxzi63lp3x1j7zrwb7f2248sgl2-libxml2-2.11.5-dev as an example. It's included in these env vars:

Env var	Env var entry
CMAKE_INCLUDE_PATH	/nix/store/nhv7xmxzi63lp3x1j7zrwb7f2248sgl2-libxml2-2.11.5-dev/include
CMAKE_PREFIX_PATH	/nix/store/nhv7xmxzi63lp3x1j7zrwb7f2248sgl2-libxml2-2.11.5-dev
NIX_CFLAGS_COMPILE	-isystem /nix/store/nhv7xmxzi63lp3x1j7zrwb7f2248sgl2-libxml2-2.11.5-dev/include
PATH	/nix/store/nhv7xmxzi63lp3x1j7zrwb7f2248sgl2-libxml2-2.11.5-dev/bin

Contents:

/nix/store/nhv7xmxzi63lp3x1j7zrwb7f2248sgl2-libxml2-2.11.5-dev
├── bin
│   └── xml2-config
├── include
│   └── libxml2
│       └── libxml
│           ├── c14n.h
│           ├── ...
├── lib
│   ├── cmake
│   │   └── libxml2
│   │       └── libxml2-config.cmake
│   └── pkgconfig
│       └── libxml-2.0.pc
├── nix-support
│   └── propagated-build-inputs
└── share
    └── aclocal
        └── libxml.m4

The same goes for all other *-dev paths. They are dev pkg outputs that contain build-related tooling and source files but no binary objects.

[andrewrk]

Hmm. When I do the following on my system:

$ nix-shell -p libxml2
$ env | grep NIX_LDFLAGS
NIX_LDFLAGS=-rpath /nix/store/2swpy93b6y5bsvmb7jawlmwwipwxmy2m-shell/lib64 -rpath /nix/store/2swpy93b6y5bsvmb7jawlmwwipwxmy2m-shell/lib  -L/nix/store/8mw6ssjspf8k1ija88cfldmxlbarl1bb-zlib-1.2.13/lib -L/nix/store/awj9x4rwndilh9hcwf121yz3xr9j64w6-libxml2-2.10.4/lib -L/nix/store/8mw6ssjspf8k1ija88cfldmxlbarl1bb-zlib-1.2.13/lib -L/nix/store/awj9x4rwndilh9hcwf121yz3xr9j64w6-libxml2-2.10.4/lib
$ ls /nix/store/awj9x4rwndilh9hcwf121yz3xr9j64w6-libxml2-2.10.4/lib
libxml2.la  libxml2.so  libxml2.so.2  libxml2.so.2.10.4

It looks to me like adding an rpath for that -L directory was the right move. I think the only confirmed regression in this issue is the missing directory that you identified above - the gcc one. I'm not convinced these other differences are problematic.

[erikarvstedt]

Yes, these are lib outputs in NIX_LDFLAGS from pkg libxml2.lib. They should be added to the rpath. #17917 didn't change that. What changed is that libxml2.dev outputs, which are not included in NIX_LDFLAGS (as shown in the env var table above), are also added to the rpath.

[erikarvstedt]

NIX_LDFLAGS:                     -L/nix/store/8g3myidcxw02mv9pzyvwq9cv0y9nayi7-libxml2-2.11.5/lib  (should be added to rpath)
NIX_CFLAGS_COMPILE (and others):   /nix/store/nhv7xmxzi63lp3x1j7zrwb7f2248sgl2-libxml2-2.11.5-dev  (should not be added to rpath)

[andrewrk]

Hmm, I see. The new implementation of -feach-lib-rpath naively adds each library search directory to the rpath list regardless of whether a dynamic library was used from inside that directory. That should be adjusted.

Alternately, we could proceed with removing that feature entirely, as I mentioned in that PR, since the original motivation for -feach-lib-rpath was NixOS anyway. On the other hand, -feach-lib-rpath as opposed to adding an rpath for each -L from NIX_LDFLAGS creates a binary with a slightly more efficient runtime startup, because it prevents the dynamic linker from searching unnecessary directories to find dynamically linked libraries. With the current strategy of adding an RPath for each -L argument in NIX_LDFLAGS it could potentially add unnecessary RPaths if those are in the nix environment but not ultimately used by the executable.

What remains to be diagnosed is

/nix/store/dghjv6hfz0s0z4kffa5ahyw2mhp79215-gcc-12.3.0-lib/lib     (Missing after #17917)

[erikarvstedt]

Only adding required rpath entries would be perfect. nixpkgs has a stdlib feature (patchelf, autoPatchelfHook) that prunes unneeded rpaths, but this is inconvenient when using Zig as a standalone build tool, outside of Nix derivations.

[oldendick]

The libstdc++.so missing from rpath issue is likely related to #18742

[erikarvstedt]

Zig somehow auto-detects libstdc++ via cmake.

Ideally, Zig should remove this heuristic or correctly setup the rpath on Nix, so that a successful build guarantees that the binary can load its libs correctly.

[Jan200101]

Zig somehow auto-detects libstdc++ via cmake.

I believe it does this in build.zig https://github.com/ziglang/zig/blob/d0c06ca7127110a8afeb0ef524a197049892db21/build.zig#L715 (There are also other non linux parts that directly link libstdc++)

Solutions include:

• ~~abandoning all hope for anything rpath related and giving in to the dark lord~~
• checking if the found stdc++ library is on a NativePath, if not add it to the rpath
  • something along those lines: https://github.com/ziglang/zig/commit/d4328e42f4c4eede8ef93125ea1a59890b71d2c6
  • technically the same issue also exists for llvm, but the Nix flags deal with this for us.
• setting use-zig-libcxx to use the included libc++

[Cloudef]

FYI I complained about this in a other issue, but IMO zig should not use the nixpkgs specific env vars at all https://github.com/ziglang/zig/issues/18998#issuecomment-1975056100

[igaryhe]

is it possible to also add the path to libstdc++.so to rpath? https://github.com/ziglang/zig/blob/fb88cfdf6aa3fabba700d8340f025e4a3e0d3fb2/build.zig#L819 it's a bit hacky, but maybe something like

exe.addRPath(.{ .cwd_relative = path_unpadded[0..path_unpadded.len - 1 - objname.len] });

so that we can ensure the path to libstdc++.so would ended up in rpath