Handling executables (scripts from the Python POV) that link to distribution libraries

[FFY00]

I don't think auditwheel fixes the binary dependencies for wheel scripts, we should confirm that. If it does not, we should put them in platlib and add a wrapper that just calls them in the scripts directory.

The issue here is that we do not know the final location of the .libs folder so we cannot set the path to point there. By placing them in platlib, we can set the rpath to $ORIGIN/../.libs.

[rgommers]

The issue here is that we do not know the final location of the .libs folder

Why isn't it staying at pkgname/.libs (every package I'm aware of has it there, and I thought it was an auditwheel thing - but could be multibuild doing it)? Or do you need an absolute path somewhere?

[FFY00]

Yes, but we need to set the rpath of the binary to path of the .libs folder, which we do not know. It would be either absolute, which we do not know, or relative (using $ORIGIN), which we also not know if we place the binary in the scripts folder.

[FFY00]

Or does auditwheel relocate the binary itself? This is what I need to check, and the reason for this issue. I was under the impression it didn't, but am not sure.

[FFY00]

I haven't had much time to look into this so I just put my thoughts down here for future reference.

[rgommers]

Or does auditwheel relocate the binary itself?

I thought it did, but I'm not 100% sure.

[FFY00]

So, auditwheel does try to do something, but it's incorrect.

$ cd tests/packages/library
$ python -m build -nxw
* Building wheel...
! Using Meson to generate the project metadata (no `project` section in pyproject.toml)
+ meson setup --prefix=/usr /home/anubis/git/mesonpy/tests/packages/library /tmp/mesonpy-8ucf00vy/build
The Meson build system
Version: 0.59.99
Source dir: /home/anubis/git/mesonpy/tests/packages/library
Build dir: /tmp/mesonpy-8ucf00vy/build
Build type: native build
Project name: library
Project version: 1.0.0
C compiler for the host machine: ccache cc (gcc 11.1.0 "cc (GCC) 11.1.0")
C linker for the host machine: cc ld.bfd 2.36.1
Host machine cpu family: x86_64
Host machine cpu: x86_64
Build targets in project: 2

Found ninja-1.10.2 at /usr/bin/ninja
+ meson compile
[5/5] Linking target example
+ meson install --destdir /tmp/mesonpy-8ucf00vy/build
ninja: Entering directory `/tmp/mesonpy-8ucf00vy/build'
ninja: no work to do.
Installing libexample.so to /tmp/mesonpy-8ucf00vy/build/usr/lib
Installing example to /tmp/mesonpy-8ucf00vy/build/usr/bin
WARNING File could not be mapped to an equivalent wheel directory: /tmp/mesonpy-8ucf00vy/build/usr/lib/libexample.so ({libdir_shared}/libexample.so)
+ auditwheel repair -w /tmp/mesonpy-8ucf00vy/build/auditwheel-output /tmp/mesonpy-8ucf00vy/build/library-1.0.0-py3-none-linux_x86_64.whl
INFO:auditwheel.main_repair:Repairing library-1.0.0-py3-none-linux_x86_64.whl
INFO:auditwheel.wheeltools:Previous filename tags: linux_x86_64
INFO:auditwheel.wheeltools:New filename tags: manylinux_2_5_x86_64, manylinux1_x86_64
INFO:auditwheel.wheeltools:Previous WHEEL info tags: py3-none-linux_x86_64
INFO:auditwheel.wheeltools:New WHEEL info tags: py3-none-manylinux_2_5_x86_64, py3-none-manylinux1_x86_64
INFO:auditwheel.main_repair:
Fixed-up wheel written to /tmp/mesonpy-8ucf00vy/build/auditwheel-output/library-1.0.0-py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Successfully built library-1.0.0-py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl

$ unzip dist/*.whl -d wheel-contents
Archive:  dist/library-1.0.0-py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl
   creating: wheel-contents/library.libs/
   creating: wheel-contents/library-1.0.0.data/
   creating: wheel-contents/library-1.0.0.dist-info/
  inflating: wheel-contents/library.libs/libexample-d5767f33.so
   creating: wheel-contents/library-1.0.0.data/scripts/
  inflating: wheel-contents/library-1.0.0.data/scripts/example
  inflating: wheel-contents/library-1.0.0.dist-info/RECORD
  inflating: wheel-contents/library-1.0.0.dist-info/WHEEL
  inflating: wheel-contents/library-1.0.0.dist-info/METADATA

$ readelf -d wheel-contents/library-1.0.0.data/scripts/example

Dynamic section at offset 0x6000 contains 28 entries:
  Tag        Type                         Name/Value
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/:$ORIGIN/../../library.libs]
 0x0000000000000001 (NEEDED)             Shared library: [libexample-d5767f33.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x1000
 0x000000000000000d (FINI)               0x11f8
 0x0000000000000019 (INIT_ARRAY)         0x3dc8
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x3dd0
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x5178
 0x0000000000000005 (STRTAB)             0x6210
 0x0000000000000006 (SYMTAB)             0x50b8
 0x000000000000000a (STRSZ)              218 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x4000
 0x0000000000000002 (PLTRELSZ)           48 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x630
 0x0000000000000007 (RELA)               0x570
 0x0000000000000008 (RELASZ)             192 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffb (FLAGS_1)            Flags: PIE
 0x000000006ffffffe (VERNEED)            0x550
 0x000000006fffffff (VERNEEDNUM)         1
 0x000000006ffffff0 (VERSYM)             0x540
 0x000000006ffffff9 (RELACOUNT)          3
 0x0000000000000000 (NULL)               0x0

As you can see, the rpath is set to $ORIGIN/:$ORIGIN/../../library.libs, which is incorrect. There's no guarantee the library will be at $ORIGIN/../../library.libs ($ORIGIN here being the location of the binary itself), in fact, that does not even happen under a normal Python installation on Linux. The executable will go to /usr/bin/example, so the rpath points to /usr/library.libs.

[rgommers]

We're no longer using auditwheel, and this issue isn't really a bug - more something very low prio that isn't supported right now. I am removing the "bug" label. Let's wait with doing anything here until a real-world use case comes along.

[rgommers]

In case someone needs this - there's at least two ways of handling this scenario right now:

1. Statically linking libraries built within the project itself
2. Installing a thin .py wrapper script to the scripts directory and installing the actual executable in site-packages (where it can reliably use RPATHs)