Helpers for handling RPATH and company

[LecrisUT]

trafficstars

The issue I am thinking of solving is to have a way to inject some LD_LIBRARY_PATH or PATH so that it accounts for the path discrepancy of CMake installed files under site_packages and bin/Scripts location. I am considering two distinct cases here

Linking to dependencies

Writing the logic to create the appropriate RPATH for a dependency can be quite tricky, but maybe we can provide some helper functions. The key part is that within scikit-build-core we have better information of if a dependency is coming from the python dependencies, and what the relative paths between the root of the dependency and the final installation path including wheel.install-dir are.

For the user interface, I was thinking of providing a CMake function to handle an imported target and append an appropriate INSTALL_RPATH. Here is a prototype

prototype

    function(scikit_build_add_rpath requesting_target imported_target)
        # Save the rpath origin symbol
        if(APPLE)
            set(origin_symbol "@rpath")
        else()
            set(origin_symbol "$ORIGIN")
        endif()
        # Check if the imported_target is imported
        get_target_property(imported ${imported_target} IMPORTED)
        if(NOT imported)
            # TODO: What to do with non-imported targets?
            return()
        endif()
        # Check what type of imported_target it is
        get_target_property(type ${imported_target} TYPE)
        if(NOT type STREQUAL "SHARED_LIBRARY")
            # We only add rpaths to shared library
            # Technically EXECUTABLE could also be linked against
            # TODO: Are there rpaths in static library and do they propagate
            # TODO: Is there some handling to do for `INTERFACE_LIBRARY`/`OBJECT_LIBRARY`?
            return()
        endif()
        # Get the library file location
        get_target_property(location ${imported_target} LOCATION)
        # Take the parent directory
        cmake_path(GET location PARENT_PATH location)
        # Get the relative path w.r.t.
        cmake_path(RELATIVE_PATH location
            # scikit-build-core: provide `mocked_wheel_install_dir` pointing to the
            # `wheel.install-dir` in the build environment
            BASE_DIRECTORY ${mocked_wheel_install_dir}
        )
        # Invert the path from CMAKE_INSTALL_LIBDIR to `wheel.install-dir`
        set(path_to_wheel_install_dir ".")
        cmake_path(RELATIVE_PATH path_to_wheel_install_dir
            BASE_DIRECTORY ${CMAKE_INSTALL_LIBDIR}
        )
        # Construct the rpath needed by the
        cmake_path(APPEND rpath
            "${path_to_wheel_install_dir}"
            "${location}"
        )
        cmake_path(NORMAL_PATH rpath)
        # We add `origin_symbol` at the end to not interfere with cmake_path
        set(rpath "${origin_symbol}/${rpath}")
        set_property(TARGET ${requesting_target} APPEND
            PROPERTY INSTALL_RPATH "${rpath}"
        )
        # TODO: How to handle the windows PATH?
    endfunction()

Probably mocked_wheel_install_dir would be passed as a cache variable?

Patching current project being build

For python bindings, constructing the RPATH is not that difficult since it's just $ORIGIN/${CMAKE_INSTALL_LIBDIR}, and I am not sure how to provide a clean interface for this. Maybe having a scikit_build_install_python_module helper? But then how do we get the current build's install path?

The more difficult part would be project.scripts entry points for wrappers or compiled binaries, i.e. something like

import subprocess

def run():
    subprocess.call(
        ["@CMAKE_INSTALL_BINDIR@/@name@"]
    )

Handling Window's PATH

WIP: No idea right now

[henryiii]

We don't know the final bin path relative to anything else, that's handled by the installer after it makes the wheel. There's no mechanism in installers to communicate it as far as I know. It's usually related, but AFAIK you can actually completely customize where data, bin, and headers go. I think providing a way to do that would require a PEP and changes to pip, uv, installer, etc.

[henryiii]

Not very related, though, I wonder if we can set variables so that install(... TYPE ...) works?

[LecrisUT]

We don't know the final bin path relative to anything else, that's handled by the installer after it makes the wheel.

Yeah, but we do know the paths relative inside the site_package, so we could generate the python wrappers that expand @CMAKE_INSTALL_BINDIR@/@name@ (e.g. in foo/__main__.py) and add it dynamically to the project.scripts. Then we could also inject necessary LD_LIBRARY_PATH and probably even os.add_dll_directory() for Windows.

Not very related, though, I wonder if we can set variables so that install(... TYPE ...) works?

Not sure how that would look like.

---

The most annoying one to design for is windows. We could maybe solve the project.scripts wrappers, but the difficult ones would be the dll. I am still playing around with install(IMPORTED_RUNTIME_ARTIFACTS) to see if it can do anything useful, or if we can copy the dll files to Scripts? And one difficult part is making python bindings work since we can't just add $ORIGIN/${CMAKE_INSTALL_LIBDIR}, instead it seems more practical to have a way to inject the os.add_dll_directory() inside the C++ source code?

[LecrisUT]

So progress report after experimenting a bit in https://github.com/LecrisUT/experiment-skbuild-wrapper. That one also shows the main dependency problem to resolve.

I have almost all moving parts figured out, and probably with some cmake-file-api magic, we can get all the information needed. Although I'm not sure where/how to insert the rpath injection after the configure/generation step. A few things I have hit:

• install(IMPORTED_RUNTIME_ARTIFACTS) does nothing for me. I don't know exactly what I am doing wrong with that
• os.add_dll_directory does not work for a binary wrapper that calls subprocess.call, instead I need to patch the PATH environment
• For some reason, when I add a python module, the INSTALL_RPATH gets resolved on an unrelated binary target. This one is a total head-scratcher https://github.com/LecrisUT/experiment-skbuild-wrapper/pull/2

---

Hmm maybe another solution would be to integrate with auditwheel as a plugin. Might still need to pass the build-time library.