CMake local install RPATH missing

[Ryanf55]

I tried consuming a locally installed version of MAVSDK in another package's install, and MAVSDK's CMake target fails to correctly export the shared library destination on an installed target. I suspected this is an rpath issue, and spent 2 hours trying various recommendations, but have not found a solution. Here is a conference talk from a CMake maintainer specifically about this use case with rpath: https://youtu.be/m0DwB4OvDXk?si=3TaOFA6OdxY4z5F6&t=2452

I tried his recommendation on MAVSDK, and that did not fix the issue.

To reproduce, here's a Dockerfile:

FROM ubuntu:22.04

WORKDIR /ws

# https://mavsdk.mavlink.io/main/en/cpp/guide/build_linux.html

RUN --mount=target=/var/lib/apt/lists,type=cache,sharing=locked \
    --mount=target=/var/cache/apt,type=cache,sharing=locked \
    apt-get update && \
    DEBIAN_FRONTEND=noninteractive apt-get -y install \
        build-essential \
        cmake \
        git \
        python3 \
        python3-pip

RUN git clone https://github.com/mavlink/MAVSDK.git
WORKDIR /ws/MAVSDK
RUN git submodule update --init --recursive

RUN cmake -DCMAKE_BUILD_TYPE=Debug -Bbuild -S.
RUN cmake --build build -j8
# Install is slightly different not using the install target, but it's the same idea.
RUN cmake --install build --prefix install


RUN --mount=target=/var/lib/apt/lists,type=cache,sharing=locked \
    --mount=target=/var/cache/apt,type=cache,sharing=locked \
    apt-get update && \
    DEBIAN_FRONTEND=noninteractive apt-get -y install \
        pkg-config \
        astyle \
        bluez \
        bluez-tools \
        libbluetooth-dev

WORKDIR /ws
RUN git clone --recurse-submodules --branch cmake-install https://github.com/ryanf55/RemoteIDTransmitter.git
WORKDIR /ws/RemoteIDTransmitter
RUN CMAKE_PREFIX_PATH=/ws/MAVSDK/install cmake -B build
RUN cmake --build build
RUN cmake --install build --prefix install
RUN ./install/bin/rid-transmitter

Just run docker build . --progress=plain and you will eventually get

 > [stage-0 17/17] RUN ./install/bin/rid-transmitter:
0.378 ./install/bin/rid-transmitter: error while loading shared libraries: libmavsdk.so.3: cannot open shared object file: No such file or directory

The example here is when I tried using an installed executable that links to MAVSDK. The linkage to MAVSDK is using the standard target_link_libraries approach: https://github.com/Ryanf55/RemoteIDTransmitter/blob/cmake-install/CMakeLists.txt#L37

FYI @dakejahl

[julianoes]

# Install is slightly different not using the install target, but it's the same idea.
RUN cmake --install build --prefix install

Why? Isn't this the root cause? Why can't you use the --target install as it is intended?

[Ryanf55]

# Install is slightly different not using the install target, but it's the same idea.
RUN cmake --install build --prefix install

Why? Isn't this the root cause? Why can't you use the --target install as it is intended?

No, you get the same error, the only difference is one allows you to set the install location at install time, while the other requires to decide during configure time. For this example, it has no effect.

[JonasVautherin]

First, a quick fix: you can set LD_LIBRARY_PATH:

LD_LIBRARY_PATH=/ws/MAVSDK/install/lib/ ./install/bin/rid-transmitter

I tried his recommendation on MAVSDK, and that did not fix the issue.

What did you try? Note that I didn't watch the video.

I would be tempted to try something like this:

diff --git a/src/mavsdk/core/CMakeLists.txt b/src/mavsdk/core/CMakeLists.txt
index 6392ba41..73f6194e 100644
--- a/src/mavsdk/core/CMakeLists.txt
+++ b/src/mavsdk/core/CMakeLists.txt
@@ -130,10 +130,19 @@ else()
     )
 endif()
 
+if(IS_ABSOLUTE "${CMAKE_INSTALL_LIBDIR}")
+    set(MAVSDK_INSTALL_RPATH "${CMAKE_INSTALL_LIBDIR}")
+else()
+    set(MAVSDK_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}")
+endif()
+
 set_target_properties(mavsdk PROPERTIES
     VERSION ${MAVSDK_VERSION_STRING}
     SOVERSION ${MAVSDK_SOVERSION_STRING}
+    INSTALL_RPATH "${MAVSDK_INSTALL_RPATH}"
+    INSTALL_RPATH_USE_LINK_PATH TRUE
 )
+
 if (MSVC)
     set_target_properties(mavsdk PROPERTIES
         DEBUG_POSTFIX "d"

Now if I build it with:

cmake -DCMAKE_INSTALL_PREFIX=install -DCMAKE_BUILD_TYPE=Debug -Bbuild -S.
cmake --build build --target install

Then the rpath is correct and it works with your rid-transmitter:

# readelf -d /ws/MAVSDK/install/lib/libmavsdk.so.3.10.2 | grep RUNPATH
 0x000000000000001d (RUNPATH)            Library runpath: [/ws/MAVSDK/install/lib]

The problem with your cmake --install build --prefix install is then that it sets the install path at install time, though the rpath is set at build time. So the way you call it, it would result in the wrong rpath anyway, because my patch above relies on CMAKE_INSTALL_PREFIX.

# readelf -d /ws/MAVSDK/install/lib/libmavsdk.so.3.10.2 | grep RUNPATH
 0x000000000000001d (RUNPATH)            Library runpath: [/usr/local/lib]

Now one may argue that it is good enough to have it set with CMAKE_INSTALL_PREFIX, and if you really need to change the install path at install time with --prefix, then you can manually set the rpath with something like:

patchelf --set-rpath /ws/MAVSDK/install/lib /ws/MAVSDK/install/lib/libmavsdk.so.3.10.2

Or set LD_LIBRARY_PATH.

@Ryanf55: what do you think? Does that correspond to what your video says? Maybe there is a better way that works with setting the --prefix at install time, but my guess is that CMake won't go change the binary at this step, so it's too late to change the runpath.

[JonasVautherin]

I actually watched the video you linked above, and it is a very nice presentation 😊. However, I don't think that setting the rpath to $ORIGIN will solve this particular problem you have. It would help MAVSDK find its own dependencies, but in your case the problem is about rid-transmitter not finding MAVSDK.

When you have your issue, the ldd output looks like this:

# ldd install/bin/rid-transmitter 
	linux-vdso.so.1 (0x00007fe0137ef000)
	libmavsdk.so.3 => not found
	libbluetooth.so.3 => /lib/x86_64-linux-gnu/libbluetooth.so.3 (0x00007fe013753000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fe013400000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe01366c000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fe01364c000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe013000000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fe0137f1000)

It could build against MAVSDK, but at runtime the dynamic linker doesn't know where to find it because libmavsdk.so.3 did not have a runpath set and is installed in a custom location.

My understanding is that the $ORIGIN trick would help your executable. Maybe it's easier to show with an example. You have this issue:

# ./install/bin/rid-transmitter 
./install/bin/rid-transmitter: error while loading shared libraries: libmavsdk.so.3: cannot open shared object file: No such file or directory

Say you copy libmavsdk.so.3 next to rid-transmitter and set the runpath to $ORIGIN, then it will work:

# cp /ws/MAVSDK/install/lib/libmavsdk.so.3 /ws/RemoteIDTransmitter/install/bin/
# patchelf --set-rpath '$ORIGIN' /ws/RemoteIDTransmitter/install/bin/rid-transmitter

# ./install/bin/rid-transmitter 
Parsing failed:
File could not be opened for reading
	(error occurred at line 0, column 0 of '/root/.local/share/rid-transmitter/config.toml')

I am assuming that the "Parsing failed" error is orthogonal to the dynamic linking issue we're discussing now 🙈.

So you can use $ORIGIN in order to find dependencies next to your binary, but setting it into your dependencies won't help you in this case.

[julianoes]

What about these three workarounds:

1. Install into /usr:

RUN git clone https://github.com/mavlink/MAVSDK.git
WORKDIR /ws/MAVSDK
RUN git submodule update --init --recursive

RUN cmake -DCMAKE_BUILD_TYPE=Debug -Bbuild -S. -DCMAKE_INSTALL_PREFIX=/usr
RUN cmake --build build -j8
RUN cmake --build build --target install

2. Install into /usr/local (which is the cmake default):

RUN git clone https://github.com/mavlink/MAVSDK.git
WORKDIR /ws/MAVSDK
RUN git submodule update --init --recursive

RUN cmake -DCMAKE_BUILD_TYPE=Debug -Bbuild -S.
RUN cmake --build build -j8
RUN cmake --build build --target install

3. Install using wget/dpkg

RUN --mount=target=/var/lib/apt/lists,type=cache,sharing=locked \
    --mount=target=/var/cache/apt,type=cache,sharing=locked \
    apt-get update && \
    DEBIAN_FRONTEND=noninteractive apt-get -y install \
        build-essential \
        cmake \
        git \
        wget

RUN wget https://github.com/mavlink/MAVSDK/releases/download/v3.10.2/libmavsdk-dev_3.10.2_ubuntu22.04_amd64.deb
RUN dpkg -i libmavsdk-dev_3.10.2_ubuntu22.04_amd64.deb

[JonasVautherin]

What about these three workarounds

Those will work because they install MAVSDK on the system. I think the original issue is that they want to install MAVSDK locally and use it from there.

The more I think about it, the more I believe that the solution is to either use LD_LIBRARY_PATH or manually patchelf:

RUN git clone https://github.com/mavlink/MAVSDK.git
WORKDIR /ws/MAVSDK
RUN git submodule update --init --recursive

RUN cmake -DCMAKE_BUILD_TYPE=Debug -Bbuild -S.
RUN cmake --build build -j8
RUN cmake --install build --prefix install
patchelf --set-rpath /ws/MAVSDK/install/lib /ws/MAVSDK/install/lib/libmavsdk.so.3.10.2

Because MAVSDK cannot guess the final use-case and should probably not leak paths of the build machine. The package knows how it will be used and can set the runpath after installation (with patchelf --set-rpath).