If a user is configuring its enviroments following superbuild documentation, test executables will link to installed libraries instead of build ones

[traversaro]

[traversaro]

We experienced this issue with @fjandrad : his test was not working as expected, and then we understood that the tests were linking the installed version of the library (instead of the build one, as expected) because LD_LIBRARY_PATH was set to <prefix>/install/lib after https://github.com/robotology/robotology-superbuild/issues/82 .

[traversaro]

This is quite confusing, especially for new developers. @diegoferigo do you think we can revert the LD_LIBRARY_PATH change in https://github.com/robotology/robotology-superbuild/pull/82 ? Do you remember the enviroment in which RPATH was not properly working for the python bindings?

[diegoferigo]

Do you remember the enviroment in which RPATH was not properly working for the python bindings?

Not really, it might have been @anqingd computer.

What about RTF? And also WBToolbox. All projects that use plugins excluded Yarp need that variable in order to find libraries in paths outside the system directories.

Personally, when I need to use libraries from the build directory, I manually edit the environment prepending to LD_LIBRARY_PATH the build directory I need. I prefer asking developers doing this rather than users.

[traversaro]

What about RTF? And also WBToolbox. All projects that use plugins excluded Yarp need that variable in order to find libraries in paths outside the system directories.

You are right, I forgot about this.

Personally, when I need to use libraries from the build directory, I manually edit the environment prepending to LD_LIBRARY_PATH the build directory I need. I prefer asking developers doing this rather than users.

The problem is that our users are also developers. : ) A new student adding an algorithm to iDynTree or UnicyclePlanner will never be able to realize this without proper documentation. We can force the value of the LD_LIBRARY_PATH when running tests via CMake, but it is even an additional layer of complexity.

I will think about it a bit.

[diegoferigo]

I know it is controversial, but I always struggled to use the superbuild for developing aims. I like the help it provides to setup the environment and to handle dependencies, but I personally always work on projects outside the superbuild (which might tho install inside build/install).

If it is a matter of documentation, I would go in this direction. Of course, if we find an alternative to LD_LIBRARY_PATH, and here also @drdanz might be interested, I would totally go in this direction without environment variables.

[diegoferigo]

@traversaro I was thinking, if the problem is inside the build tree, why not using CMake to solve this situation. I explain myself better.

I assume the by default CMake reads the system LD_LIBRARY_PATH and at link time the linker finds the libraries following that hierarchy. However, if:

1. The libraries in the build folder are always installed inside a given directory (e.g. build/lib as we often do)
2. An optional CMake option prepends this folder to the LD_LIBRARY_PATH

Under these circumstances I think the linker called by CMake will find the libraries inside the build folder first and the problem on a single repository (that can be the entire superbuild) can be solved.

This would not solve it system-wise, but specifically for the discussed problem it is the first solution I imagined.

[traversaro]

Note that the problem is that the loader uses the installed libraries, not the linker, or perhaps it affects also the linker, but the problem described here arise due to the loader.

[diegoferigo]

I thought that if RPath is used, the loader uses the relative path stored directly in the executable (or library). This should be done at link time. Am I mistaken?

Plugins are all another story.

[traversaro]

I thought that if RPath is used, the loader uses the relative path stored directly in the executable (or library). This should be done at link time. Am I mistaken?

I am not sure, but i guess (given http://man7.org/linux/man-pages/man8/ld.so.8.html) that CMake sets DT_RUNPATH elf attribute, and for the loader that attribute has lower precedence w.r.t. LD_LIBRARY_PATH .

[diegoferigo]

Interesting, I forgot that LD_LIBRARY_PATH has higher priority than the attributes stored inside the library / executable.

For who's reading this conversation, https://gitlab.kitware.com/cmake/community/wikis/doc/cmake/RPATH-handling is another nice resource it came out in the past from other discussions.

[diegoferigo]

As discussed more in detail in https://github.com/robotology/ycm/issues/199, we cannot solve this issue unless if we remove the ${ROBOTOLOGY_SUPERBUILD_INSTALL_PREFIX}/lib entry from the LD_LIBRARY_PATH introduced in https://github.com/robotology/robotology-superbuild/pull/82.

[traversaro]

As discussed more in detail in https://github.com/robotology/ycm/issues/199, we cannot solve this issue unless if we remove the ${ROBOTOLOGY_SUPERBUILD_INSTALL_PREFIX}/lib entry from the LD_LIBRARY_PATH introduced in https://github.com/robotology/robotology-superbuild/pull/82.

And you agree that this is a good idea (once we fix everything), right?

[diegoferigo]

For some while we didn't have the need of it. I hope there won't be unforeseen by reverting to the previous state (once we fix everything). So yes, :+1:

[traversaro]

One of the reasons we had this LD_LIBRARY_PATH was for some RPATH issues in iDynTree Python bindings that should be fixed by https://github.com/robotology/idyntree/pull/741 .

[traversaro]

Related issue: https://github.com/robotology/yarp/issues/2865 . In that case, the problem was even worse as it was just not related to test executables, but to executables used in the build process of a library.