AppImage is loading system libraries when run

[Zaraka]

I've started to experiencing a really weird bug. Built appimages works without problem on deploy machine. ldd on extracted AppImage shows correctly linked libraries. But when they are transferred to other machines which have Qt installed in their system path, Appimage will try to load system libraries which of course will result in errors like

./MapXXX.AppImage: /tmp/.mount_MapTilLZZXTY/lib/libQt5Core.so.5: version `Qt_5.11.3_PRIVATE_API' not found (required by /lib64/libQt5OpenGL.so.5)
./MapXXX.AppImage: /tmp/.mount_MapTilLZZXTY/lib/libQt5Widgets.so.5: version `Qt_5.11.3_PRIVATE_API' not found (required by /lib64/libQt5OpenGL.so.5)
./MapXXX.AppImage: /tmp/.mount_MapTilLZZXTY/lib/libQt5Gui.so.5: version `Qt_5.11.3_PRIVATE_API' not found (required by /lib64/libQt5OpenGL.so.5)

If I --appimage-extract AppImage and ldd binary it will show libraries linked from system paths. AppImage only works when I run it with LD_LIBRARY_PATH=lib ./MapXXX.AppImage. This started suddenly happening after I added new dependency which needs Qt libraries as well. But I've made sure that dependency is correctly compiled with the same Qt as this AppImage is.

[probonopd]

What happens if you unset LD_LIBRARY_PATH before you run linuxdeployqt?

[Zaraka]

No change

[probonopd]

Can you post a link to the AppImage for testing? Thanks.

[probonopd]

The issue seems to be that something within the AppImage is loading libQt5OpenGL from the system rather than from within the AppImage. We must find out what needs libQt5OpenGL and whether libQt5OpenGL is deployed into the AppDir.

Runing sudo strace -f ./Your.AppImage 2>&1 | grep libQt5OpenGL can help to answer this question...

[Zaraka]

When run with strace, binary works just fine. It loads libQt5OpenGL from inside the appimage

$ sudo strace -f ./Map.AppImage 2>&1 | grep libQt5OpenGL [pid 14021] openat(AT_FDCWD, "/tmp/.mount_MapTiluHzs5E/lib/libQt5OpenGL.so.5", O_RDONLY|O_CLOEXEC <unfinished ...>

I've checked if main binary have every dependency satisfied and yes it does. Here is the strace with grep Qt https://pastebin.com/REANfebF and here is contents of the lib directory. https://pastebin.com/dZ7t7gFF I guess something is wrong with the AppRun, but I have no idea if I can debug it somehow. I probably can't post the whole appimage, it's closed source project, but I'm going to try to create a minimal working example.

[Zaraka]

So, I've created a minimal working example. It still has all the library dependencies (which is what causes this bug in the first place). You can access it at https://drive.google.com/file/d/1F-x2TOvzW227coySI5zBlaA9UyHcXsJ8/view?usp=sharing when successfully run it will only display an empty window and a bunch of SSL errors, this can be ignored. When run on a system which has system Qt5, then it will fail to start.

[Zaraka]

I've had an idea that maybe my non-standard flat directory structure may be causing this. So I've changed the structure to follow usr/bin directories. Unfortunately this didn't do anything. Then I've found out the wiki article about custom AppRun, so I've implemented custom wrapper script

#!/bin/bash
HERE="$(dirname "$(readlink -f "${0}")")"
export LD_LIBRARY_PATH=${HERE}/usr/lib:$LD_LIBRARY_PATH
exec "${HERE}/usr/bin/ApplicationBinary "$@"

and used appimage tool and then it all started working. I wouldn't close this bug, this feels more like a workaround/hack than actual solution.

[probonopd]

Thanks for the update @Zaraka. A LSB-like filesystem structure inside the AppImage is needed, It's the only structure that linuxdeployqt is really tested with, and I don't have the time to test it against other scenarios. People are free to send pull requests with improvements, though.

To make things simpler and to remove variability, we could throw an error if linuxdeployqt does not find a LSB-like filesystem structure inside the AppImage, and exit.

[Zaraka]

Please keep in mind that changing it to LSB didn't fix it! Deploying custom AppRun did. I'll try later uploading AppImage with the same issue and correct filesystem.

[probonopd]

Then that is an indication that some rpath does not get set correctly. LD_LIBRARY_PATH should not be needed if everything goes correctly. We need to compare what gets loaded without and with that fix, and find out what is loading libs from the wrong place.

[haampie]

I had a similar issue where system libraries where used according to ldd, but this was in fact because the executable just has a RUNPATH set up and LD_LIBRARY_PATH takes precedence over that, meaning that it will search for libraries in LD_LIBRARY_PATH first and in $ORIGIN/lib second:

$ readelf myApp -d | grep path
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/lib]

in my case I just emptied the LD_LIBRARY_PATH variable and then ldd gives the right local dependencies.

Maybe we could change RUNPATH into RPATH s.t. locating the local dependencies is not spoiled by LD_LIBRARY_PATH?

[probonopd]

Why would you have LD_LIBRARY_PATH set to something? And if so, why should it not take preference?

[haampie]

Well, I thought Qt Creator was setting LD_LIBRARY_PATH to the directories you're linking to when running the app, so I thought it was the way to go. Only later I read about rpath and added the linker flags.

Maybe a reason to prefer rpath is related to https://github.com/probonopd/linuxdeployqt/issues/39#issuecomment-495292784: if I don't want to bundle a system lib and delete the bundled copy, I might want to specify LD_LIBRARY_PATH to include /usr/lib/x86_64-linux-gnu/, but this directory also happens to have the wrong version of all Qt's libs, so the application won't run. This could be fixed by setting the rpath s.t. bundled dependencies take preference. (But probably the right way is to just fix the full path of the system lib in the binary then and forget about LD_LIBRARY_PATH.)

[probonopd]

I think those are very special edge cases, we have not run into this issue so far. How does https://github.com/linuxdeploy/linuxdeploy-plugin-qt handle those cases?