Do not fail if libraries listed in the exclude list are missing

[mardy]

Update: this bug was previously files with the title "Keeping excludelist up-to-date". During the course of the discussion it emerged that this was not the real issue, but rather that linuxdeployqt exits with an error if some libraries listed in the exclude list are missing.

Original bug description:

The current excludelist.h file does not contain libEGL.so.1, which was added to the pkg2appimage repository.

Beside being a request for an update, in opening this issue I would like to start a discussion about having a way to keep the list up to date. A few options (one does not exclude the others):

1. Add an --excludelist-file= command line option; the user could have a script to download the excludelist from the pkg2appimage repo, and this would override the internally hardcoded list.
2. Update the excludelist.h header at build time, instead of keeping it in git and requiring a manual update.
3. Update the internal list at run time, by having linuxdeployqt connect to github and download the updated list.

Given how easy it is to write a script to fecth the current list, I believe that option 1 should be enough. If you agree with the suggestion, please let me know and I can try to provide a patch myself.

[TheAssassin]

Update the excludelist.h header at build time, instead of keeping it in git and requiring a manual update.

This should be happening already:

• https://github.com/probonopd/linuxdeployqt/blob/master/tools/generate-excludelist.sh
• https://github.com/probonopd/linuxdeployqt/blob/5542062897811e9ff061e8debd929467e921be4e/tools/linuxdeployqt/linuxdeployqt.pro#L43
• https://github.com/probonopd/linuxdeployqt/blob/5542062897811e9ff061e8debd929467e921be4e/tools/linuxdeployqt/CMakeLists.txt#L16

If you have a couple minutes, could you try to find the issue? The script (and the build systems) should be logging messages on the console, so there should be an error message or so in the Travis logs.

[mardy]

I only assumed that it was out of date because I got an error message about libEGL.so.1 (required by the qt xcb plugin) not being found. Then I added it to the command line as -exclude-libs=libEGL.so.1 and got the same error, so it looks like this is not related to the exclude list. After installing libEGL.so.1 in my machine, it worked.

So maybe the issue is that linuxdeployqt applies the exclude list at a very late stage, after all library dependency have been resolved; I would expect that if a library cannot be resolved, but it happens to be in the exclude list, then the appimage creation should proceed normally. Does this make sense?

[TheAssassin]

Excluding a library doesn't mean that the other libs don't depend on it. linuxdeployqt doesn't remove that dependency, of course. It just means this specific library is expected to be on the system, and therefore is not bundled.

In this case, you didn't have the library, but some other lib in the AppImage depended on it. Of course it starts working when you install the library system wide.

This is how things are supposed to work. Nothing to fix here.

[mardy]

Well, ignoring libraries which we assume to be present on the system could be an optimization. But I do agree that it's a limited use-case (I was building in Docker), so let's close this.

[TheAssassin]

The fact you were building in Docker has nothing to do with this.

[mardy]

It has a lot to do with this, because I didn't care about running my application, so I didn't need all dependencies to be installed.

[TheAssassin]

I think you missed a few points.

The exclude list is about excluding libraries which would normally be bundled (as in, they're listed by ldd). It doesn't make a difference if the library is installed or not on your system; it will be discarded anyway.

Excluding however doesn't remove that depenency. The end result is that if the lib would be loaded by the linker, it couldn't be found in the AppImage; therefore it needs to be provided system wise.

Whether stuff is in the excludelist or is excluded by hand with the parameter listed doesn't make a difference either.

required by the qt xcb plugin) not being found

If you wanted to fix that, you would've needed to bundle that library, not exclude it.

[probonopd]

linuxdeployqt cannot remove dependencies, it can only decide whether a dependency can be considered to be "there" on all target systems (distributions), and hence does not need to be bundled privately with the application (e.g., glibc), or whether it cannot be considered to be "there" on all target systems, and hence must be bundled (e.g., Qt).

We are collecting information about what we can consider to be "there" on all target systems on https://github.com/AppImage/pkg2appimage/blob/master/excludelist, and we are updating this list over time. The linuxdeployqt build process in this repository should fetch, when it is being built, the most recent version of this information, and it gets compiled into linuxdeployqt, so that linuxdeployqt can decide (when it is run) whether a particular library needs to be bundled with the application or not.

Instead of manually overriding the excludelist for a particular application, excludelist changes should be discussed in the project and applied to the global excludelist, so that everyone can benefit.

I hope this clarifies it.

[mardy]

I think you missed a few points.

Not really, I'm familiar with how the excludelist works and the things you wrote below. Maybe I didn't express myself clearly enough:

The exclude list is about excluding libraries which would normally be bundled (as in, they're listed by ldd). It doesn't make a difference if the library is installed or not on your system; it will be discarded anyway.

That's exactly my point: if we are going to discard it anyway, why complain if the library cannot be found?

required by the qt xcb plugin) not being found

If you wanted to fix that, you would've needed to bundle that library, not exclude it.

No, because the library is libEGL.so, which we don't want to ship. The plugin requiring it is an optional one, which I believe does not even get loaded during the normal execution of the application, though that does not matter much.

The reason why I brought in Docker is because, when developing on one's own desktop (without containers), one typically has all the optionally dependencies installed. I do have libEGL.so.1 in my system. But in Docker I'm using a minimal image, which doesn't have libEGL.so and other optional libraries required by the Qt plugins; that's why it would have been convenient for me if linuxdeployqt would ignore the missing libraries, if they are in the excludelist anyway. On the other hand, I do admit it's a limited use case, and installing one more library in Docker to stop linuxdeployqt from complaining is not a big issue, so I'm fine with the current implementation.

[probonopd]

Let me rephrase: If a library is on the excludelist, then this library should not cause a fatal error of linuxdeployqt even if the payload application requires it but it is missing on the build system (and hence cannot be found by ldd).

Is this a valid summary of your point?

[mardy]

Yes it is! But now I realize that it's actually a very minor issue, unlikely to have a real impact on most users. :-)

[TheAssassin]

Erm... so, what complains here is linuxdeployqt? Your statement sounded like it was a loading issue with the final AppImage. Yes, that's a bug that should be fixed.