linuxdeploy
Qt application deployment and openssl
My Qt (using the latest 5.13.2) based app is using SSL connectivity. I want to distribute it as AppImage working 'everywhere' so I build it on Ubuntu 16.04 LTS which by default is still using openssl 1.0.2. Qt 5.13 by default only uses openssl 1.1 - not big deal I build the latest openssl from source:
./config shared --prefix=/opt/openssl-1.1.1/ && make && sudo make install
and include in LD_LIBRARY_PATH
export LD_LIBRARY_PATH="/opt/openssl-1.1.1/lib/:$LD_LIBRARY_PATH"
Now I can build my app and all works fine on building machine.
However when I package it with linuxdeploy+linuxdeploy-plugin-qt+linuxdeploy-plugin-appimage custom openssl is not packaged. So obviously when I try to run it on another system which does not have openssl 1.1 SSL part does not work.
I managed to find a rather crude but working way to package it myself but my question is why it is not taken care by linuxdeploy-plugin-qt ?
Bundling OpenSSL is (or has been) bloat in most cases before. Furthermore, it's a security problem to bundle any security related libraries. Therefore, OpenSSL is one of the libraries on the excludelist maintained by the AppImage team. The list includes libraries which are not safe to bundle or just don't have to be bundled.
Your case is very special. The tool and the excludelist usually orientate on what most users work with, and that's distro packages. Whenever we're in need of newer Qt packages, we use Stephan Binner's PPAs, which provide recent enough releases usually.
I managed to find a rather crude but working way to package it myself but my question is why it is not taken care by linuxdeploy-plugin-qt ?
You should've told about your way. I don't know what the heck crude is supposed to mean... o_O
As a workaround, you can force add your library with the -l option. Or you just copy it to AppDir/usr/lib before calling linuxdeploy. Then, it'll be deployed. However, this is untested, but please feel free to report back if you test those. I've head other people used -l with success to bundle OpenSSL.
Yes. I realised after playing with it that is not straight forward. But it is the key component (ssl) of any modern software. In ideal world it would be easy and clean solution but ideal world is a fairy tale. I made it working and will post details. Just few more tests to make sure I don’t make fool of myself:)
As explained in my firs post I would like to make sure that my application work on any supported system.
In case of Linux (I also release it for Windows and macOS but there all works) Qt/SSL pair turned out to be a bit problematic. Qt 5.13 I use is compiled against openssl 1.1 (1.1.1b to be precise but it does not matter much as all 1.1 are API/ABI compatible). The issue is that some LTS Linux distributions (e.g. CentOS 7, Ubuntu 16.04) are using openssl 1.0. Even that 1.0 will be dead next month those distros will keep using it back-porting required security fixes.
So I had to find a way to make my Qt app to work in situations when openssl 1.1 is not available - obvious way is to bundle required openssl libs into AppImage. But in other cases where openssl 1.1 is available I would prefer to use one provided by OS. I poked around Qt source code and realised that they are doing their magic to find required libs e.g.
https://github.com/qt/qtbase/blob/50deb8cf70f61e21fb0c35182341477af11adbc1/src/network/ssl/qsslsocket_openssl_symbols.cpp#L745-L765
// Try to find the libssl library on the system.
//
// Up until Qt 4.3, this only searched for the "ssl" library at version -1, that
// is, libssl.so on most Unix systems. However, the .so file isn't present in
// user installations because it's considered a development file.
//
// The right thing to do is to load the library at the major version we know how
// to work with: the SHLIB_VERSION_NUMBER version (macro defined in opensslv.h)
//
// However, OpenSSL is a well-known case of binary-compatibility breakage. To
// avoid such problems, many system integrators and Linux distributions change
// the soname of the binary, letting the full version number be the soname. So
// we'll find libssl.so.0.9.7, libssl.so.0.9.8, etc. in the system. For that
// reason, we will search a few common paths (see findAllLibSsl() above) in hopes
// we find one that works.
//
// It is important, however, to try the canonical name and the unversioned name
// without going through the loop. By not specifying a path, we let the system
// dlopen(3) function determine it for us. This will include any DT_RUNPATH or
// DT_RPATH tags on our library header as well as other system-specific search
// paths. See the man page for dlopen(3) on your system for more information.
It gave me an idea that I can put 'bundled' openssl somewhere late in Qt search path so if OS provided version is found Qt will use it and only go for AppImage included as a last resort. Good news was that current folder is searched as a last one so I can put libssl.so.1.1 and libcrypto.so.1.1 there (/AppDir/usr/bin/) before I create AppImage.
Now in case of Appimage current folder is not the same as folder where binary resides so it requires small cheat to help Qt to find libs. I had to put in my Qt app code "openssl initialisation" sequence. It has to to be done before any SSL functions are used. The obvious place is somewhere at the start of main.cpp
Proof of concept code:
// remember what current directory is
QString currentDir=QDir::currentPath();
//only for testing - show openssl version provided by OS
QProcess process; process.start("openssl version");
process.waitForFinished(-1);
QString stdout = process.readAllStandardOutput();
qDebug() << "OS openssl version: " << stdout;
// change current Dir to application directory path (it will point into AppImage filesystem usr/bin)
QDir::setCurrent(QCoreApplication::applicationDirPath());
// get SSL support status and runtime openssl version
// this forces Qt to find (initialise) openssl libs and use them for all later SSL operations
qDebug() << "SSL supported:" << QSslSocket::supportsSsl();
qDebug() << "Qt is using:" << QSslSocket::sslLibraryVersionString();
// restore current dir
QDir::setCurrent(currentDir);
it can be shortened in final release to:
#if defined(Q_OS_LINUX)
QString currentDir=QDir::currentPath();
QDir::setCurrent(QCoreApplication::applicationDirPath());
QSslSocket::supportsSsl();
QDir::setCurrent(currentDir);
#endif
On Linux distros where openssl 1.1 is present Qt finds it and use it. When not present its search get to current folder and uses bundled libraries
Example debug output for AppImage build on CentOS 7 (I switched to CentOS 7 from Ubuntu 16.04 as it has earlier glib so provides better compatibility coverage):
Debian 9 - app is using system openssl
OS openssl version: "OpenSSL 1.1.0l 10 Sep 2019\n"
SSL supported: true
Qt is using: "OpenSSL 1.1.0l 10 Sep 2019"
Ubuntu 16.04 - app is using bundled openssl
OS openssl version: "OpenSSL 1.0.2g 1 Mar 2016\n"
SSL supported: true
Qt is using: "OpenSSL 1.1.1d 10 Sep 2019"
OpenSuse Leap 15.1 - app is using system openssl
OS openssl version: "OpenSSL 1.1.0i-fips 14 Aug 2018\n"
SSL supported: true
Qt is using: "OpenSSL 1.1.0i-fips 14 Aug 2018"
Fedora 31 - app is using system openssl
OS openssl version: "OpenSSL 1.1.1d FIPS 10 Sep 2019\n"
SSL supported: true
Qt is using: "OpenSSL 1.1.1d FIPS 10 Sep 2019"
CentOS 7.7 - app is using bundled openssl
OS openssl version: "OpenSSL 1.0.2k-fips 26 Jan 2017\n"
SSL supported: true
Qt is using: "OpenSSL 1.1.1d 10 Sep 2019"
now SSL works on anything I run it on.
What I was thinking that it would be great if AppImage runtime could actually do this job. It could determine if openssl is present on host system and use bundled one only if requirement for specific openssl version is not met.
What I was thinking that it would be great if AppImage runtime could actually do this job. It could determine if openssl is present on host system and use bundled one only if requirement for specific openssl version is not met.
This could be done by preloading the right libssl library before running the app. The LD_PRELOAD environment var could be used for this matter. Yet preloading tend to have side effects and must be used with caution.
Your proposition requires code changes, @kapitainsky. @azubieta's doesn't, but it's pretty unsafe IMO, as he also points out. Any idea how to automate this kind of "fallback" from the Qt plugin's point of view?
@azubieta's method doesn't work anymore. Or maybe it's system-specific. What I did instead is append this to linuxdeploy-plugin-qt-hook.sh:
case "$(openssl version -v)" in
'OpenSSL 1.1.'*)
true
;;
*)
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH":"$(readlink -f "$(dirname "$0")")"/usr/openssl
;;
esac
To bundle OpenSSL in the appimage, I copied its libs into AppDir/usr/openssl.
If the host system's version is anything else than 1.1, the bundled version will be used. I believe the patch version doesn't matter when it comes to ABI compatibility, so it's only important to check major and minor version.
This seems to work fine. As a test, I bundled OpenSSL 1.1.1k in the appimage and ran it on a system that has the more recent version 1.1.1l and it says:
Qt is using: "OpenSSL 1.1.1l 24 Aug 2021"
I then ran it on Ubuntu 16.04 which ships 1.0.2g, and there the bundled version is used:
Qt is using: "OpenSSL 1.1.1k 25 Mar 2021"
However, a possible drawback here is that if the system has an older 1.1 version than the bundled one, the host's older version will be used. For example if you bundle 1.1.1l but the system has 1.1.1k, then 1.1.1k is used. I'm not sure if this is a real drawback though. The user of the host would be using the older version anyway even if they would build the application from source themselves rather than downloading an appimage of it. So I don't think it's worth implementing some smarter version detection.
This requires no modifications to the application's source code. However, you need to bundle an OpenSSL version that matches what linuxdeploy-plugin-qt-hook.sh checks for. So if you bundle OpenSSL 3.0 (not sure if Qt supports it yet, but it might in the future) then that check needs to change.
If anyone wants to adopt this into their project, what I do is save the above shell snippet into its own file appimage_openssl_hook in the project's root directory (same directory the .pro file is in.) Then I add the following at the end of my .pro file:
# Makefile target for AppImage creation (make appimage).
# Needs https://github.com/linuxdeploy and a binary called "linuxdeploy" (can be a symlink to the
# linuxdeploy AppImage.) Also needs the linuxdeploy "qt" and "appimage" plugins.
linux {
appimage.target = appimage
appimage.commands = \
rm -f QTads.AppImage \
&& rm -rf AppDir \
&& "$$QMAKE_QMAKE" PREFIX="$$OUT_PWD"/AppDir/usr -config release "$$_PRO_FILE_" \
&& make -j"$$QMAKE_HOST.cpu_count install" \
&& mkdir "$$OUT_PWD"/AppDir/usr/openssl \
&& cp -a "$(shell ldconfig -p | grep libssl.so.1 | head -n1 | tr ' ' '\n' | grep /)" "$$OUT_PWD"/AppDir/usr/openssl \
&& cp -a "$(shell ldconfig -p | grep libcrypto.so.1 | head -n1 | tr ' ' '\n' | grep /)" "$$OUT_PWD"/AppDir/usr/openssl \
&& rm -rf AppDir/usr/share/metainfo \
&& env PATH="$$(PATH)" QMAKE="$$QMAKE_QMAKE" linuxdeploy \
-v2 \
--appdir AppDir \
--plugin qt \
&& cat "$$_PRO_FILE_PWD_"/appimage_openssl_hook >> AppDir/apprun-hooks/linuxdeploy-plugin-qt-hook.sh \
&& env PATH="$$(PATH)" QMAKE="$$QMAKE_QMAKE" OUTPUT=QTads.AppImage linuxdeploy \
-v2 \
--appdir AppDir \
--output appimage
QMAKE_EXTRA_TARGETS += appimage
}
This allows me to build an appimage with just qmake && make appimage. For this to work, your linuxdeploy-x86_64.AppImage (or linuxdeploy-i386.AppImage) needs to be called simply linuxdeploy. I just use a symlink. If you don't like that, you can just change the above qmake snippet obviously.
"QTads" is just the name of my project. You should change that as well.
Thank @kapitainsky. I am sticking with OpenSSL loading thing in Qt too, but your comment helps me a lot.
