Unable to build for Android (even on NDK r23b)

[sjevtic]

I have been having considerable problems compiling for Android. In particular, the error from #9 is familiar:

../src/fileencryptor/../cryptbase/../common_be.h:145:10: error: no matching function for call to 'bytesToHexString'

@bepaald concluded here:

Apparently, clang is the problem (on MacOS, it works fine on Linux).

Following a deeper investigation, I concluded that the problem is not actually clang itself, but rather libc++, which is LLVM's implementation of the C++ standard library. The problem is fairly uncommon on the Linux desktop, where GNU libstdc++ is still the most commonly used C++ standard library, even when using clang to compile. On Ubuntu 20.04 for example, builds with clang are by default still done by including headers and linking against libraries from GNU libstdc++.

The exact same issue was opened against the NDK itself. It is a longstanding issue (2.5 years) with no resolution, apparently in large part because the problem has not been fully addressed within upstream libc++. However, an astute comment notes that there is an open but still unmerged review for a fix.

To be clear, this is not a defect in signalbackup-tools. Rather, libc++ has a defect that substantially breaks its C++ 17 compatibility. However, given the current (and increasing) prevalence of the LLVM toolchain, I believe that compatibility with LLVM's libc++ remains a worthy goal, especially in the case of Android where Google has been steadily removing support for the GNU tools for years already. Unlike the workaround discussed in #9 for the macOS use case, the use of gcc and libstdc++ is not a practical option for Android, thus motivating me to open this new issue.

Further examination of the fix reveals that it is effectively a header-only change, so it can be easily deployed without either a standard library rebuild or even a system configuration change. In particular what I did was create a new subdirectory in my signalbackup-tools workspace ("sysinclude") into which I placed the new header. Then I prepended the subdirectory to the system include search path with a clang command line option: -isystem./sysinclude". The matter cannot be readily addressed with a source change since in some cases the included system headers include <memory> ahead of the explicit inclusions of the header directly by the signalbackup-tools source itself.

As an alternative approach which would likely lead to increased compatibility on older platforms, signalbackup-tools could be reworked such that it targets an older C++ standard (e.g., C++ 14) for which support is more mature across a larger range of environments.

It is also worth noting that in this environment, setlocale() is in the global namespace rather than the "std" namespace. I addressed this naively in sqlitedb/widestring.cc with a preprocessor definition:

#ifdef NDK_LOCALE_WORKAROUND
  setlocale(LC_ALL, "en_US.utf8");
#else
  std::setlocale(LC_ALL, "en_US.utf8");
#endif

The change may then be activated as needed as needed with a clang command line option: -DNDK_LOCALE_WORKAROUND.

Note that earlier NDK versions cause even more problems: of particular note, std::filesystem support needed by signalbackup-tools was not supported bu the bundled libc++ shipped with NDK versions prior to r22.

Of course, at the end of the day, the user attempting to build signalbackup-tools for his/her preferred platform should not need to be concerned with such details; administration of such particulars should be handled with Autoconf/Makefile or a similar technology; this has already been discussed in both #11 and #13.

[bepaald]

Thanks for your thorough investigation, it's very informative.

To be clear, this is not a defect in signalbackup-tools. Rather, libc++ has a defect

That's good to hear.

I'm not sure if you wrote this just to help others to compile on Android, or if you actually want me to implement something. Should I ship that entire custom llvm header in my project? And then learn autoconf to add that special workaround option to clang builds? Not sure I'm willing to do that. At the end of the day, software has minimal requirements, I'd say one of the requirements for this program is a C++17 compliant library, which apparently LLVM's is not. Not much I can do about that, I'm also not going to support openssl 0.9.1 or sqlite 2.0.

As an alternative approach which would likely lead to increased compatibility on older platforms, signalbackup-tools could be reworked such that it targets an older C++ standard (e.g., C++ 14) for which support is more mature across a larger range of environments.

I agree it would be nice to support older standards, but that is absolutely not going to happen. I would if it were easy, but I'm not sure it is even possible at all to support C++14 without a full rewrite (for which I do not have the time). I will accept patches for this though, I suggest you start with the ProtobufParser class ;)

It is also worth noting that in this environment, setlocale() is in the global namespace rather than the "std" namespace.

Thanks for pointing this out. That at least was an easy fix, as the widestring function was actually just an old leftover and was never called in the program at all. So I just removed it.

Of course, at the end of the day, the user attempting to build signalbackup-tools for his/her preferred platform should not need to be concerned with such details

I'd say, at the end of the day, the developer writing in his/her preferred language need not be concerned with such details either. I write perfectly valid, standards-compliant code. It is up to LLVM handle this.

I admit, some of this goes over my head, so if there's anything I can easily do to help address this issue, please let me know. But I'd rather not learn new languages or complicated tools and increase maintenance too much, all just to work around other projects' bugs.

[sjevtic]

I'm not sure if you wrote this just to help others to compile on Android, or if you actually want me to implement something. Should I ship that entire custom llvm header in my project? And then learn autoconf to add that special workaround option to clang builds?

My goal is actually both. First, my approach provides a useful workaround for anyone using LLVM's libc++, notably including all modern Android users and apparently those on macOS. It is not elegant, but it provides substantial utility in that it achieves full functionality without requiring modification to the system environment, and thus avoids impact to other packages the user may build or run.

At the end of the day, software has minimal requirements, I'd say one of the requirements for this program is a C++17 compliant library, which apparently LLVM's is not.

I would argue the requirement is actually not so minimal, and the proof is right here: one of the most prevalent modern standard libraries does not even support this functionality correctly yet.

I agree it would be nice to support older standards, but that is absolutely not going to happen. I would if it were easy, but I'm not sure it is even possible at all to support C++14 without a full rewrite (for which I do not have the time). I will accept patches for this though, I suggest you start with the ProtobufParser class ;)

I will not claim for a second to have an aptitude for C++ comparable to yours: ProtobufParser gives me headaches. :) I do not doubt this would be a nontrivial work package. It is definitely the more challenging of the two alternatives I presented.

I'd say, at the end of the day, the developer writing in his/her preferred language need not be concerned with such details either.

I could not disagree with you more on this point: as the developer, you are responsible for the user experience created by your software. In the case of a source distribution where you are not distributing binaries for common platforms, that also very much includes the experience of building the software. There are likely non-developers that would like to use your software but for which this barrier to entry is a non-starter. Moreover, a straightforward build process is also largely a prerequisite for your package to be picked up for binary distribution by open source platforms (think Ubuntu, or in the particular case I am looking at right now, Termux). Note that I am not the first to suggest an improved build system.

I write perfectly valid, standards-compliant code. It is up to LLVM handle this.

While this statement is factually true, it is of little practical value. No software, including tool chains and operating systems, is perfect, and even though LLVM is ultimately at fault here, the bottom line at the moment is that your software does not work in a variety of environments where it is desired. Most large projects, whether open source or commercial, are typically slow-moving, and this situation is a painfully obvious example of that. At the end of the day, in both software and broader life, the need for compromise to achieve the desired outcome is a common theme.

Now, all of this said, this is your personal project, not some commercial software for which you have customers and leadership to which you must answer. You are certainly free to disregard my position/requests, and at the end of the day I will still be thankful for the best-in-class solution that you have developed and continue to maintain; you have done an absolutely tremendous job in a space where both Signal and others have come up very short. If this is simply a personal interest activity for which you are just generously sharing your work and not interested in your user base or widespread adoption of your solution, that is absolutely fine too.

[bepaald]

Ok, while I still agree with what I said, I guess I also reluctantly agree with what you're saying.

The proposed solutions however still don't seem right. I really don't think this project should ship a full (patched) implementation of a system header file (I would then also have to track changes in that header and keep the patch working). And, as mentioned earlier, a full c++14 rewrite is simply too much work.

I will however attempt a partial c++14 rewrite. That is, considering std::shared_ptr<array_type[]> seems to be the only c++17 extension that LLVM is not supporting, I thought I'd try to write around just that.

I have been trying to set up an environment where I run into this issue, but it has proven difficult. I've tried just setting up a LLVM toolchain on my machine, then on various VM's (including FreeBSD, which supposedly doesn't even ship gcc due to licensing issues), but the program always just compiles fine (with -stdlib=libc++ where necessary) and ldd shows it's linked to libc++ and not libstdc++. So that has led me back to a VM running MacOS where I do encounter the problem.

I will try to tackle this, one error at a time, but it will take a while. I'm going into a busy period at work, most other issues will get higher priority and the MacOS VM runs like sh*t which is not encouraging. Also, a quick grep shows I use a lot of these pointer types, so even this partial rewrite might simply turn out to be impossible.

I'll leave this open while I'm working on it. If you have any more insights, feel free to add them on.

Thanks!

[sjevtic]

The proposed solutions however still don't seem right. I really don't think this project should ship a full (patched) implementation of a system header file (I would then also have to track changes in that header and keep the patch working). And, as mentioned earlier, a full c++14 rewrite is simply too much work.

Both are bad. They are at best workarounds.

I will however attempt a partial c++14 rewrite. That is, considering std::shared_ptr<array_type[]> seems to be the only c++17 extension that LLVM is not supporting, I thought I'd try to write around just that.

That concurs with my assessment. I spent a bit of time on this but ended up with a mess that was substantial runtime issues. But it compiled and linked under libc++, which leads me to believe that if I had gotten the logic right (i.e., memory management) it would have worked.

the program always just compiles fine (with -stdlib=libc++ where necessary) and ldd shows it's linked to libc++ and not libstdc++. So that has led me back to a VM running MacOS where I do encounter the problem.

I am not surprised that it was able to link against libc++ just fine. The problem here lies exclusively in the libc++ headers, so what actually matters is what you are including at compile time rather than what you are linking against. I suspect that if you looked at one of your working builds where you are linking against libc++ the compiler is somehow pulling in the libstdc++ headers.

[WSLUser]

Considering C++20 is out with a new standard once again being developed and released this year, Microsoft shared it's STL code with libc++ to make them more equivalent of each other a couple years back now and most notably llvm most assuredly is C++17 compliant, if not C++20 compliant yet, there seems little reason that building for Andriod for the past few versions at least should be a problem. I personally would prefer to use this project directly on my phone with an apk file I can simply just install and then run this with a nice friendly GUI to help me switch off of Signal proper to use a fork which has both plaintext backup and more notably SMS support still available. You will find this project is going to be in very high demand because of the drop of SMS support in Signal and having an easy to use from phone option is what alot of folks are going to want and expect for this project.

[bepaald]

So, what's the problem? Are you asking whether building on Android is possible or just informing us it is? (I wouldn't be surprised if it's possible now, since I could only ever reproduce this in macOS where updates to llvm have solved the problem as well.)

So have you actually tried to compile this on Android? Nobody is keeping you from the code, you can download and build it on any platform you like (that meets the small requirements this program has).

with a nice friendly GUI

Now, that is asking for a whole new project. I am not picking that up, I am not an Android developer (really, I would not be able to do that if I wanted to) and I absolutely do not have the time.