Add options for iOS and Android cross-compilation

[nicholasfulton]

I'm attempting to build liboqs for both iOS and Android for a Xamarin.Forms project, and haven't been able to manipulate the included build and configuration files to make that work. If it would be possible, add configuration or documentation to help with cross-compiling the library for iOS and Android, ideally targeting multiple processor configurations. Thanks!

[dstebila]

Hi Nicholas, a while ago someone created a cross-compilation script for Android (https://github.com/open-quantum-safe/liboqs/blob/master/configure-android) but it is not something we actively maintain since we don't have anyone in the project who uses it nor do we have any continuous integration testing set up for Android. We don't have much expertise in cross-compilation. If you are able to get it working, please let us know what you've done so we can incorporate it.

[nicholasfulton]

I believe I've made good progress with developing a build script for Android. However, developing a more general solution runs into an issue with building using Clang. It seems that the build scripts specifically require GCC and disallow Clang. That's problematic because Clang is necessary to cross-compile for iOS and Android, as GCC doesn't include toolchains for those platforms. Is there a reason that liboqs requires GCC? Otherwise, I'll work on a more elegant way to allow building with Clang as well.

EDIT: It appears the existing "configure-android" script is built for a previous version of the Android ndk which supported GCC, and will no longer work because of that.

[dstebila]

Hi @nicholasfulton, in general liboqs should compile just fine with clang -- our macOS continuous integration builds use the default compiler on macOS, which is clang.

The now-unmaintained configure-android script does explicitly refer to gcc, and that was just because that was what the person putting that script together was able to get working, not intentionally made to restrict use to gcc-only.

[zugaldia]

I've got basic Java bindings working, including Android, on https://github.com/zugaldia/javacpp-presets. It uses the JavaCPP infrastructure to build the bindings. A few notes for those willing to give it a try:

• The Linux bindings support the full API.
• The Android bindings only support a subset of the API and are much trickier to build. They should be considered a WIP:
  • This patch enables cross-compilation.
  • NDK doesn't come with OpenSSL, I'm using the binaries provided by Teska Labs.
  • Unlike configure-android discussed above which uses gcc, I've only tested the latest NDK version with clang.
  • sig-picnic needs to be disabled to avoid a compilation error.
• Build instructions, including an example using qTesla, is available in the README.md.

It's interesting that, while the Linux bindings build rather smoothly, Android (via NDK) throws multiple compilations warnings and errors for some header files. If someone more familiar with the project wants to check out the setup, this is the main script to review.

[xvzcf]

We are in the process of moving the build system to CMake and dropping support for autotools (PR https://github.com/open-quantum-safe/liboqs/pull/556).

@zugaldia would it be possible to update cppbuild.sh to use CMake?

[zugaldia]

@xvzcf yup - that should not be a problem, in fact other JavaCPP Presents projects (example) use CMake. Feel free to ping me here once that switch is ready to be tested and I'll happily update cppbuild.sh.

[dstebila]

Should be okay to start looking at the iOS and Android cross-compilation again since our change to CMake has landed. FYI I've started a PR for cross-compilation on Raspberry Pi #618.

[nicholasfulton]

I believe I have a fully functioning Android build script, which I've tested on the Windows Subsystem for Linux with Ubuntu running. I was able to compile for all major ABI's with no issue. In the 'add-cpu-extensions-flag.cmake' file, I had to comment out the 'try_run' command in order to compile for Android x86 architectures. It seems that it was trying to run tests without awareness that the build machine and the host machine are completely different. My app is developed in Xamarin Forms, so I'm working through some issues with the shared library not being found. I'll update when I know whether the libraries produced by the script actually function. Also, once I finish that, I'll work on the iOS build. CMake definitely made the Android compilation way more straightforward, and it seems it'll do the same for iOS.

[dstebila]

Thanks for the update @nicholasfulton!

[Hatzen]

What will the next steps look like? Will this issue be handled as is or are there plans to make a demo App or at some documentation for creating an aar (android library)? I m asking as i am looking for using this awesome library collection for my android app, but i am not very familiar with using native libs. So i would like to avoid spending time with getting into it :)
The timetable looks like 17th April will be a release?

[nicholasfulton]

At the moment, I'm working on setting up a CI/CD container for the Android compilation, and my next step is to develop a working build script for iOS (the quarantine has messed with my access to a Mac which was essential to that). The Android build script should be fully function at this time, but it only works on Linux (I used Ubuntu Windows Subsystem to build and test). To use it you'll need to run './build-android.sh --help' (from the scripts directory) and input the required options. The root directory of the Android NDK is required and best practice is to build for all 4 supported Android ABI's, which can be any of "armeabi-v7a," "arm64-v8a," "x86," or "x86_64." Optionally, you can specify an sdk version as well. If you have any further questions about how the compilation works, feel free to ask (once I get CI/CD working, I'll definitely need to make it a priority to include the build instructions in README).

[Hatzen]

@nicholasfulton thanks for your fast response. The last days i tried it a few times but could not get it working. Probably it is my bad but i cant get any useful informations from the logs or readme files. I tried around using different versions and installing different packages without any effect. Do you have any thoughts what i am doing wrong?

Detailed setup of ubuntu subsystem and error logs

Using a clean installed ubuntu 18.04 LTS on Windows subsystem

pwd
>/home/hatzen
sudo su
apt-get update   
mkdir liboqs    
cd liboqs
git clone https://github.com/open-quantum-safe/liboqs.git   
mkdir ndk      
cd ndk
wget https://dl.google.com/android/repository/android-ndk-r21-linux-x86_64.zip 
apt-get install unzip  
unzip android-ndk-r21-linux-x86_64.zip
apt-get install -y build-essential cmake libssl-dev    
cd /home/hatzen/liboqs/liboqs/scripts     
bash build-android.sh /home/hatzen/liboqs/ndk/android-ndk-r21
    Valid directory for NDK at /home/hatzen/liboqs/ndk/android-ndk-r21
    Compiling for ABI armeabi-v7a
    Compiling for SDK 21
    Cleaning up previous build
    Building in directory build
    -- The ASM compiler identification is Clang
    -- Found assembler: /home/hatzen/liboqs/ndk/android-ndk-r21/toolchains/llvm/prebuilt/linux-x86_64/bin/clang
    -- Check for working C compiler: /home/hatzen/liboqs/ndk/android-ndk-r21/toolchains/llvm/prebuilt/linux-x86_64/bin/clang
    -- Check for working C compiler: /home/hatzen/liboqs/ndk/android-ndk-r21/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -- works
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - done
    -- Detecting C compile features
    -- Detecting C compile features - done
    -- Looking for pthread.h
    -- Looking for pthread.h - found
    -- Looking for pthread_create
    -- Looking for pthread_create - found
    -- Found Threads: TRUE
    CMake Error at .CMake/gcc_clang_intrinsics.cmake:6 (message):
      .CMake/detect_gcc_clang_intrinsics.c returned exit code:
    Call Stack (most recent call first):
      .CMake/compiler_opts.cmake:13 (include)
      CMakeLists.txt:40 (include)
    
    
    -- Configuring incomplete, errors occurred!
    See also "/home/hatzen/liboqs/liboqs/build/CMakeFiles/CMakeOutput.log".

A Copy of /home/hatzen/liboqs/liboqs/build/CMakeFiles/CMakeOutput.log CMakeOutput.log

Maybe other useful version informations

cmake --version
cmake version 3.10.2
openssl version
OpenSSL 1.1.1  11 Sep 2018
gcc --version
gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

[xvzcf]

The error is due to this commit introduced by me. I can fix it in an upcoming PR, or @nicholasfulton perhaps you could fix it as part of your work on setting up a CI workflow for Android?

[greebie]

I have successfully built portions of the C code from liboqs into a Flutter/Dart project (which is currently private, but will be open for viewing once I have a reasonable MVP). More than happy to help prevent someone else from pulling their hair out based on my learning.

Some challenges I found for cross-compilation via the Flutter framework:

• iOS requires that added C or C++ code be signed using XCode.
• Documentation for inclusion of .so and .a binaries is slim and I have not been able to get it to work.
• Documentation about how Android and iOS handle CMakeLists.txt in subdirectories is not clear.
• Compiling the source code is possible, but it requires some sideways hacking which would be hard to maintain for a project where new algos appear, disappear and re-appear over time.

[nicholasfulton]

I'll continue researching CI/CD integration for Android and iOS. I started but got sidetracked and quit working on it for a while. It does seem like the biggest obstacle is some algorithms' issues on ARM. I'll look into the errors, because it'd be ideal to be able to build the full set of algorithms for ARM instead of needing to skip specific ones.

[dstebila]

Just checking in to see if there are any updates to this work.

[nicholasfulton]

I think I'm pretty close to a Docker container for CI/CD on Android, working with the configuration for some other build profiles/targets.

[dstebila]

Thanks for the update Nicholas. Looking forward to it!

[nicholasfulton]

Sorry for the delay. I created a draft pull request (#875) which I believe should test building the library. I'm not sure of format or whether it works correctly with the CircleCI workflows. I will be looking into it more, but I published the draft PR for potential feedback.

[SWilson4]

Given that we now have CI builds for both Android and iOS, I think this issue can be safely closed. @dstebila @baentsch Feel free to reopen if you have objections.