Android Buildscript problem with Cross Compilation

[Hatzen]

trafficstars

I try to setup a CI pipeline for compiling liboqs with liboqs-java for android.
But with the current master version (and release 0.6.0) of this repo it is not possible to run the android build script (https://github.com/open-quantum-safe/liboqs/blob/main/scripts/build-android.sh)

i run in a similar issue like compiling for windows. But cannot use wine for this problem afaik.
https://github.com/open-quantum-safe/liboqs/issues/727

When i run
bash build-android.sh $ndkPath -s 19-a x86_64

1. it first runs into the problem that "try_run" is not working as the target architecture is not the same.
   https://github.com/open-quantum-safe/liboqs/blob/2db7f97922a7d3e50aa9f56b7e86566f62c0365a/.CMake/gcc_clang_intrinsics.cmake#L3
   Excluding that file let the compile process start.

2. But then it crashes afterwards for the common c files

common.c:169:9: error: incompatible integer to pointer conversion returning 'int' from a function with result type 'void *' [-Werror,-Wint-conversion]
        return aligned_alloc(alignment, size);

click for full build log

-- Build files have been written to: /home/hatzen/LibOQSTestApp--12-add-ci-for-prebuild-liboqs/liboqs-android/prebuild-liboqs/liboqs/build
Scanning dependencies of target falcon_512_clean
[  0%] Building C object src/sig/falcon/CMakeFiles/falcon_512_clean.dir/sig_falcon_512.c.o
[  0%] Building C object src/sig/falcon/CMakeFiles/falcon_512_clean.dir/pqclean_falcon-512_clean/codec.c.o
[  0%] Building C object src/sig/falcon/CMakeFiles/falcon_512_clean.dir/pqclean_falcon-512_clean/common.c.o
[  1%] Building C object src/sig/falcon/CMakeFiles/falcon_512_clean.dir/pqclean_falcon-512_clean/fft.c.o
[  1%] Building C object src/sig/falcon/CMakeFiles/falcon_512_clean.dir/pqclean_falcon-512_clean/fpr.c.o
[  1%] Building C object src/sig/falcon/CMakeFiles/falcon_512_clean.dir/pqclean_falcon-512_clean/keygen.c.o
[  1%] Building C object src/sig/falcon/CMakeFiles/falcon_512_clean.dir/pqclean_falcon-512_clean/pqclean.c.o
[  1%] Building C object src/sig/falcon/CMakeFiles/falcon_512_clean.dir/pqclean_falcon-512_clean/rng.c.o
[  1%] Building C object src/sig/falcon/CMakeFiles/falcon_512_clean.dir/pqclean_falcon-512_clean/sign.c.o
[  1%] Building C object src/sig/falcon/CMakeFiles/falcon_512_clean.dir/pqclean_falcon-512_clean/vrfy.c.o
[  1%] Built target falcon_512_clean
Scanning dependencies of target common
[  1%] Building C object src/common/CMakeFiles/common.dir/aes/aes.c.o
[  1%] Building C object src/common/CMakeFiles/common.dir/aes/aes_c.c.o
[  1%] Building C object src/common/CMakeFiles/common.dir/sha2/sha2_c.c.o
[  1%] Building C object src/common/CMakeFiles/common.dir/sha3/xkcp_sha3.c.o
[  1%] Building C object src/common/CMakeFiles/common.dir/sha3/xkcp_sha3x4.c.o
[  2%] Building C object src/common/CMakeFiles/common.dir/common.c.o
/home/hatzen/LibOQSTestApp--12-add-ci-for-prebuild-liboqs/liboqs-android/prebuild-liboqs/liboqs/src/common/common.c:169:9: error: implicit declaration of function 'aligned_alloc' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        return aligned_alloc(alignment, size);
               ^
/home/hatzen/LibOQSTestApp--12-add-ci-for-prebuild-liboqs/liboqs-android/prebuild-liboqs/liboqs/src/common/common.c:169:9: error: incompatible integer to pointer conversion returning 'int' from a function with result type 'void *' [-Werror,-Wint-conversion]
        return aligned_alloc(alignment, size);
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2 errors generated.
src/common/CMakeFiles/common.dir/build.make:182: recipe for target 'src/common/CMakeFiles/common.dir/common.c.o' failed
make[2]: *** [src/common/CMakeFiles/common.dir/common.c.o] Error 1
CMakeFiles/Makefile2:260: recipe for target 'src/common/CMakeFiles/common.dir/all' failed
make[1]: *** [src/common/CMakeFiles/common.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

As i would like to make a CI pipe for my android integration here it would be a charm if we can make the android build script work without the need of changing anything in the cloned files.
So i would suggest

1. For the first problem to pass a property which disables "try_run"
2. and for the second problem i need help as i am not very familiar with the code or error. So can anyone provide a suggestion or solution?

Thank you

[baentsch]

Thanks for your interest in this project and apologies in advance for possibly silly questions below as I have 0 experience with Android and definitely am no cross-building expert, but I wonder:

without the need of changing anything in the cloned files.

Why do you have this goal? If something does not work as intended, it should be corrected, no?

For the first problem to pass a property which disables "try_run"

AFAIK running this file is essential to inform the whole build process (setting correct compiler flags for the platform), so it can't simply be dropped but either should be replaced by an alternative mechanism that works just for Android (in the worst case disabling all CPU feature flags/performance optimizations for Android) and the platform(s) you want to support OR one might follow the suggestions in the files created by cmake:

# This file was generated by CMake because it detected TRY_RUN() commands
# in crosscompiling mode. It will be overwritten by the next CMake run.
# Copy it to a safe location, set the variables to appropriate values
# and use it then to preset the CMake cache (using -C).


# RUN_RESULT
#    indicates whether the executable would have been able to run on its
#    target platform. If so, set RUN_RESULT to
#    the exit code (in many cases 0 for success), otherwise enter "FAILED_TO_RUN".
# RUN_RESULT__TRYRUN_OUTPUT
#    contains the text the executable would have printed on stdout and stderr.
#    If the executable would not have been able to run, set RUN_RESULT__TRYRUN_OUTPUT empty.
#    Otherwise check if the output is evaluated by the calling CMake code. If so,
#    check what the source file would have printed when called with the given arguments.
# The COMPILE_RESULT variable holds the build result for this TRY_RUN().

--> As you obviously have experience with Android, wouldn't this be something you might be willing to do (and contribute back)? Either on real hardware or a suitable emulator? Also, only you know which architectures you want to run the resultant binaries on. What little I know about Android is that there is a wide variety of chips supporting Android, so the CPU feature flags that are supported (and that the file .CMake/detect_gcc_clang_intrinsics.c outputs) will probably be different from platform to platform (i.e., the -a and -s parameters you are passing to the build command). Long story short, I have the gut feeling that a generic solution would encompass running emulators executing this file on the requested architectures.

Regarding the second question, the issue ultimately seems to come down to another Android issue:

implicit declaration of function 'aligned_alloc' is invalid in C99

A quick google search returned others dealing with the same problem. Thus, my question would be whether you could envision using your Android knowledge, looking into this and contributing a suitable change to our project that rectifies this for Android?

Finally, I'm tagging @nicholasfulton who contributed this code quite some time ago asking for his suggestions (if he has time) on how to resolve this issue.

[Hatzen]

Thanks for the fast, good and detailed answer.

1. i would like to contribute and will have a look at the compile and test process android. I cant promise anything, especially witihn my repository i would already test liboqs on android completly. But at the very end.

2. implicit declaration of function 'aligned_alloc' is invalid in C99 I did Not thought about it being a specific Android Problem i will have a closer look at this.

TL;DR without the need of changing anything in the cloned files. Why do you have this goal? If something does not work as intended, it should be corrected, no? Thats why i created an issue. I Just didnt want to fix the Problem Just in my Android repo and copying 90% of the Code :) I hopefully can provide some parts of the solution.

---

EDIT:
I tried to investigate on 2 a bit but i can not find the reason why std=C99 is set and i could not overwrite this setting in any way.
I just noticed that changing the SDK version to greater 27 seems to resolve this problem, but leading to failing compilation with a similar problem on random.c
And 32 Bit arm systems crash related to space issues with longs. And 32-bit x86 system does not support processor i686.
This looks to multiple problems and with my lack of knowledge i am not sure whether i can solve these problems in the next few months.

[dstebila]

Wondering if @greebie has any insight on to this?

[greebie]

I definitely dropped any hope of being able to work with 32 bit architectures in my use case/prototype (using Mac with Flutter via https://www.github.com/greebie/liboqs-flutter). I thought it was a Mac thing (and even more especially true now that I moved to M1). Unfortunately, I have not used the java version of liboqs, other than to help (or not help) via Stack overflow.

For testing within Flutter, I had to use a hack that is not particularly fantastic from a safety point of view. I recorded it here: https://stackoverflow.com/questions/66641396/testing-library-functions-in-a-flutter-package-with-native-functions-using-dart/66722572#66722572.

It may be worth exploring a fork tailored for ingesting the c code directly into mobile via an ffi library. I had more luck in both iOS and Android just adding the C folders and compiling liboqs into the apps during run time. I got the .so working in Android, but not iOS. It would probably work for 32-bit in a linux environment as well.

[geovandro]

I was able to fork @Hatzen's original testing project and get it to compile and test libOQS for the 4 major Android ABIs: x86, x86_64, armeabi-v7a, and arm64-v8a. There was an insufficient stack issue with Classic McEliece that also needed to be solved. The project has a CI workflow with tests on emulators (had to disable the "s-robust" variants of Sphincs+ since they would almost double the workflow time for ARMv8 emulation, but tests pass locally and instructions to reproduce are given in the README). Project: https://github.com/geovandro/PQCryptoOnAndroid

[baentsch]

@geovandro Thanks for letting us know about it. If you feel "like it", we'd very much welcome a PR to add build-and-test (CI) for android in this project. Alternatively, if you prefer to maintain it at https://github.com/geovandro/PQCryptoOnAndroid (?) shall we add a link to your repo at https://github.com/open-quantum-safe/liboqs/wiki/Platform-specific-notes-for-building-liboqs under heading "Android"?

[geovandro]

I could prepare a CI file that would checkout https://github.com/geovandro/PQCryptoOnAndroid and the current libOQS source (this is default since it would run from the libOQS repo), crosscompile libOQS to generate the .so files, and finally use them for the Android testing part. How does this sound? I think you can also point to the independent repo for Android instructions (at https://github.com/open-quantum-safe/liboqs/wiki/Platform-specific-notes-for-building-liboqs). I will expand the README file to give more detailed instructions.

[SWilson4]

Closing this as resolved via #1531.