Cross compilation

[meow464]

Let's see if the tests pass.

My cross-compilation branch.

For now you need to do some manual copying and pasting, the next pull request will enable automatic routing based on the triple. I tested this on x86_64 -> aarch64, macOS and Linux. If you don't have an aarch64 machine you can use qemu+linux to test the binaries. The instructions are relatively simple:

First compile lfortran as per the docs:

./build0.sh
mkdir build && cd build

cmake ..\
      -DCMAKE_BUILD_TYPE=Debug\
      -DWITH_LLVM=YES\
      -DWITH_TARGET_AARCH64=Yes\
      -DCMAKE_INSTALL_PREFIX="$(pwd)/../install"

make -j12
make install

Then cross compile the runtime library. You will need a cmake toolchain file for the target architecture, I used aarch64-linux and iOS.

On Linux will also need to install gcc and g++ for aarch64, I use debian and installed them with apt: sudo apt install gcc make gcc-aarch64-linux-gnu binutils-aarch64-linux-gnu g++-aarch64-linux-gnu

mkdir build_aarch64 && cd build_aarch6

cmake ..\
      -DCMAKE_TOOLCHAIN_FILE=./aarch64-linux-gnu.toolchain.cmake\
      -DCMAKE_BUILD_TYPE=Debug\
      -DWITH_LLVM=yes\
      -DCMAKE_INSTALL_PREFIX="$(pwd)/../install_aarch64"\
      -DCMAKE_Fortran_COMPILER="$(pwd)/../install/bin/lfortran"\
      -DWITH_ZLIB=NO\
      -DWITH_RUNTIME_LIBRARY_ONLY=YES

make -j12
make install

Now replace the old runtime with the cross compiled one:

mv install/share install/share-x86_64
cp -rf install_aarch64/share install/

You can test it with a hello world or this simple program.

On Linux use LFORTRAN_CC=/usr/bin/aarch64-linux-gnu-gcc lfortran --static --link-with-gcc --target arm64-linux. You don't need to use --static but then you will have to copy liblfortran_runtime.so to the appropriate location on the target system.

On macOS just call your new lfortran with --target aarch64-apple-ios. To actually use the binary you would need to copy liblfortran_runtime_static.a to your XCode project, but it won't work because we still can't generate libraries. As things are it's probably possible to cross compile between aarch64 <-> x86_64 macOS but I don't have the hardware to test.

Note:

1. The change on lfortran.cpp is because gcc doesn't actually accept the --target option, I thought it did because on macOS by default gcc is just a front for clang;
2. I'm not very familiar with windows cross compilation and I don't have an aarch64 windows machine. I think cross compiling from linux to windows is possible but I'm not sure;
3. I'm not very familiar with WASM;
4. Is ZLIB necessary for the runtime? For now I'm keeping it off but it's just a matter of installing it on the sysroot or passing linker parameters -L.
5. The next pull requests will add share/lfortran-aarch64-linux and the likes and automatically select the runtime based on --target, better organize the CMakeLists.txt and move the libraries to lib/lfortran instead of share/lfortran.
6. I will rebase this before merge.

[meow464]

I built this multiple times on Linux and macOS, in case anyone wants to test. I will fix the CI problems this weekend.

[certik]

Is this how to cross-compile the LFortran compiler itself, that is, compile on Linux/Intel and run on macOS M1?

But it seems you are also using LFortran on linux to make LFortran create/compile binaries for macOS M1?

We want both, but I am a little confused which one this PR gets working.

[meow464]

This is only for cross compiling the runtime and turning lfotran into a cross compiler. My initial goal was to cross compile for iOS and Android where a compiler doesn't make much sense.

Cross compiling the compiler itself could come later, I suspect it won't take much besides specifying a toolchain file. Cross compiling from linux to macOS is not impossible but would require a few extra steps.

[certik]

Got it! Thanks for the clarification. Yes, it turns lfortran into a cross-compiler. We want that for sure.

[meow464]

I'm accepting suggestions of what could be going wrong here. I ran ci/build_tmp.xsh locally on an conda env with the deps from ci/environment_linux.yml and it works fine.

I'm using debian, the CI is using ubuntu but that shouldn't matter since it fails on windows and mac too.

[certik]

We would need to debug it.

Is there some subset of this PR that we can simply merge into master, and leave just the part that does not work to figure out later?

I would like to also setup a CI test that will run on, say, Linux Intel, and it will cross compile to arm. Even better if we could run it somehow, perhaps in Qemu. But even just compiling to arm, and just checking via "file" that the generated binary is an arm binary would be super useful, that way we would know that the cross compilation capability of LFortran works at least to the extent that it generates an arm binary (whether the binary works or not).

[meow464]

This PR is kinda monolithic, it just turns the runtime custom_command in src/bin/CMakeLists.txt into targets in src/runtime/CMakeLists.txt. Cmake doesn't know how to use the custom_commands for cross compilation.

I could make a different PR that adds automatic runtime selection based on --target but people would still need to build this manually.

In any case I will keep trying to replicate this.

I would like to also setup a CI test that will run on, say, Linux Intel, and it will cross compile to arm. Even better if we could run it somehow, perhaps in Qemu.

I would like this too. On linux we can test x86_64 and arm64 both ways with qemu user space emulation or box64 or fex-emu whichever is easier. On macos there is rossetta. I don't know about windows.

I would like to setup iOS and android somehow at some point but I'm still exploring the idea.

[certik]

Perfect. Just setting up Linux at the CI would be enough to ensure our cmake and everything else keeps working for cross compilation.

[meow464]

The tests passed because I was using make but they fail with Ninja. I don't get quite the same error but it's around the same file. I used -j12 with make too so it shouldn't be a race condition. I will have to think a bit...

[93/132] Building Fortran preprocessed src/runtime/CMakeFiles/lfortran_intrinsic_builtin.dir/builtin/lfortran_intrinsic_builtin.f90-pp.f90
FAILED: src/runtime/CMakeFiles/lfortran_intrinsic_builtin.dir/builtin/lfortran_intrinsic_builtin.f90-pp.f90 src/runtime/CMakeFiles/lfortran_intrinsic_builtin.dir/builtin/lfortran_intrinsic_builtin.f90.o.ddi 
/home/tuco/Projects/LFortran/lfortran/build/src/runtime/../bin/lfortran -cpp -DHAVE_TARGET_X86=1 -DHAVE_WHEREAMI=1 -DHAVE_ZLIB=1  -g -Jsrc/runtime --backend=cpp -E /home/tuco/Projects/LFortran/lfortran/src/runtime/builtin/lfortran_intrinsic_builtin.f90 -o src/runtime/CMakeFiles/lfortran_intrinsic_builtin.dir/builtin/lfortran_intrinsic_builtin.f90-pp.f90 && /usr/bin/cmake -E cmake_ninja_depends --tdi=src/runtime/CMakeFiles/lfortran_intrinsic_builtin.dir/FortranDependInfo.json --lang=Fortran --pp=src/runtime/CMakeFiles/lfortran_intrinsic_builtin.dir/builtin/lfortran_intrinsic_builtin.f90-pp.f90 --dep=src/runtime/CMakeFiles/lfortran_intrinsic_builtin.dir/builtin/lfortran_intrinsic_builtin.f90-pp.f90.d --obj=src/runtime/CMakeFiles/lfortran_intrinsic_builtin.dir/builtin/lfortran_intrinsic_builtin.f90.o --ddi=src/runtime/CMakeFiles/lfortran_intrinsic_builtin.dir/builtin/lfortran_intrinsic_builtin.f90.o.ddi
The following argument was not expected: -pp
Run with --help for more information.
[94/132] Building Fortran preprocessed src/runtime/CMakeFiles/lfortran_intrinsic_kind.dir/pure/lfortran_intrinsic_kind.f90-pp.f90
FAILED: src/runtime/CMakeFiles/lfortran_intrinsic_kind.dir/pure/lfortran_intrinsic_kind.f90-pp.f90 src/runtime/CMakeFiles/lfortran_intrinsic_kind.dir/pure/lfortran_intrinsic_kind.f90.o.ddi 
/home/tuco/Projects/LFortran/lfortran/build/src/runtime/../bin/lfortran -cpp -DHAVE_TARGET_X86=1 -DHAVE_WHEREAMI=1 -DHAVE_ZLIB=1  -g -Jsrc/runtime --backend=cpp -E /home/tuco/Projects/LFortran/lfortran/src/runtime/pure/lfortran_intrinsic_kind.f90 -o src/runtime/CMakeFiles/lfortran_intrinsic_kind.dir/pure/lfortran_intrinsic_kind.f90-pp.f90 && /usr/bin/cmake -E cmake_ninja_depends --tdi=src/runtime/CMakeFiles/lfortran_intrinsic_kind.dir/FortranDependInfo.json --lang=Fortran --pp=src/runtime/CMakeFiles/lfortran_intrinsic_kind.dir/pure/lfortran_intrinsic_kind.f90-pp.f90 --dep=src/runtime/CMakeFiles/lfortran_intrinsic_kind.dir/pure/lfortran_intrinsic_kind.f90-pp.f90.d --obj=src/runtime/CMakeFiles/lfortran_intrinsic_kind.dir/pure/lfortran_intrinsic_kind.f90.o --ddi=src/runtime/CMakeFiles/lfortran_intrinsic_kind.dir/pure/lfortran_intrinsic_kind.f90.o.ddi
The following argument was not expected: -pp
Run with --help for more information.
[95/132] Building Fortran preprocessed src/runtime/CMakeFiles/lfortran_intrinsic_iso_fortran_env.dir/pure/lfortran_intrinsic_iso_fortran_env.f90-pp.f90
FAILED: src/runtime/CMakeFiles/lfortran_intrinsic_iso_fortran_env.dir/pure/lfortran_intrinsic_iso_fortran_env.f90-pp.f90 src/runtime/CMakeFiles/lfortran_intrinsic_iso_fortran_env.dir/pure/lfortran_intrinsic_iso_fortran_env.f90.o.ddi 
/home/tuco/Projects/LFortran/lfortran/build/src/runtime/../bin/lfortran -cpp -DHAVE_TARGET_X86=1 -DHAVE_WHEREAMI=1 -DHAVE_ZLIB=1  -g -Jsrc/runtime --backend=cpp -E /home/tuco/Projects/LFortran/lfortran/src/runtime/pure/lfortran_intrinsic_iso_fortran_env.f90 -o src/runtime/CMakeFiles/lfortran_intrinsic_iso_fortran_env.dir/pure/lfortran_intrinsic_iso_fortran_env.f90-pp.f90 && /usr/bin/cmake -E cmake_ninja_depends --tdi=src/runtime/CMakeFiles/lfortran_intrinsic_iso_fortran_env.dir/FortranDependInfo.json --lang=Fortran --pp=src/runtime/CMakeFiles/lfortran_intrinsic_iso_fortran_env.dir/pure/lfortran_intrinsic_iso_fortran_env.f90-pp.f90 --dep=src/runtime/CMakeFiles/lfortran_intrinsic_iso_fortran_env.dir/pure/lfortran_intrinsic_iso_fortran_env.f90-pp.f90.d --obj=src/runtime/CMakeFiles/lfortran_intrinsic_iso_fortran_env.dir/pure/lfortran_intrinsic_iso_fortran_env.f90.o --ddi=src/runtime/CMakeFiles/lfortran_intrinsic_iso_fortran_env.dir/pure/lfortran_intrinsic_iso_fortran_env.f90.o.ddi
The following argument was not expected: -pp
Run with --help for more information.
ninja: build stopped: subcommand failed.

[meow464]

There are two things that need to be fixed.

First one is easy: just tell Ninja not to call lfortran with preprocess syntax because it tries to use the gfortran syntax. The current custom_command doesn't do it. At some point in the future I will make a .cmake file so CMake knows how to call lfortran.

Second: Ninja actually uses the -J flag while make does cd dir && lfortran -J. so -J needs fixing. I will look into this today or tomorrow.

For reference: https://gitlab.com/lfortran/lfortran/-/issues/617

[certik]

Do you know how to patch cmake to call lfortran correctly? We quite urgently need it on Windows, where the integration_tests don't work at all, because on Windows cmake can't call lfortran, it fails with some internal cmake errors.

[meow464]

The problems I met could be fixed by implementing one of these files and fixing -J. I don't know much about Windows but I could install it on a VM and run the tests with --verbose because otherwise the errors tell you nothing (some internal cmake errors).

If it's one of these errors it would be -J being broken because the other one only happens when using cmake targets and not custom_command.

[meow464]

Hmm... I have to think of a way of getting -I and -J down to find_and_load_module. There are lots of calls in the way.

Not all options are put in CompilerOptions is this by design or historical reasons? Could CompilerOptions hold -J and -I and be a static variable instead? Or is that against the code guidelines? I could make it a "set once" with a singleton class but it's already passed around a lot with no problems.

[certik]

Go ahead and implement the -I and -J by using global variables or CompilerOption, whatever it takes to fix it. Then submit a PR. Then we will see what changes you needed to make, and we will brainstorm how to best implement it, and from the actual changes it will be quite obvious, whether CompilerOption, or refactoring the code and pass via arguments, or some other approach is the best.

[meow464]

The function find_and_load_module searches mod files in rl_path but form man ld:

-rpath=dir
           Add a directory to the runtime library search path.  This is used when linking an ELF executable with shared objects.  All -rpath arguments are concatenated and passed to the runtime linker, which
           uses them to locate shared objects at runtime.

[...]

-rpath-link=dir
           When using ELF or SunOS, one shared library may require another.  This happens when an "ld -shared" link includes a shared library as one of the input files.

I will keep the behavior but I don't know if it should be here.

[meow464]

These last commits tell CMake how to call LFortran but there are two issues.

1: Bootstrapping lfortran + the runtime. CMake needs the path to lfortran to exist, we can disable compiler checks but we can't disable path checks. Clang seems to get around this problem somehow but I wasn't able to understand how. The only solution I can think of is splitting the build into two cmake calls, one for the compiler and other for the libraries. Would that be ok?

2: CMake tests are disabled for now because LFortran can't pass them yet.