Unable to build engine for arm64 export template on x86_64 Linux host

[hihi-wang]

Tested versions

Mostly on 4.2.2-stable Checked same on 4.2.3-stable

System information

Ubuntu 22.04 (x86_64)

Issue description

When I build Godot engine from source for export template as arm64 linux, it builds successfully.

scons platform=linuxbsd target=template_debug arch=arm64
scons: Reading SConscript files ...
Auto-detected 24 CPU cores available for build parallelism. Using 23 cores by default. You can override it with the -j argument.
collect2 version 11.4.0
/usr/bin/ld -plugin /usr/lib/gcc/x86_64-linux-gnu/11/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/11/lto-wrapper -plugin-opt=-fresolution=/tmp/ccTl1rLP.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu --as-needed -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie -z now -z relro /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/Scrt1.o /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/11/crtbeginS.o -L/usr/lib/gcc/x86_64-linux-gnu/11 -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/11/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/11/../../.. --version -lgcc --push-state --as-needed -lgcc_s --pop-state -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/x86_64-linux-gnu/11/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/crtn.o
Building for platform "linuxbsd", architecture "arm64", target "template_debug".
Checking for C header file mntent.h... (cached) yes
scons: done reading SConscript files.
scons: Building targets ...
[100%] progress_finish(["progress_finish"], [])
[100%] scons: done building targets.
[Time elapsed: 00:00:05.278]

However, when I try to export my Godot project with this custom template binary, it only generates x86_64 executable, not arm64 one.

Some observation on my env:

1. Exporting with installed export templates works well for both arm64 and x86_64
2. Exporting with custom templates works well for x86_64
3. Exporting with custom templates does not works for arm64. Still, it generates x86_64 executable

Steps to reproduce

1. Build Godot Engine from source with

scons platform=linuxbsd target=template_debug arch=arm64

2. Set custom_template field on export preset.

3. Try to export any Godot project for arm64 linux.

4. Check if it's executes well for arm64 and x64. Note that this should only executes on arm64.

Minimal reproduction project (MRP)

Tested on MultiplayerBomber But same on any Godot project as issue relies on engine compliation.

[bruvzg]

Maybe custom template in the export preset is not set correctly, it should point to the export template executable, and make sure it is set for both release and debug.

In case of Linux, export process simply make a copy of provided or export template executable, it is not modified in any way (unless PCK embedding is enabled). If it picked up the wrong executable, you can replace the exported executable with your custom template executable after the export.

[hihi-wang]

@bruvzg

Thanks for help.

I've double-checked that my export preset points to the template binary correctly, which are generated on $path_to_engine_code/bin/ Same as well when I overwrite my custom templates to the $HOME/.local/share/godot/export_templates/<version>/. (Exported project works well on arm64 before overwriting) PCK embedding is disabled as well.

This is why I'm guessing the problem may lie during the build process of the engine code rather than during the exporting Godot project...

[bruvzg]

Are you cross-building or building on a ARM64 device? I would suggest using official build containers - https://github.com/godotengine/build-containers/, since a lot can go wrong when setting up cross-compiler manually.

[hihi-wang]

@bruvzg

I'm cross-building on x86-64 host (ubuntu 22.04).

Is build container supports custom engine build? As it says...

This repository contains the Dockerfiles for the official Godot engine builds. 

Exporting with official build is not my case and works well with installed templates. I'm adding some features on engine codes, so I need to build and export with it.

[bruvzg]

Containers are used with these build scrips https://github.com/godotengine/godot-build-scripts, you can pass -c to the build.sh skip checkout, and it should build source in the current folder, or modify it to checkout your custom version.

[hihi-wang]

Well.. I got lots of errors while running container. I commented out all other systems other than linux on build.sh.

tar: x86_64-godot-linux-gnu_sdk-buildroot/x86_64-godot-linux-gnu/sysroot/dev: Cannot change mode to rwxr-xr-x: Operation not permitted
tar: x86_64-godot-linux-gnu_sdk-buildroot/x86_64-godot-linux-gnu/sysroot: Cannot change mode to rwxr-xr-x: Operation not permitted
tar: x86_64-godot-linux-gnu_sdk-buildroot/x86_64-godot-linux-gnu: Cannot change mode to rwxr-xr-x: Operation not permitted
tar: x86_64-godot-linux-gnu_sdk-buildroot: Cannot change mode to rwxr-xr-x: Operation not permitted
tar: Exiting with failure status due to previous errors
Error: error building at STEP "RUN dnf install -y wayland-devel &&     curl -LO https://downloads.tuxfamily.org/godotengine/toolchains/linux/2024-01-17/x86_64-godot-linux-gnu_sdk-buildroot.tar.bz2 &&     tar xf x86_64-godot-linux-gnu_sdk-buildroot.tar.bz2 &&     rm -f x86_64-godot-linux-gnu_sdk-buildroot.tar.bz2 &&     cd x86_64-godot-linux-gnu_sdk-buildroot &&     ./relocate-sdk.sh &&     cd /root &&     curl -LO https://downloads.tuxfamily.org/godotengine/toolchains/linux/2024-01-17/i686-godot-linux-gnu_sdk-buildroot.tar.bz2 &&     tar xf i686-godot-linux-gnu_sdk-buildroot.tar.bz2 &&     rm -f i686-godot-linux-gnu_sdk-buildroot.tar.bz2 &&     cd i686-godot-linux-gnu_sdk-buildroot &&     ./relocate-sdk.sh &&     cd /root &&     curl -LO https://downloads.tuxfamily.org/godotengine/toolchains/linux/2024-01-17/aarch64-godot-linux-gnu_sdk-buildroot.tar.bz2 &&     tar xf aarch64-godot-linux-gnu_sdk-buildroot.tar.bz2 &&     rm -f aarch64-godot-linux-gnu_sdk-buildroot.tar.bz2 &&     cd aarch64-godot-linux-gnu_sdk-buildroot &&     ./relocate-sdk.sh &&     cd /root &&     curl -LO https://downloads.tuxfamily.org/godotengine/toolchains/linux/2024-01-17/arm-godot-linux-gnueabihf_sdk-buildroot.tar.bz2 &&     tar xf arm-godot-linux-gnueabihf_sdk-buildroot.tar.bz2 &&     rm -f arm-godot-linux-gnueabihf_sdk-buildroot.tar.bz2 &&     cd arm-godot-linux-gnueabihf_sdk-buildroot &&     ./relocate-sdk.sh": error while running runtime: exit status 2

Besides, still my issue remains : If some cross-compile setup was wrong on my machine, at least it should pops up some error message, not generating x86_64 executable...

[akien-mga]

I can't reproduce the issue on 4.2.2.stable with this config:

Exported file:

$ file new-test-project.arm64 
new-test-project.arm64: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 5.15.0, stripped

I just reused the pre-existing official templates for arm64 to try out the "custom template" workflow, but it would behave the same with another binary.

Can you make sure you selected the "arm64" architecture in the drop down menu, and that your custom template points at the .arm64 binary you built?

[bruvzg]

Cannot change mode to rwxr-xr-x: Operation not permitted

Sorry, completely forgot about this. This issue was reported few times before, and seems to be specific to Ubuntu, not sure if anyone found a solution or reason for these errors (seems like something is wrong with the way podman mount folders, but no idea what). Official builds are done on Fedora host.

[hihi-wang]

@akien-mga

Thanks for check

I just reused the pre-existing official templates for arm64 to try out the "custom template" workflow, but it would behave the same with another binary.

Same on my machine as well. As I noted above, exporting with official templates works well.

Can you make sure you selected the "arm64" architecture in the drop down menu, and that your custom template points at the .arm64 binary you built?

Yes, I've selected arm64 of course. fyi)

[preset.3]

name="test_server"
platform="Linux/X11"
runnable=false
dedicated_server=true
custom_features=""
export_filter="customized"
customized_files={
"res://": "strip"
}
include_filter=""
exclude_filter=""
export_path="test_server/test_server.arm64"
encryption_include_filters=""
encryption_exclude_filters=""
encrypt_pck=false
encrypt_directory=false

[preset.3.options]

custom_template/debug="path_to_my_custom_or_official_template"
custom_template/release="path_to_my_custom_or_official_template"
debug/export_console_wrapper=1
binary_format/embed_pck=false
texture_format/bptc=true
texture_format/s3tc=true
texture_format/etc=false
texture_format/etc2=false
binary_format/architecture="arm64"
ssh_remote_deploy/enabled=false
ssh_remote_deploy/host="user@host_ip"
ssh_remote_deploy/port="22"
ssh_remote_deploy/extra_args_ssh=""
ssh_remote_deploy/extra_args_scp=""
ssh_remote_deploy/run_script="#!/usr/bin/env bash
export DISPLAY=:0
unzip -o -q \"{temp_dir}/{archive_name}\" -d \"{temp_dir}\"
\"{temp_dir}/{exe_name}\" {cmd_args}"
ssh_remote_deploy/cleanup_script="#!/usr/bin/env bash
kill $(pgrep -x -f \"{temp_dir}/{exe_name} {cmd_args}\")
rm -rf \"{temp_dir}\""
texture_format/s3tc_bptc=true
texture_format/etc2_astc=false

Would you check again if it's same if you build engine from source with following command?

scons platform=linuxbsd target=template_debug arch=arm64

[hihi-wang]

@bruvzg

Opps... thanks for letting me know. So it seems like container is not the option if I use ubuntu host...

[akien-mga]

Can you share the output of file bin/godot.linuxbsd.* in your Godot build folder?

[akien-mga]

Ok I can reproduce the bug too with the 4.2 branch.

scons platform=linuxbsd target=template_debug arch=arm64

$ file bin/godot.linuxbsd.*
bin/godot.linuxbsd.template_debug.arm64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=7ad87bf5e6af81b745987ffa920be4a30eecde8d, for GNU/Linux 3.2.0, stripped

You need to select a proper cross-compilation toolchain manually most likely. I don't know if we can pass flags to the default gcc in PATH to target arm64 properly, or whether only clang supports this kind of cross-compilation.

Edit: On master, it outright fails linking with the same command:

/usr/bin/ld: core/libcore.linuxbsd.template_debug.arm64.a(huf_decompress.linuxbsd.template_debug.arm64.o): in function `HUF_decompress4X1_usingDTable_internal':
huf_decompress.c:(.text+0x5d96): undefined reference to `HUF_decompress4X1_usingDTable_internal_fast_asm_loop'
/usr/bin/ld: core/libcore.linuxbsd.template_debug.arm64.a(huf_decompress.linuxbsd.template_debug.arm64.o): in function `HUF_decompress4X2_usingDTable_internal':
huf_decompress.c:(.text+0x70da): undefined reference to `HUF_decompress4X2_usingDTable_internal_fast_asm_loop'
collect2: error: ld returned 1 exit status

Tested both on Fedora 40 with the default GCC package.

[akien-mga]

Note that you can work this around by using the Linux toolchains we provide for this, and use in our official build containers. So even without podman working on your host, you can use those toolchains directly: https://downloads.tuxfamily.org/godotengine/toolchains/linux/

See https://github.com/godotengine/buildroot/?tab=readme-ov-file#using-the-sdks for how to set it up, or reproduce these steps locally:

https://github.com/godotengine/build-containers/blob/610584dd29f5259290c9b759cce89ad49ba82e7e/Dockerfile.linux#L23-L27

You can then force using the right SDK for arm64 with:

PATH=path/to/aarch64-godot-linux-gnu_sdk-buildroot/bin:$PATH scons ...

That should solve the issue for your use case. We still need to figure out how to enable easier cross-compilation without setting a PATH, or aborting in case the host gcc doesn't match the target arch.

[hihi-wang]

Thanks, just checked that it works on my machine with forcing SDK path.