Could not compile kernel modules on arm64

[iurly]

Description

Trying to compile a kernel module on arm64 results in the following:

/bin/sh: line 1: scripts/basic/fixdep: cannot execute binary file: Exec format error make[3]: *** [/host/lib/modules/6.6.83-flatcar/source/scripts/Makefile.build:243: mymodule.o]

Apparently the host tools shipped with the image are the ones used to cross-compile the kernel for arm64 on x86_64, so they won't naturally run on a native arm64.

Impact

It is not possible to natively compile kernel modules on arm64.

Environment and steps to reproduce

1. Set-up: Flatcar on ARM64 (checked 3033.2.1 and 4152.2.2)
2. Task: Compile a custom kernel module
3. Action(s): None
4. Error: The files present on the image are x86_64 binaries:

find usr/lib/modules | xargs file | grep ELF | less
usr/lib/modules/5.10.93-flatcar/build/scripts/basic/fixdep:                                     ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, not stripped
usr/lib/modules/5.10.93-flatcar/build/scripts/dtc/dtc:                                          ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, not stripped
usr/lib/modules/5.10.93-flatcar/build/scripts/selinux/genheaders/genheaders:                    ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, not stripped
usr/lib/modules/5.10.93-flatcar/build/scripts/selinux/mdp/mdp:                                  ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, not stripped
usr/lib/modules/5.10.93-flatcar/build/scripts/extract-cert:                                     ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, not stripped
usr/lib/modules/5.10.93-flatcar/build/scripts/mod/mk_elfconfig:                                 ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, not stripped
usr/lib/modules/5.10.93-flatcar/build/scripts/mod/modpost:                                      ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, not stripped
usr/lib/modules/5.10.93-flatcar/build/scripts/sign-file:                                        ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, not stripped
usr/lib/modules/5.10.93-flatcar/build/scripts/asn1_compiler:                                    ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, not stripped
usr/lib/modules/5.10.93-flatcar/build/scripts/kconfig/conf:                                     ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, not stripped
usr/lib/modules/5.10.93-flatcar/build/scripts/sorttable:                                        ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, not stripped
usr/lib/modules/5.10.93-flatcar/build/scripts/kallsyms:                                         ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, not stripped

Expected behavior

Those should be native binaries, like for instance bash

./usr/bin/bash:                                                                       ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, strippe
d

Additional information

I found a similar issue https://github.com/radxa/apt/issues/11 which was however fixed by purchasing real ARM hardware 😿 I suppose there should be a way to build host tools for the target architecture as well.

[jepio]

This was fixed here: https://github.com/flatcar/scripts/pull/2694/files#diff-9e87c1b0b8883c8571d32ef546e9e42fece7cab70f0da4f124be6dc8254b9d01

which is in current beta (4230.1.0) and alpha (4284.0.0). The changelog captured this as:

The kernel module build directory now contains native binaries in arm64 images instead of the previous amd64 binaries (scripts#2694)

[iurly]

@jepio thanks for your prompt feedback! Do you have any idea when this fix might hit the stable channel? I suppose we'll have to wait for beta to get promoted, but I have no clue when that might happen. Thanks!

[sayanchowdhury]

@iurly Most likely it would land in the July release, but do keep an eye if Stable gets a major promotion in June.

[tormath1]

Hello @iurly those changes are now available in stable-4230.2.0 - can you confirm that it solves your issue? Feel free to close this issue if yes. Thanks! :)

[tormath1]

I will go ahead and close this issue as it has been solved and available on Stable.