Unable to build Linux kernel with ARM cross toolchain

[Ypnose]

Hi there, I successfully built a cross compiler for ARM using musl-cross-make. I'm able to build programs for armv6z (Raspberry Pi) with the following CFLAGS using the cross compiler, but I'm unable to build the Linux kernel.

-march=armv6z -mtune=arm1176jzf-s -mfloat-abi=hard -mfpu=vfp

config.mak

TARGET = arm-linux-musleabihf
OUTPUT = /build/cross-arm

DL_CMD = curl -C - -L -o

MUSL_CONFIG += --disable-shared

COMMON_CONFIG += --disable-nls --disable-shared --enable-languages=c
COMMON_CONFIG += CC="gcc -static --static" CXX="g++ -static --static"

GCC_CONFIG += --enable-languages=c
GCC_CONFIG += --disable-libquadmath --disable-decimal-float
GCC_CONFIG += --disable-multilib

GCC output

$ ./arm-linux-musleabihf-gcc -v                                                             
Using built-in specs.
COLLECT_GCC=./arm-linux-musleabihf-gcc
COLLECT_LTO_WRAPPER=/build/cross-arm/bin/../libexec/gcc/arm-linux-musleabihf/9.2.0/lto-wrapper
Target: arm-linux-musleabihf
Configured with: ../src_gcc/configure --enable-languages=c,c++ --with-float=hard --disable-nls --disable-shared --enable-languages=c CC='gcc -static --static' CXX='g++ -static --static' --enable-languages=c --disable-libquadmath --disable-decimal-float --disable-multilib --disable-bootstrap --disable-assembly --disable-werror --target=arm-linux-musleabihf --prefix= --libdir=/lib --disable-multilib --with-sysroot=/arm-linux-musleabihf --enable-tls --disable-libmudflap --disable-libsanitizer --disable-gnu-indirect-function --disable-libmpx --enable-libstdcxx-time=rt --with-build-sysroot=/tmp/musl-cross-make/build/local/arm-linux-musleabihf/obj_sysroot AR_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-linux-musleabihf/obj_binutils/binutils/ar AS_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-linux-musleabihf/obj_binutils/gas/as-new LD_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-linux-musleabihf/obj_binutils/ld/ld-new NM_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-linux-musleabihf/obj_binutils/binutils/nm-new OBJCOPY_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-linux-musleabihf/obj_binutils/binutils/objcopy OBJDUMP_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-linux-musleabihf/obj_binutils/binutils/objdump RANLIB_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-linux-musleabihf/obj_binutils/binutils/ranlib READELF_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-linux-musleabihf/obj_binutils/binutils/readelf STRIP_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-linux-musleabihf/obj_binutils/binutils/strip-new --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu
Thread model: posix
gcc version 9.2.0 (GCC)

Kernel build output

$ pwd
/tmp/linux-4.14.171
$ export PATH="/build/cross-arm/bin:${PATH}"
$ export CROSS_COMPILE="arm-linux-musleabihf-"
$ make mrproper && make ARCH="arm" menuconfig && make V=1 ARCH="arm" zImage dtbs
[...]
  arm-linux-musleabihf-gcc -Wp,-MD,init/.main.o.d -nostdinc -isystem /build/cross-arm/bin/../lib/gcc/arm-linux-musleabihf/9.2.0/include -I./arch/arm/include -I./arch/arm/include/generated  -I./include -I./arch/arm/include/uapi -I./arch/a
rm/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -D__KERNEL__ -mlittle-endian -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -Werror
-implicit-function-declaration -Wno-format-security -std=gnu89 -fno-PIE -fno-dwarf2-cfi-asm -fno-omit-frame-pointer -mapcs -mno-sched-prolog -fno-ipa-sra -mabi=aapcs-linux -mfpu=vfp -funwind-tables -marm -D__LINUX_ARM_ARCH__=7 -march=arm
v5t -Wa,-march=armv7-a -msoft-float -Uarm -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-int-in-bool-context -Wno-address-of-packed-member -Wno-attribute-alias -O2 --param=allow-store-
data-races=0 -DCC_HAVE_ASM_GOTO -Wframe-larger-than=1024 -fno-stack-protector -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-omit-frame-pointer -fno-optimize-sibling-calls -fno-var-tracking-assignments -g -pg -Wdeclaration-
after-statement -Wno-pointer-sign -Wno-stringop-truncation -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=implicit-int -Werror=strict-prototypes -Werror=date-time -Werror=incompa
tible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -Wno-packed-not-aligned -fno-function-sections -fno-data-sections    -DKBUILD_BASENAME='"main"'  -DKBUILD_MODNAME='"main"' -c -o init/.tmp_main.o init/main.c
/tmp/ccOUdmcs.s: Assembler messages:
/tmp/ccOUdmcs.s:2161: Error: selected processor does not support `cpsid i' in ARM mode
/tmp/ccOUdmcs.s:2644: Error: selected processor does not support `cpsid i' in ARM mode
/tmp/ccOUdmcs.s:2789: Error: selected processor does not support `cpsie i' in ARM mode
/tmp/ccOUdmcs.s:3318: Error: selected processor does not support `cpsie i' in ARM mode
/tmp/ccOUdmcs.s:3600: Error: selected processor does not support `cpsid i' in ARM mode
/tmp/ccOUdmcs.s:3658: Error: selected processor does not support `dmb ishst' in ARM mode
/tmp/ccOUdmcs.s:3713: Error: selected processor does not support `dmb ishst' in ARM mode

Full log can be see here. Same problem with 5.5.4.

Do you have any idea ?

Thanks

[richfelker]

I think you selected an ARMv7 cpu in your kernel config (via menuconfig) rather than a v6 one. The kernel should be passing -march=armv7-a itself if you do that, rather than erroring out, but it doesn't seem to be doing so, but in any case it's likely that you didn't mean to do this.

[Ypnose]

Hi Rich, thanks for your answer. Indeed, I just forgot the defconfig before menuconfig. I do not modify options in menuconfig, it is just a check. Unfortunately, specifying it doesn't seem to change anything as the error is the same:

$ make mrproper && make ARCH="arm" bcm2835_defconfig && make ARCH="arm" menuconfig && make V=1 ARCH="arm" zImage dtbs
[...]
arm-linux-musleabihf-gcc -Wp,-MD,init/.main.o.d -nostdinc -isystem /build/cross-arm/bin/../lib/gcc/arm-linux-musleabihf/9.2.0/include -I./arch/arm/include -I./arch/arm/include/generated  -I./include -I./arch/arm/include/uapi -I./arch/arm/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -mlittle-endian -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Wno-format-security -std=gnu89 -fno-dwarf2-cfi-asm -fno-omit-frame-pointer -mapcs -mno-sched-prolog -fno-ipa-sra -mabi=aapcs-linux -mfpu=vfp -marm -Wa,-mno-warn-deprecated -D__LINUX_ARM_ARCH__=6 -march=armv5t -Wa,-march=armv6k -mtune=arm1136j-s -msoft-float -Uarm -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-address-of-packed-member -Os -Wno-maybe-uninitialized --param=allow-store-data-races=0 -Wframe-larger-than=1024 -fstack-protector-strong -Wno-unused-but-set-variable -Wimplicit-fallthrough -Wno-unused-const-variable -fno-omit-frame-pointer -fno-optimize-sibling-calls -fno-var-tracking-assignments -g -pg -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wno-stringop-truncation -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -Wno-packed-not-aligned -fno-function-sections -fno-data-sections    -DKBUILD_BASENAME='"main"' -DKBUILD_MODNAME='"main"' -c -o init/main.o init/main.c
/tmp/ccGdE9M1.s: Assembler messages:
/tmp/ccGdE9M1.s:3920: Error: selected processor does not support `cpsid i' in ARM mode
/tmp/ccGdE9M1.s:4394: Error: selected processor does not support `cpsid i' in ARM mode
/tmp/ccGdE9M1.s:4669: Error: selected processor does not support `cpsie i' in ARM mode
/tmp/ccGdE9M1.s:5394: Error: selected processor does not support `cpsie i' in ARM mode
make[1]: *** [scripts/Makefile.build:266: init/main.o] Error 1
make: *** [Makefile:1693: init] Error 2

I can provide the full log if you need it, even if it's quite the same.

As it is working quite flawlessly with software building, I do not really know if it's related to musl-cross-make. At the end of 2018, I remember it was OK.

[richfelker]

You said with CFLAGS:

-march=armv6z -mtune=arm1176jzf-s -mfloat-abi=hard -mfpu=vfp

but where are you putting these? Note that your errors did change (the one about dmb is gone) because now the kernel is only trying to build for v6, but the cpsid is still present, and that instruction is v6 and up, indicating maybe your toolchain is still for v4t or v5.

[koluckirafal]

Hello, I've got the same problem while trying to build a kernel for ARMv7 platform.

$ grep -v "^#" config.mak | awk 'NF'
TARGET = arm-linux-musleabihf
$ make
$ make install
$ export PATH=/home/bluraf/code/embed/musl-cross-make/output/bin:$PATH
$ arm-linux-musleabihf-gcc -v
Using built-in specs.
COLLECT_GCC=arm-linux-musleabihf-gcc
COLLECT_LTO_WRAPPER=/home/bluraf/code/embed/musl-cross-make/output/bin/../libexec/gcc/arm-linux-musleabihf/9.2.0/lto-wrapper
Target: arm-linux-musleabihf
Configured with: ../src_gcc/configure --enable-languages=c,c++ --with-float=hard --disable-bootstrap --disable-assembly --disable-werror --target=arm-linux-musleabihf --prefix= --libdir=/lib --disable-multilib --with-sysroot=/arm-linux-musleabihf --enable-tls --disable-libmudflap --disable-libsanitizer --disable-gnu-indirect-function --disable-libmpx --enable-libstdcxx-time=rt --with-build-sysroot=/home/bluraf/code/embed/musl-cross-make/build/local/arm-linux-musleabihf/obj_sysroot AR_FOR_TARGET=/home/bluraf/code/embed/musl-cross-make/build/local/arm-linux-musleabihf/obj_binutils/binutils/ar AS_FOR_TARGET=/home/bluraf/code/embed/musl-cross-make/build/local/arm-linux-musleabihf/obj_binutils/gas/as-new LD_FOR_TARGET=/home/bluraf/code/embed/musl-cross-make/build/local/arm-linux-musleabihf/obj_binutils/ld/ld-new NM_FOR_TARGET=/home/bluraf/code/embed/musl-cross-make/build/local/arm-linux-musleabihf/obj_binutils/binutils/nm-new OBJCOPY_FOR_TARGET=/home/bluraf/code/embed/musl-cross-make/build/local/arm-linux-musleabihf/obj_binutils/binutils/objcopy OBJDUMP_FOR_TARGET=/home/bluraf/code/embed/musl-cross-make/build/local/arm-linux-musleabihf/obj_binutils/binutils/objdump RANLIB_FOR_TARGET=/home/bluraf/code/embed/musl-cross-make/build/local/arm-linux-musleabihf/obj_binutils/binutils/ranlib READELF_FOR_TARGET=/home/bluraf/code/embed/musl-cross-make/build/local/arm-linux-musleabihf/obj_binutils/binutils/readelf STRIP_FOR_TARGET=/home/bluraf/code/embed/musl-cross-make/build/local/arm-linux-musleabihf/obj_binutils/binutils/strip-new --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu
Thread model: posix
gcc version 9.2.0 (GCC)
$ export CROSS_COMPILE=arm-linux-musleabihf-
$ export ARCH=arm
$ cd ../linux-5.5.7
$ make sunxi_defconfig
$ make zImage
  CALL    scripts/checksyscalls.sh
  CALL    scripts/atomic/check-atomics.sh
  CC      init/main.o
/tmp/cce6OTk8.s: Assembler messages:
/tmp/cce6OTk8.s:1081: Error: selected processor does not support `cpsid i' in ARM mode
/tmp/cce6OTk8.s:1213: Error: selected processor does not support `cpsid i' in ARM mode
/tmp/cce6OTk8.s:1272: Error: selected processor does not support `cpsie i' in ARM mode
/tmp/cce6OTk8.s:1494: Error: selected processor does not support `cpsie i' in ARM mode
make[1]: *** [scripts/Makefile.build:266: init/main.o] Error 1
make: *** [Makefile:1693: init] Error 2

Full make V=1 zImage log attached: zimage.log

[richfelker]

Your toolchain is targeting baseline arm (v4t) but the kernel build is wrongly assuming it's targeting the same ISA level as the machine you're building a kernel for.

Add GCC_CONFIG += --with-arch=armv6 or similar in your config.mak, or figure out how to get the kernel build to pass -march=armv6 in CFLAGS with your existing toolchain.

[Ypnose]

Forcing armv6 or armv7 using --with-arch in config.mak did the trick. Anyway, as I changed nothing in my personal scripts and all my config.mak, I am intrigued by this behavior.

[isinyaaa]

@Ypnose did you manage to compile the kernel using this toolchain? (I ask because this issue is still open) if so, do you maintain this setup or use something else? I'm currently trying to compile the kernel on the M1 to test stuff on some x86_64 machines and it's giving me an absurd headache to get some setup that works... Any help is appreciated :)

[rofl0r]

trying to compile the kernel on the M1

M1 is aarch64, not arm, so you would have to set your target triple accordingly. note that the kernel uses in this case a different arch string, namely "arm64", which you need to keep in mind when using the original kernel tarball for your kernel headers. the built-in support for sabotage's independent kernel-headers tarball otoh doesnt need any special casing here, as the tarball provides a symlink aarch64->arm64. just put LINUX_VER=headers-4.19.88-1 into your config.mak.

[isinyaaa]

trying to compile the kernel on the M1

M1 is aarch64, not arm, so you would have to set your target triple accordingly. note that the kernel uses in this case a different arch string, namely "arm64", which you need to keep in mind when using the original kernel tarball for your kernel headers. the built-in support for sabotage's independent kernel-headers tarball otoh doesnt need any special casing here, as the tarball provides a symlink aarch64->arm64. just put LINUX_VER=headers-4.19.88-1 into your config.mak.

Sounds reasonable, I'll try this when I've some spare time, thanks :)

[richfelker]

note that the kernel uses in this case a different arch string, namely "arm64", which you need to keep in mind when using the original kernel tarball for your kernel headers

I'm confused why it's necessary to keep this in mind. The kernel uses odd names for most archs, and mcm already knows how to remap them or it wouldn't work at all to get the kernel headers.

[rofl0r]

mcm already knows how to remap them or it wouldn't work at all to get the kernel headers

thanks for clarification. i wasn't aware of this and have always used the kernel-headers tarball.

[Ypnose]

@isinyaaa I was able to build successful armv6 and armv7 toolchain in 2020. You can find various config.mak in my personal project here. About your question about M1, I didn't test it, I won't be able to help you.