Raspberry Pi Zero W: LLVM optimization causes "unsupported FP instruction in kernel mode"

[mojyack]

Describe the bug

On Raspberry Pi Zero W, If a kernel is built with LLVM, using vfp causes an "unsupported FP instruction in kernel mode" exception and kernel panic.

Steps to reproduce the behaviour

• Clone repo: git clone --depth=1 -b rpi-6.6.y https://github.com/raspberrypi/linux/
• To build with llvm, patch some files(not related to this issue): curl https://github.com/raspberrypi/linux/commit/f43a67d9309a988eb1def20bc422c0882ce65e5a.patch | patch -p1
• Build kernel with LLVM: make ARCH=arm CC=clang LLVM=1
• Boot into a new kernel
• Compile and run test program from https://github.com/raspberrypi/linux/issues/859#issuecomment-84036262
• Kernel crashes

Device (s)

Raspberry Pi Zero W / WH

System

Kernel: 6.6.12+(7df1d3b44368209517fc7dc0b35ee2621a94722e) Compiler: clang version 16.0.6 / clang version 17.0.6 Distribution: Gentoo Linux

Logs

No response

Additional context

The stack trace above shows an "undefined instruction" occurs at vfp_support_entry + 0xd4. I disassembled the problematic vfp_support_entry dump.txt vfp_support_entry + 0xd4 = 0x1540:

0000146c <vfp_support_entry>:
...
    1518:	e317020a 	tst	r7, #-1610612736	@ 0xa0000000
    151c:	1a000005 	bne	1538 <vfp_support_entry+0xcc>
    1520:	eef10a10 	vmrs	r0, fpscr
    1524:	e3100a01 	tst	r0, #4096	@ 0x1000
    1528:	1a000002 	bne	1538 <vfp_support_entry+0xcc>
    152c:	e3100807 	tst	r0, #458752	@ 0x70000
    1530:	0a000061 	beq	16bc <vfp_support_entry+0x250>
    1534:	e3877202 	orr	r7, r7, #536870912	@ 0x20000000
    1538:	e594003c 	ldr	r0, [r4, #60]	@ 0x3c
    153c:	e3a09010 	mov	r9, #16
    1540:	eef16a10 	vmrs	r6, fpscr
    1544:	e389940f 	orr	r9, r9, #251658240	@ 0xf000000
    1548:	e2800004 	add	r0, r0, #4
    154c:	e584003c 	str	r0, [r4, #60]	@ 0x3c
    1550:	e3e0009f 	mvn	r0, #159	@ 0x9f
    1554:	e240032e 	sub	r0, r0, #-1207959552	@ 0xb8000000
    1558:	e0070000 	and	r0, r7, r0
    155c:	eee80a10 	vmsr	fpexc, r0
    1560:	eef00a10 	vmrs	r0, fpsid
...

This part is originated from inlined VFP_bounce https://github.com/raspberrypi/linux/blob/7df1d3b44368209517fc7dc0b35ee2621a94722e/arch/arm/vfp/vfpmodule.c#L324-L350 In VFP_bounce, there are three fmxr/fmrx instructions in this order: fmxr(FPEXC, ...), fmrx(FPSID), fmrx(FPSCR). But as we can see in dump.txt, LLVM reorders them into fmrx(FPSCR)(0x1540), fmxr(FPEXC, ...)(0x155c), fmrx(FPSID)(0x1560) and this is the cause of panic.

I tried to prevent llvm from reordering these instructions and found that removing static from VFP_bounce signature works. Here is a disassemble snippet of the patched kernel: dump-patched.txt With this fix, the test program runs without kernel panic.