spike fails with hello world compiled with -march=rv32imac

[hirooih]

spike works with hello-world compiled with -march=rv32i, but fails with one compiled with -march=rv32imac. What am I doing wrong?

The last comment in #453 by @jundijiujieke seems to be the same as my problem.

---

gcc

gcc 12.2.0, latest homebrew version on macOS

$ riscv64-unknown-elf-gcc -v
Using built-in specs.
COLLECT_GCC=riscv64-unknown-elf-gcc
COLLECT_LTO_WRAPPER=/opt/homebrew/Cellar/riscv-gnu-toolchain/main/libexec/gcc/riscv64-unknown-elf/12.2.0/lto-wrapper
Target: riscv64-unknown-elf
Configured with: /private/tmp/riscv-gnu-toolchain-20230223-34216-18jz0sg/gcc/configure --target=riscv64-unknown-elf --prefix=/opt/homebrew/Cellar/riscv-gnu-toolchain/main --disable-shared --disable-threads --enable-languages=c,c++ --with-pkgversion=g2ee5e430018-dirty --with-system-zlib --enable-tls --with-newlib --with-sysroot=/opt/homebrew/Cellar/riscv-gnu-toolchain/main/riscv64-unknown-elf --with-native-system-header-dir=/include --disable-libmudflap --disable-libssp --disable-libquadmath --disable-libgomp --disable-nls --disable-tm-clone-registry --src=/private/tmp/riscv-gnu-toolchain-20230223-34216-18jz0sg/gcc --enable-multilib --with-abi=lp64d --with-arch=rv64imafdc --with-tune=rocket --with-isa-spec=2.2 'CFLAGS_FOR_TARGET=-Os   -mcmodel=medany' 'CXXFLAGS_FOR_TARGET=-Os   -mcmodel=medany'
Thread model: single
Supported LTO compression algorithms: zlib

spike and pk

I built the latest spike and pk from GitHub.

• https://github.com/riscv-software-src/riscv-isa-sim
• https://github.com/riscv-software-src/riscv-pk

spike build

$ mkdir build
$ cd build
$ ../configure --prefix=/opt/riscv
$ make install

pk build

$ mkdir build
$ cd build
$ ../configure --prefix=/opt/riscv --host=riscv64-unknown-elf --with-arch=rv32i
$ make install

hello world with -march=rv32i

$ riscv64-unknown-elf-gcc -g -O -march=rv32i -mabi=ilp32    hello.c   -o hello
$ spike --isa=rv32imac /opt/riscv/riscv32-unknown-elf/bin/pk hello
bbl loader
hello world!

Works fine.

hello world with -march=rv32imac

$ riscv64-unknown-elf-gcc -g -O -march=rv32imac -mabi=ilp32    hello.c   -o hello
$ spike --isa=rv32imac /opt/riscv/riscv32-unknown-elf/bin/pk hello
bbl loader
z  00000000 ra 8000452c sp 80216cd0 gp 00000000
tp 00000000 t0 8000d184 t1 80216e14 t2 00000000
s0 80216cd1 s1 8001405c a0 0000002e a1 00000000
a2 0000003f a3 10000000 a4 10000005 a5 00000000
a6 00000000 a7 00000000 s2 80014064 s3 80014068
s4 80012000 s5 8000f2f4 s6 00000001 s7 00000000
s8 00000000 s9 8001229e sA 80012260 sB 00000001
t3 80216e4c t4 00000000 t5 00000003 t6 00000000
pc 80005624 va/inst 10000005 sr 80006100
Kernel load segfault @ 0x10000005

Segfault on a load instruction from address 0x10000005 at instruction address 0x80005624.

$ riscv64-unknown-elf-objdump -D /opt/riscv/riscv32-unknown-elf/bin/pk
...
80005620 <uart16550_getchar>:
80005620:	0000f697          	auipc	a3,0xf
80005624:	a406a683          	lw	a3,-1472(a3) # 80014060 <uart16550_reg_shift>  <- fails here!!!
80005628:	00500793          	li	a5,5
...

uart16550.c

#define UART_REG_LSR       5    // line status register
...
int uart16550_getchar()
{
  if (uart16550[UART_REG_LSR << uart16550_reg_shift] & UART_REG_STATUS_RX)
    return uart16550[UART_REG_QUEUE << uart16550_reg_shift];
  return -1;
}

load segfault causes on access to uart16550[UART_REG_LSR << 0].

$ spike --isa=rv32imac --dump-dts /opt/riscv/riscv32-unknown-elf/bin/pk hello
    SERIAL0: ns16550@10000000 {
      compatible = "ns16550a";
      clock-frequency = <10000000>;
      interrupt-parent = <&PLIC>;
      interrupts = <1>;
      reg = <0x0 0x10000000 0x0 0x100>;
      reg-shift = <0x0>;
      reg-io-width = <0x1>;
    };
...

Access to 0x10000005 should be fine.

[luismarques]

You need to compile pk with --with-arch=rv32imac as well.

pk tests for the condition of loading an RVC ELF program when pk itself is not compiled for RVC, which would prevent your program from running and thus prevent the crash. Unfortunately, the test only checks the ELF header: https://github.com/riscv-software-src/riscv-pk/blob/7520a474199df8b9ef7cee32eebcea7577813814/pk/elf.c#L33. That test is no longer adequate for binaries that indicate RVC in the newer .riscv.attributes section, through the Tag_RISCV_arch tag. I'll see if I can make a PR to strengthen that RVC test.

I still need to look at why the crash happens when accessing 0x10000005 for rv32imac but not rv32i. I have a vague memory I once knew what the pk RVC test was protecting against but I don't remember now and it's not immediately obvious how an uncompressed lbu instruction accessing that mmio address would cause the crash in one case but not the other.

(BTW, I believe your comment "<- fails here!!" was taken from the wrong disassembly. I can reproduce your crash but it crashes in a place that makes more sense, the lbu instruction that actually accesses the mmio address, not the one you point to, which isn't even in the correct function. Still, thanks for the otherwise high-quality bug report!)

[hirooih]

@luismarques

Thank you for your response.

You need to compile pk with --with-arch=rv32imac as well.

If this is correct, we should update README.md of pk.

By default, 64-bit (RV64) versions of pk and bbl are built. To built 32-bit (RV32) versions, supply a --with-arch=rv32i flag to the configure command.

But as you wrote:

I have a vague memory I once knew what the pk RVC test was protecting against but I don't remember now ...

I also does not understand why --with-arch=rv32i is wrong.

---

(BTW, I believe your comment "<- fails here!!" was taken from the wrong disassembly. ...

In my original report I used riscv-pk 54de960a and riscv-isa-sim 76b0027c. I can reproduce the fail even now.

I took the PC value, 0x80005624, from the next line of dump.

pc 80005624 va/inst 10000005 sr 80006100

The fail occurs on the load word from uart16550_reg_shift (0x80014060) at 0x80005624, (not a load instruction from address 0x10000005 as I reported).

$ riscv64-unknown-elf-objdump -D /opt/riscv/riscv32-unknown-elf/bin/pk
...
80005620 <uart16550_getchar>:
80005620:	0000f697          	auipc	a3,0xf
80005624:	a406a683          	lw	a3,-1472(a3) # 80014060 <uart16550_reg_shift>  <- fails here!!!
80005628:	00500793          	li	a5,5
8000562c:       0000f717                auipc   a4,0xf
80005630:       a3872703                lw      a4,-1480(a4) # 80014064 <uart16550>
80005634:       00d797b3                sll     a5,a5,a3
80005638:       00f707b3                add     a5,a4,a5
8000563c:       0007c783                lbu     a5,0(a5)
80005640:       0017f793                and     a5,a5,1
...
        ...

80014060 <uart16550_reg_shift>:
80014060:       0000                    .2byte  0x0
        ...

But some values in the dump are strange.

80005620:	0000f697          	auipc	a3,0xf

This set the value 0x80005620 + 0xf000 = 0x80014620 to a3.

However the value of a3 is 0x10000000 in the dump.

80005624:	a406a683          	lw	a3,-1472(a3) # 80014060 <uart16550_reg_shift>  <- fails here!!!

0x80014620 - 1472 = 0x80014060. This is the correct address of uart16550_reg_shift. A fail should not occur on 0x80005624 which the dump shows.

Kernel load segfault @ 0x10000005

This line should be from the lbu instruction on 0x8000563c.

I can reproduce your crash but it crashes in a place that makes more sense, the lbu instruction that actually accesses the mmio address, not the one you point to, which isn't even in the correct function.

Could you share your dump and disassembly with me? It may be a clue for me.

Thanks in advance.

[hirooih]

Dear @luismarques and @aswaterman,

I am revisiting this issue.

In my original report I used riscv-pk 54de960a and riscv-isa-sim 76b0027.

This time I am using the current latest versions.

• spike: 6f28e4b
• riscv-pk: 4f3debe
• gcc (the current homebrew version)

riscv64-unknown-elf-gcc -v
Using built-in specs.
COLLECT_GCC=riscv64-unknown-elf-gcc
COLLECT_LTO_WRAPPER=/opt/homebrew/Cellar/riscv-gnu-toolchain/main/libexec/gcc/riscv64-unknown-elf/12.2.0/lto-wrapper
Target: riscv64-unknown-elf
Configured with: /private/tmp/riscv-gnu-toolchain-20230223-34216-18jz0sg/gcc/configure --target=riscv64-unknown-elf --prefix=/opt/homebrew/Cellar/riscv-gnu-toolchain/main --disable-shared --disable-threads --enable-languages=c,c++ --with-pkgversion=g2ee5e430018-dirty --with-system-zlib --enable-tls --with-newlib --with-sysroot=/opt/homebrew/Cellar/riscv-gnu-toolchain/main/riscv64-unknown-elf --with-native-system-header-dir=/include --disable-libmudflap --disable-libssp --disable-libquadmath --disable-libgomp --disable-nls --disable-tm-clone-registry --src=/private/tmp/riscv-gnu-toolchain-20230223-34216-18jz0sg/gcc --enable-multilib --with-abi=lp64d --with-arch=rv64imafdc --with-tune=rocket --with-isa-spec=2.2 'CFLAGS_FOR_TARGET=-Os   -mcmodel=medany' 'CXXFLAGS_FOR_TARGET=-Os   -mcmodel=medany'
Thread model: single
Supported LTO compression algorithms: zlib zstd
gcc version 12.2.0 (g2ee5e430018-dirty) 

Now "hello world with -march=rv32imac" causes an assertion fail, as @luismarques wrote:

pk tests for the condition of loading an RVC ELF program when pk itself is not compiled for RVC, which would prevent your program from running and thus prevent the crash. Unfortunately, the test only checks the ELF header: https://github.com/riscv-software-src/riscv-pk/blob/7520a474199df8b9ef7cee32eebcea7577813814/pk/elf.c#L33.

riscv64-unknown-elf-gcc -g -O -march=rv32imac -mabi=ilp32    hello.c   -o hello
spike --isa=rv32imac /opt/riscv/riscv32-unknown-elf/bin/pk hello
assertion failed @ 0x80001360: !(eh.e_flags & EF_RISCV_RVC)

Now the assertion is on L47

#ifndef __riscv_compressed
  kassert(!(eh.e_flags & EF_RISCV_RVC));
#endif

If I comment out the assertion, it works well (as I expected).

riscv64-unknown-elf-gcc -g -O -march=rv32imac -mabi=ilp32    hello.c   -o hello
spike --isa=rv32imac /opt/riscv/riscv32-unknown-elf/bin/pk hello
loading ELF program: hello
hello world!

The fix was added by @aswaterman Disallow execution of RVC binaries on non-RVC pk.

I don't see why the pk has to be built with --with-arch=rv32ic when it is used with the binaries with C-extensions.

From README.md of pk

To built 32-bit (RV32) versions, supply a --with-arch=rv32i flag to the configure command.

If that constraint is really necessary, this statement must be updated.

Regards,

[aswaterman]

It is true that pk needs to be compiled with RVC in order to process RVC binaries. I can understand why you find this surprising, but there is a reason: the emulation code (for floating-point and for misaligned loads and stores) is different in the RVC case.

I agree the pk README should reflect this; please submit a PR on that repo.