RVVM
RVVM copied to clipboard
RV32 U-Mode JIT is nonconformant and slow on ctx switch
The issue
- Running riscv32 userland with riscv64 kernel is broken
Steps to reproduce
- Use those firmware/kernel/rootfs: rv32_umode.zip
- Run
rvvm fw_jump.bin -k linux_6.8 -i rootfs_rv32.ext2 -nojit - The userland will crash almost immediately with random unhandled pagefaults inside the kernel or segfaults in userspace
Investigation
- RV32 U-mode RVJIT is most likely broken. It doesn't sign-extend the dirty 32-bit registers upon spilling them into hart ctx. So just disable it for now and fix later.
- Interpreter properly sign-extends writes into rv32 registers to full 64-bit register by intermediate cast to
sxlen_t: https://github.com/LekKit/RVVM/blob/9929fdd1cc297228198c97afa75b9f0fc9371b21/src/riscv_cpu.h#L422 - Interpreter load/stores are computing effective virtual address as
xlen_twhich is unsigned 32-bit in rv32. It's okay for bare or SV32 MMU, but apparently rv32 U-mode uses an SV39/48 MMU so the upper VA half is not properly sign-extended. Eitherriscv_mmushould manually sign-extend rv32 addresses on SV39, or switch to sxlen_t addresses in interpreter and making Bare/SV32 modes work with this. - The CSR subsystem is likely broken. Applying a 32-bit CSR mask when running in rv32 mode allows to see buildroot MOTD, but most other userspace parts are still crashing.
No CSR access is happening in rv32 umode (at least not before it crashes). I'd assume kernel never uses smth like rv32 S-mode so S-mode CSRs can't be messed up this way.
SV32 MMU doesn't seem to be used at all.
Sign-extending or zero-extending rv32 VA pointers in SV39/SV48 translation doesn't help.
The kernel always faults on some magical address 0x3803b0af8. Perhaps this could lead debugging somewhere.
Crashes are caused by riscv_handle_irqs() in RV32 U-Mode. Using a workaround that disables IRQs when hart is in rv32 mode & disabling RVJIT allows RV32 Buildroot to boot properly on RV64 kernel.
This is due to invalid IRQ mask computed for origin priv_mode and not the one we are switched into on IRQ.
Fixed the interpreter side in 8aef67b
RV32 U-Mode somewhat works with JIT since e108450. Note it still isn't spec conformant (RVJIT should sign-extend 32-bit results to 64-bit register, but doesn't now). Performance is pretty low since the JIT cache is flushed on each XLEN change.