RVVM icon indicating copy to clipboard operation
RVVM copied to clipboard

RV32 U-Mode JIT is nonconformant and slow on ctx switch

Open LekKit opened this issue 1 year ago • 5 comments

The issue

  • Running riscv32 userland with riscv64 kernel is broken

Steps to reproduce

  • Use those firmware/kernel/rootfs: rv32_umode.zip
  • Run rvvm fw_jump.bin -k linux_6.8 -i rootfs_rv32.ext2 -nojit
  • The userland will crash almost immediately with random unhandled pagefaults inside the kernel or segfaults in userspace

Investigation

  • RV32 U-mode RVJIT is most likely broken. It doesn't sign-extend the dirty 32-bit registers upon spilling them into hart ctx. So just disable it for now and fix later.
  • Interpreter properly sign-extends writes into rv32 registers to full 64-bit register by intermediate cast to sxlen_t: https://github.com/LekKit/RVVM/blob/9929fdd1cc297228198c97afa75b9f0fc9371b21/src/riscv_cpu.h#L422
  • Interpreter load/stores are computing effective virtual address as xlen_t which is unsigned 32-bit in rv32. It's okay for bare or SV32 MMU, but apparently rv32 U-mode uses an SV39/48 MMU so the upper VA half is not properly sign-extended. Either riscv_mmu should manually sign-extend rv32 addresses on SV39, or switch to sxlen_t addresses in interpreter and making Bare/SV32 modes work with this.
  • The CSR subsystem is likely broken. Applying a 32-bit CSR mask when running in rv32 mode allows to see buildroot MOTD, but most other userspace parts are still crashing.

LekKit avatar Mar 29 '24 08:03 LekKit

No CSR access is happening in rv32 umode (at least not before it crashes). I'd assume kernel never uses smth like rv32 S-mode so S-mode CSRs can't be messed up this way.

SV32 MMU doesn't seem to be used at all.

LekKit avatar Mar 29 '24 09:03 LekKit

Sign-extending or zero-extending rv32 VA pointers in SV39/SV48 translation doesn't help.

The kernel always faults on some magical address 0x3803b0af8. Perhaps this could lead debugging somewhere.

LekKit avatar Mar 29 '24 09:03 LekKit

Crashes are caused by riscv_handle_irqs() in RV32 U-Mode. Using a workaround that disables IRQs when hart is in rv32 mode & disabling RVJIT allows RV32 Buildroot to boot properly on RV64 kernel.

This is due to invalid IRQ mask computed for origin priv_mode and not the one we are switched into on IRQ.

LekKit avatar Mar 31 '24 10:03 LekKit

Fixed the interpreter side in 8aef67b

LekKit avatar Mar 31 '24 12:03 LekKit

RV32 U-Mode somewhat works with JIT since e108450. Note it still isn't spec conformant (RVJIT should sign-extend 32-bit results to 64-bit register, but doesn't now). Performance is pretty low since the JIT cache is flushed on each XLEN change.

LekKit avatar Mar 31 '24 14:03 LekKit