Context switching with only M/U modes is broken

[lindemer]

A new CsrPluginConfig that generates only M and U mode was included with the new PMP plugin. https://github.com/SpinalHDL/VexRiscv/blob/d2855fcfca5410c6986b0c3e816cbb867e9b22b9/src/main/scala/vexriscv/plugin/CsrPlugin.scala#L267

This is used to instantiate a new core called VexRiscv_Secure in the LiteX source code here. I'm using this core in LiteX to test Zephyr's userspace features, which require only M and U mode and at least 8 PMP registers. (It's quite complicated to replicate my setup so I won't bother explaining it.)

Zephyr fails to instantiate userspace threads because the CPU doesn't switch privilege levels when it should. I've ruled out the PMP plugin at this point because the same behavior occurs if I allow all accesses in the plugin source code.

Is it possible that the return-from-interrupt implementation is incorrect for builds with only M/U modes? It worked well enough to pass the PMP plugin test, but Zephyr has nested traps, etc. so I could've missed something. The XRET implementation (below) looks correct but I don't quite understand the interrupt implementation. https://github.com/SpinalHDL/VexRiscv/blob/d2855fcfca5410c6986b0c3e816cbb867e9b22b9/src/main/scala/vexriscv/plugin/CsrPlugin.scala#L936

[Dolu1990]

hmmm so first which version of VexRIscv do you use exactly ? (hash) Because i fixed a bug not so long ago that was hit in linux 5.0.10.

i'm not aware of anything else. Is it possible to run a sim with your setup ? Debug thing with wave can realy help, debugging in a blackbox is hell ^^

Zephyr has nested traps

Rawrrrrr. Never realy tried that in baremetal ^^

Another possibility, is that zephyr use a feature which isn't enabled / implemented in VexRiscv.

[lindemer]

hmmm so first which version of VexRIscv do you use exactly ? (hash)

Actually I'm using the latest hash because it's the only one with the PMP plugin 👽

i'm not aware of anything else. Is it possible to run a sim with your setup ? Debug thing with wave can realy help, debugging in a blackbox is hell ^^

I know 😭 I don't think I can track this down without GDB, but I can't find any good documentation on how to debug LiteX. I asked about this before I got started because I knew I'd end up here eventually 😅 (https://github.com/enjoy-digital/litex/issues/676). I haven't managed to learn any more about this since then. It seems clear that lxsim was designed for use with GDB but I can't find instructions anywhere.

[Dolu1990]

hmmm, I'm not talking about GDB debugging, but realy hardware simulation, which produce a VCD file to look at the internal signals. Normaly, litex has support for verilator, isn't it ?

[lindemer]

Yeah that's the lxsim command I'm referring to - it boots the SoC/CPU in Verilator and displays the BIOS interface. I'm pretty certain you can connect to this with GDB but it doesn't come with instructions. I think finding this bug with the VCD waveform would be difficult since I don't know exactly which line triggers the bug.