Custom RoCC is working under Rocket64b1, but not b2/b4

[reitowo]

I know it could probably not be the right place to ask this common RocketChip question here, please tell me a better place if you know one. 🫶

I created a simple RoCC design under the generators folder called reito. It has a very basic implementation structure:

---

All source codes are here:

build.sbt

name := "reito" 
version := "1.0.0"   
scalaVersion := "2.12.10" 

build.properties

sbt.version=1.3.13

config.scala

package reito

import Chisel._
import freechips.rocketchip.config._
import freechips.rocketchip.subsystem._
import freechips.rocketchip.system._
import freechips.rocketchip.tile._
import freechips.rocketchip.diplomacy._
import reito._

class WithReito extends Config ((site, here, up) => {  
    case BuildRoCC => up(BuildRoCC) ++ Seq(
        (p: Parameters) => {
            val reito = LazyModule(new ReitoSafe(OpcodeSet.all)(p))
            reito
        }
    )
})

reito.scala

package reito

import chisel3._
import chisel3.util._
import freechips.rocketchip.rocket._
import freechips.rocketchip.config._
import freechips.rocketchip.tile._  

class ReitoSafe(opcodes: OpcodeSet)(implicit p: Parameters) extends LazyRoCC(opcodes) {
	override lazy val module = new ReitoSafeModuleImp(this)
}

class ReitoSafeModuleImp(outer: ReitoSafe)(implicit p: Parameters) extends LazyRoCCModuleImp(outer) with HasCoreParameters { 
    val req_rd = Reg(chiselTypeOf(io.cmd.bits.inst.rd))  
    val req_rs1 = Reg(chiselTypeOf(io.cmd.bits.rs1))  
    val req_rs2 = Reg(chiselTypeOf(io.cmd.bits.rs2))  
    val req_funct = Reg(chiselTypeOf(io.cmd.bits.inst.funct))  

    val res_data = Reg(chiselTypeOf(io.resp.bits.data))  

    val s_idle :: s_req :: s_ok :: s_resp :: Nil = Enum(4)
    val state = RegInit(s_idle)
    val antiDead = RegInit(0.U(32.W))

    io.cmd.ready := (state === s_idle)

    when (state === s_ok) { 
        state := s_resp
    }

    when (state === s_req) { 
        when (req_funct === 10.U) {
			res_data := req_rs1 + req_rs2
		} .elsewhen (req_funct === 11.U) {
			res_data := req_rs1 - req_rs2
		} .elsewhen (req_funct === 12.U) {
			res_data := req_rs1 ^ req_rs2
		} .elsewhen (req_funct === 13.U) {
			res_data := req_rs1 | req_rs2
		} .elsewhen (req_funct === 14.U) {
			res_data := req_rs1 & req_rs2
		} .otherwise { 
			res_data := 114514.U(32.W)
		} 
        state := s_ok
    }

    when (io.cmd.fire()) {
        req_rd := io.cmd.bits.inst.rd
        req_rs1 := io.cmd.bits.rs1
        req_rs2 := io.cmd.bits.rs2
        req_funct := io.cmd.bits.inst.funct 
        state := s_req
    }  

    when (io.resp.fire()) { 
        state := s_idle 
        antiDead := 0.U
    } 

    when (antiDead >= 1000.U) {
        state := s_idle 
        antiDead := 0.U
    } .elsewhen (state =/= s_idle) {
        antiDead := antiDead + 1.U
    }

    io.resp.valid := (state === s_resp)
    io.resp.bits.rd := req_rd
    io.resp.bits.data := res_data

    io.busy := (state =/= s_idle)
    io.interrupt := false.B
    io.mem.req.valid := false.B 
}

---

I added this generator folder in Makefile to let sbt find that:

CHISEL_SRC_DIRS = \
  src/main \
  rocket-chip/src/main \
  generators/gemmini/src/main \
  generators/riscv-boom/src/main \
  generators/sifive-cache/design/craft \
  generators/testchipip/src/main \
  generators/reito/src/main

I modified Rocket64b1 and Rocket64b2 in the rocket.scala:

class Rocket64b1 extends Config(
  new reito.WithReito     ++
  new WithNBreakpoints(8) ++
  new WithNBigCores(1)    ++
  new RocketBaseConfig)

class Rocket64b2 extends Config(
  new reito.WithReito     ++
  new WithNBreakpoints(8) ++
  new WithNBigCores(2)    ++
  new RocketBaseConfig) 

I generate bitstream for board kc705 and foreach design, and I add this code in opensbi/lib/sbi/sbi_hard.c function sbi_hard_delegation_dump:

#define ROCC_INSTRUCTION_DSS(X, rd, rs1, rs2, funct) \
    ROCC_INSTRUCTION_R_R_R(X, rd, rs1, rs2, funct)

#define ROCC_XD     0x4
#define ROCC_XS1    0x2
#define ROCC_XS2    0x1

#define ROCC_INSTRUCTION_R_R_R(X, rd, rs1, rs2, funct)                  \
    do {                                                                \
        __asm__ __volatile__ (                                          \
            ".insn r CUSTOM_" #X ", %3, %4, %0, %1, %2\n\t"             \
            : "=r" (rd)                                                 \
            : "r" (rs1), "r" (rs2),                                     \
              "i" (ROCC_XD | ROCC_XS1 | ROCC_XS2), "i" (funct));        \
    } while (0)

void sbi_hart_delegation_dump(struct sbi_scratch *scratch,
			      const char *prefix, const char *suffix)
{ 
	sbi_printf("%sMSTATUS%s: 0x%" PRILX "\n",
		   prefix, suffix, csr_read(CSR_MSTATUS));  
	sbi_printf("%sMISA   %s: 0x%" PRILX "\n",
		   prefix, suffix, csr_read(CSR_MISA));
	
	uint64_t result = 0x12345678;
	asm volatile ("fence");
	ROCC_INSTRUCTION_DSS(2, result, 15, 10, 10);
	asm volatile ("fence" ::: "memory");
	
	sbi_printf("rocc result %lu\n", result);

	if (!misa_extension('S'))
		/* No delegation possible as mideleg does not exist*/
		return;

	sbi_printf("%sMIDELEG%s: 0x%" PRILX "\n",
		   prefix, suffix, csr_read(CSR_MIDELEG));
	sbi_printf("%sMEDELEG%s: 0x%" PRILX "\n",
		   prefix, suffix, csr_read(CSR_MEDELEG));
}

(I added here because it is faster to verify the RoCC design and not need to wait for Linux boot up. It is also because I was trying to resolve MSTATUS bit issue earlier.)

After flash the design and get everything ready (recompile opensbi and etc, make sdcard again), the Rocket64b1 works correctly.

However, the Rocket64b2 will just hang at this RoCC instruction.

---

I've done full search and I also tried to debug the core with ILA, but I can't find why this RoCC design will hang the processor and only work in single core situation. Can anyone provide any precious suggestions? Paid support is also ok!