HW breakpoints: "invalid hex 116" message from GDB during Opentitan Verilator simulation

[bilgiday]

My Environment

EDA tool and version: Verilator 4.210 2021-07-07 rev v4.210 (openocd) Open On-Chip Debugger 0.11.0 (riscv32-unknown-elf-gdb) GNU gdb (crosstool-NG 1.24.0.498_5075e1f) 11.1

Operating system: PRETTY_NAME="Debian GNU/Linux rodete" NAME="Debian GNU/Linux" VERSION_ID="rodete" VERSION="12 (rodete)" VERSION_CODENAME=rodete ID=debian

Version of the Ibex source code: Two versions:

• Version 1: I work on the OpenTitan repo v: dd14447ef716bb62ce539e48b4841c7f4e1d2ed7

  • In [rtl] Add support for additional HW breakpoints #1125, DbgHwBreakNum were added to support multiple hardware breakpoints in the Ibex core. I did the following modifications in my local Opentitan repo to enable 8 breakpoints in the earlgrey configuration of Opentitan:

  diff --git a/hw/ip/rv_core_ibex/data/rv_core_ibex.hjson b/hw/ip/rv_core_ibex/data/rv_core_ibex.hjson
  index 0f05f52cf..d82feb8b9 100644
  --- a/hw/ip/rv_core_ibex/data/rv_core_ibex.hjson
  +++ b/hw/ip/rv_core_ibex/data/rv_core_ibex.hjson
  @@ -317,6 +317,13 @@
         expose:  "true"
       },
  
  +    { name:    "DbgHwBreakNum"
  +      type:    "int unsigned"
  +      default: "8"
  +      local:   "false"
  +      expose:  "true"
  +    },
  +
       { name:    "SecureIbex"
         type:    "bit"
         default: "0"
  diff --git a/hw/ip/rv_core_ibex/rtl/rv_core_ibex.sv b/hw/ip/rv_core_ibex/rtl/rv_core_ibex.sv
  index 847168636..56ca88c81 100644
  --- a/hw/ip/rv_core_ibex/rtl/rv_core_ibex.sv
  +++ b/hw/ip/rv_core_ibex/rtl/rv_core_ibex.sv
  @@ -29,6 +29,7 @@ module rv_core_ibex
     parameter bit                   ICacheScramble   = 1'b1,
     parameter bit                   BranchPredictor  = 1'b1,
     parameter bit                   DbgTriggerEn     = 1'b1,
  +  parameter int unsigned          DbgHwBreakNum     = 1,
     parameter bit                   SecureIbex       = 1'b1,
     parameter ibex_pkg::lfsr_seed_t RndCnstLfsrSeed  = ibex_pkg::RndCnstLfsrSeedDefault,
     parameter ibex_pkg::lfsr_perm_t RndCnstLfsrPerm  = ibex_pkg::RndCnstLfsrPermDefault,
  @@ -355,6 +356,7 @@ module rv_core_ibex
       .ICacheScramble           ( ICacheScramble           ),
       .BranchPredictor          ( BranchPredictor          ),
       .DbgTriggerEn             ( DbgTriggerEn             ),
  +    .DbgHwBreakNum            ( DbgHwBreakNum            ),
       // SEC_CM: LOGIC.SHADOW
       // SEC_CM: PC.CTRL_FLOW.CONSISTENCY, CTRL_FLOW.UNPREDICTABLE, CORE.DATA_REG_SW.SCA
       // SEC_CM: EXCEPTION.CTRL_FLOW.GLOBAL_ESC, EXCEPTION.CTRL_FLOW.LOCAL_ESC
  diff --git a/hw/top_earlgrey/data/top_earlgrey.hjson b/hw/top_earlgrey/data/top_earlgrey.hjson
  index e9422f146..6ae37d98c 100644
  --- a/hw/top_earlgrey/data/top_earlgrey.hjson
  +++ b/hw/top_earlgrey/data/top_earlgrey.hjson
  @@ -721,6 +721,7 @@
                      ICacheScramble: "1",
                      BranchPredictor: "0",
                      DbgTriggerEn: "1",
  +		   DbgHwBreakNum: "8",
                      SecureIbex: "1",
                      DmHaltAddr: "tl_main_pkg::ADDR_SPACE_RV_DM__ROM + dm::HaltAddress[31:0]",
                      DmExceptionAddr: "tl_main_pkg::ADDR_SPACE_RV_DM__ROM + dm::ExceptionAddress[31:0]",
  
  

• Version 2: Unmodified, fresh Opentitan repo supporting only a single debug point (OT repo v: f7dd866865479c79e07a086f073dd44d3a87e13d)

Issue

• I followed Opentitan Verilator Setup to build the Verilator executable, build the Opentitan firmware, and to connect to the Verilator via Openocd+GDB.
• After running a few commands without any problems, GDB displays the following message and simulation suspends/freezes: Ignoring packet error, continuing...Invalid hex digit 116. After this point, the only way to end the simulation is to use kill -9 <verilator-proc-id> (See the examples below)
• The same hardware configuration (with 8 debug points) runs without any problem on an CW310 FPGA board.
• I also observe a similar behavior with the original Opentitan configuration with a single hardware breakpoint supported (see the GDB example 3 below).
• I tried to increase the timeout for remote commands but it did not solve the issue: set remotetimeout 30.

Examples

• Here are a few example outputs from different debug sessions:

  • verilator command: build-out/hw/sim-verilator/Vchip_sim_tb --meminit=rom,build-bin/sw/device/lib/testing/test_rom/test_rom_sim_verilator.scr.39.vmem --meminit=flash,build-bin/sw/device/tests/otp_ctrl_smoketest_sim_verilator.64.scr.vmem --meminit=otp,build-bin/sw/device/otp_img/otp_img_sim_verilator.vmem

  • openocd command and output: openocd -s util/openocd -f board/lowrisc-earlgrey-verilator.cfg

  Info : only one transport option; autoselect 'jtag'
  Info : Hardware thread awareness created
  force hard breakpoints
  Info : Listening on port 6666 for tcl connections
  Info : Listening on port 4444 for telnet connections
  Info : Initializing remote_bitbang driver
  Info : Connecting to localhost:44853
  Info : remote_bitbang driver initialized
  Info : This adapter doesn't support configurable speed
  Info : JTAG tap: riscv.tap tap/device found: 0x04f5484d (mfg: 0x426 (Google Inc), part: 0x4f54, ver: 0x0)
  Info : datacount=2 progbufsize=8
  Info : Examined RISC-V core; found 1 harts
  Info :  hart 0: XLEN=32, misa=0x40101106
  Info : starting gdb server for riscv.tap.0 on 3333
  Info : Listening on port 3333 for gdb connections
  Info : accepting 'gdb' connection on tcp/3333
  Warn : keep_alive() was not invoked in the 1000 ms timelimit. GDB alive packet not sent! (1043 ms). Workaround: increase "set remotetimeout" in GDB
  Info : [0] Found 8 triggers
  Warn : keep_alive() was not invoked in the 1000 ms timelimit. GDB alive packet not sent! (1083 ms). Workaround: increase "set remotetimeout" in GDB
  Warn : keep_alive() was not invoked in the 1000 ms timelimit. GDB alive packet not sent! (1018 ms). Workaround: increase "set remotetimeout" in GDB
  Warn : keep_alive() was not invoked in the 1000 ms timelimit. GDB alive packet not sent! (1008 ms). Workaround: increase "set remotetimeout" in GDB
  Warn : keep_alive() was not invoked in the 1000 ms timelimit. GDB alive packet not sent! (1003 ms). Workaround: increase "set remotetimeout" in GDB
  Warn : keep_alive() was not invoked in the 1000 ms timelimit. GDB alive packet not sent! (1091 ms). Workaround: increase "set remotetimeout" in GDB
  Warn : keep_alive() was not invoked in the 1000 ms timelimit. GDB alive packet not sent! (1002 ms). Workaround: increase "set remotetimeout" in GDB
  Warn : keep_alive() was not invoked in the 1000 ms timelimit. GDB alive packet not sent! (1059 ms). Workaround: increase "set remotetimeout" in GDB
  

  • GDB example 1: Modified OT, supporting 8 hardware breakpoints
    • gdb command and output: riscv32-unknown-elf-gdb -iex "set remotetimeout 30" -iex "target extended-remote :3333" -iex "info reg" build-bin/sw/device/lib/testing/test_rom/test_rom_sim_verilator.elf

  (gdb) tbreak rom_test_main 
  Temporary breakpoint 1 at 0x823a: file ../sw/device/lib/testing/test_rom/test_rom.c, line 52.
  (gdb) tbreak *0x20000480 ==> ENTRY POINT FOR THE FLASH
  Temporary breakpoint 2 at 0x20000480
  (gdb) j *0x8084 ==> CHIP RESET VECTOR'S ADDRESS
  Continuing at 0x8084.
  
  Temporary breakpoint 1, rom_test_main () at ../sw/device/lib/testing/test_rom/test_rom.c:52
  52	  CHECK_DIF_OK(dif_pinmux_init(
  (gdb) c
  Continuing.
  Ignoring packet error, continuing...
  Warning:
  Cannot insert breakpoint 2: Invalid hex digit 116
  
  Command aborted.
  (gdb)
  

  • GDB example 2: Modified OT, supporting 8 hardware breakpoints :

  (gdb) tbreak rom_test_main 
  Temporary breakpoint 1 at 0x823a: file ../sw/device/lib/testing/test_rom/test_rom.c, line 52.
  (gdb) j *0x8084
  Line 91 is not in `mmio_region_read32'.  Jump anyway? (y or n) y
  Continuing at 0x8084.
  
  Temporary breakpoint 1, rom_test_main () at ../sw/device/lib/testing/test_rom/test_rom.c:52
  52	  CHECK_DIF_OK(dif_pinmux_init(
  (gdb) si
  0x0000823c	52	  CHECK_DIF_OK(dif_pinmux_init(
  (gdb) si
  0x0000823e	52	  CHECK_DIF_OK(dif_pinmux_init(
  (gdb) si
  0x00008240	52	  CHECK_DIF_OK(dif_pinmux_init(
  (gdb) tbreak rom_test_main 
  Temporary breakpoint 2 at 0x823a: file ../sw/device/lib/testing/test_rom/test_rom.c, line 52.
  (gdb) j *0x8084
  Line 91 is not in `rom_test_main'.  Jump anyway? (y or n) y
  Continuing at 0x8084.
  
  Temporary breakpoint 2, rom_test_main () at ../sw/device/lib/testing/test_rom/test_rom.c:52
  52	  CHECK_DIF_OK(dif_pinmux_init(
  (gdb) si
  0x0000823c	52	  CHECK_DIF_OK(dif_pinmux_init(
  (gdb) 
  0x0000823e	52	  CHECK_DIF_OK(dif_pinmux_init(
  (gdb) 
  0x00008240	52	  CHECK_DIF_OK(dif_pinmux_init(
  (gdb) 
  0x00008242	52	  CHECK_DIF_OK(dif_pinmux_init(
  (gdb) 
  0x00008246	52	  CHECK_DIF_OK(dif_pinmux_init(
  (gdb) 
  0x0000824a	52	  CHECK_DIF_OK(dif_pinmux_init(
  (gdb) 
  Ignoring packet error, continuing...
  Invalid hex digit 116
  (gdb) 
  

  • GDB example 3: Modified OT, supporting only a single hardware breakpoint:

    (gdb) si
    [0] Found 1 triggers
    mmio_region_write32 (base=..., offset=28, value=48) at /usr/local/google/home/bilgiday/Documents/opentitan/ot-repo3/sw/device/lib/base/mmio.h:145
    145	  ((volatile uint32_t *)base.base)[offset / sizeof(uint32_t)] = value;
    (gdb) tbreak rom_test_main 
    Temporary breakpoint 1 at 0x823a: file ../sw/device/lib/testing/test_rom/test_rom.c, line 52.
    (gdb) j *0x8084
    Line 91 is not in `mmio_region_write32'.  Jump anyway? (y or n) y
    Continuing at 0x8084.
  
    Temporary breakpoint 1, rom_test_main () at ../sw/device/lib/testing/test_rom/test_rom.c:52
    52	  CHECK_DIF_OK(dif_pinmux_init(
    (gdb) si
    0x0000823c	52	  CHECK_DIF_OK(dif_pinmux_init(
    (gdb) si
    0x0000823e	52	  CHECK_DIF_OK(dif_pinmux_init(
    (gdb) 
    0x00008242	52	  CHECK_DIF_OK(dif_pinmux_init(
    (gdb) 
    0x00008246	52	  CHECK_DIF_OK(dif_pinmux_init(
    (gdb) 
    0x0000824a	52	  CHECK_DIF_OK(dif_pinmux_init(
    (gdb) 
    Ignoring packet error, continuing...
    Invalid hex digit 116
    (gdb) 
  
  

Question

• When I run the GDB sequences in the examples on Verilator, I consistently get the same errors. On the FPGA, everything works fine.
• I am not sure if something is wrong with my environment (e.g, openocd/GDB settings) or if there is a problem with how Verilator model handles the requests from GDB/Openocd. I just wanted to open a thread to discuss the issue and check if anyone has an idea on the root cause of it.

[GregAC]

Thanks for the report. I suspect it's something to do with the Verilator/OpenOCD interface given it works fine on FPGA. I'll see if 8 hardware breakpoints works on Ibex Super System (FPGA system that gives you an Ibex core and debug and not much else, far simpler than OT so easier to debug) as a first step.

[johngt]

@GregAC - not sure if there were any updates on this or if this is something that has been resolved in one-to-one communications? When you have a chance could you see if this is something that is still a live issue.

[GregAC]

Quick update: I've had a look at an Ibex Super System build with 8 trigger points on FPGA and all looks to work fine. So whatever is happening here is specific to the OT Verilator infrastructure.