CV32E40X simulation failures with Questa

[MikeOpenHWGroup]

This issue is motivated by a discussion in pull-request #1092 in which @silabs-robin reported problems compiling and simulating the CV32E40X in the core-v-verif UVM environment with Questa. The error reported is:

# ** Fatal: (SIGSEGV) Bad handle or reference.
# Time: 78405300 ps Iteration: 8 Process: /uvmt_cv32e40x_tb/iss_wrap/cpu/#ALWAYS#245 File: /work/ropeders/src/core-v-verif-mike/vendor_lib/imperas/imperas_DV_COREV/sv/imperas_CV32.sv
# Fatal error in Module CPU at /work/ropeders/src/core-v-verif-mike/vendor_lib/imperas/imperas_DV_COREV/sv/imperas_CV32.sv line 212

I strongly suspect the primary issue here is related to bit-rot of the Questa makefile for the UVM environment, mk/uvmt/vsim.mk. This file has not been updated for 3 months and I am aware that Questa is not in regular use by an members involved in CV32E40X verification in core-v-verif.

[MikeOpenHWGroup]

Hi @silabs-robin, can you add a comment here to provide the command-line necessary to reproduce the above issue? Thanks!

[silabs-robin]

I ran ci_check for 40x using questa. So I think that should be this command: ./bin/ci_check -s vsim --core=cv32e40x.

The result I got was:

CI Check results:
vsim-hello-world.log : PASSED
vsim-illegal.log : PASSED
vsim-csr_instructions.log : PASSED
vsim-riscv_arithmetic_basic_test_0.log : PASSED
CI Check FAILED: Expected 11 tests to run but found only 4 PASSED or FAILED messages

Contrasted to the latest cv32e40x/dev before the makefile refactor, the "corev_rand" tests seem to be missing:

CI Check results:
vsim-hello-world.log : PASSED
vsim-illegal.log : PASSED
vsim-csr_instructions.log : PASSED
vsim-riscv_arithmetic_basic_test_0.log : PASSED
vsim-corev_rand_arithmetic_base_test.log : PASSED
vsim-corev_rand_instr_test.log : PASSED
vsim-corev_rand_jump_stress_test.log : PASSED
vsim-corev_rand_jump_stress_test.log : PASSED
CI Check FAILED: Expected 11 tests to run but found only 8 PASSED or FAILED messages

[MikeOpenHWGroup]

Ah, sorry, I was not clear. What I'm most interested in is the command-line for individual tests that fail. You can get ci_check to give you that with the -p command-line arg:

./bin/ci_check -s vsim --core=cv32e40x -p`

This will dump the generated make commands (but not run them). I'd like to know which test fails and what the failure signature is.

[silabs-robin]

The complete output is this:

[core-v-verif](326)> ./bin/ci_check -s vsim --core=cv32e40x -p
ci_check: topdir  : /work/ropeders/src/core-v-verif
ci_check: core : cv32e40x
ci_check: name_of_ci_check_regression : cv32e40x_ci_check_dev
cd cv32e40x/sim/uvmt; make corev-dv CV_CORE=cv32e40x; make comp CV_CORE=cv32e40x
cd cv32e40x/sim/uvmt; make test TEST=hello-world SIMULATOR=vsim GEN_START_INDEX=0 RUN_INDEX=0 USE_ISS=YES
cd cv32e40x/sim/uvmt; make test TEST=interrupt_test SIMULATOR=vsim GEN_START_INDEX=0 RUN_INDEX=0 USE_ISS=YES
cd cv32e40x/sim/uvmt; make gen_corev-dv test TEST=corev_rand_interrupt SIMULATOR=vsim GEN_START_INDEX=0 RUN_INDEX=0 USE_ISS=YES
cd cv32e40x/sim/uvmt; make gen_corev-dv test TEST=corev_rand_interrupt SIMULATOR=vsim GEN_START_INDEX=1 RUN_INDEX=1 USE_ISS=YES
cd cv32e40x/sim/uvmt; make test TEST=illegal SIMULATOR=vsim GEN_START_INDEX=0 RUN_INDEX=0 USE_ISS=YES
cd cv32e40x/sim/uvmt; make test TEST=csr_instructions SIMULATOR=vsim GEN_START_INDEX=0 RUN_INDEX=0 USE_ISS=YES
cd cv32e40x/sim/uvmt; make test TEST=riscv_arithmetic_basic_test_0 SIMULATOR=vsim GEN_START_INDEX=0 RUN_INDEX=0 USE_ISS=YES
cd cv32e40x/sim/uvmt; make gen_corev-dv test TEST=corev_rand_arithmetic_base_test SIMULATOR=vsim GEN_START_INDEX=0 RUN_INDEX=0 USE_ISS=YES
cd cv32e40x/sim/uvmt; make gen_corev-dv test TEST=corev_rand_instr_test SIMULATOR=vsim GEN_START_INDEX=0 RUN_INDEX=0 USE_ISS=YES
cd cv32e40x/sim/uvmt; make gen_corev-dv test TEST=corev_rand_jump_stress_test SIMULATOR=vsim GEN_START_INDEX=0 RUN_INDEX=0 USE_ISS=YES
cd cv32e40x/sim/uvmt; make gen_corev-dv test TEST=corev_rand_jump_stress_test SIMULATOR=vsim GEN_START_INDEX=1 RUN_INDEX=1 USE_ISS=YES
[core-v-verif](327)>

Note: the first commands, "make corev-dv" and "make comp", do not specify SIMULATOR.

Note: "hello-world" and a few other pass for me, using questa.

Note: The first failure is in "interrupt_test". (This is one of the tests that can give the "# ** Fatal: (SIGSEGV) Bad handle or reference." above.)

Note: After I manually add SIMULATOR=vsim to "make corev-dv" I can run "corev_rand_arithmetic_base_test" to PASSED. (So corev random tests work if corev-dv is just properly compiled first.)

Note: "corev_rand_interrupt" also gave me the SIGSEGV (after I manually did "corev-dv"), so it seems interrupts might be a clue here?

[silabs-robin]

I looked a bit into it and I think I might have found where it goes wrong. (Though I don't yet have any concrete suggestions for how to fix it.)

In the file bin/ci_check.py we have "# Get the build command" and "# Get the simulation command(s)". The build commands seem to be fetched from .metrics.json, which in the cv32e40x/dev branch does not have SIMULATOR on each make command. (Note, the master branch has this set to SIMULATOR=dsim.) Normally this is not a problem, since one has one's terminal window set up with a simulator of choice. The reason I had a problem with it was because I tested with xcelium and then with questa.

I still think the script is broken, because the -s simulator flag doesn't work completely. It seems like it still relies on the SIMULATOR environment variable when it comes to building?

(The "# Get the simulation command(s)" appends options to the end of each command, so if the "make comp" and "make corev-dv" was done in two separate lines instead of one, then we could simply append the SIMULATOR option.)

[MikeOpenHWGroup]

You are correct @silabs-robin, this is latent bug in ci_check. The script sets the simulator correctly for individual test runs, but not for building corev-dv. I'll fix it.

[silabs-robin]

Related: https://github.com/openhwgroup/core-v-verif/pull/2314

[silabs-hfegran]

@MikeOpenHWGroup Is this fixed/still an issue?

[MikeOpenHWGroup]

Still on my ToDo list...