[Confirmation] Regression and Coverage data metrics

[Shaileshkavar]

Hi,

Below are the regression and coverage statistics for the most recent version of IBEX (ec32fb1a6460d8512f9192a84d5fc395502c13f5). Could you kindly confirm which is correct or not?

Regression data: 94.49% PASS 1285 PASSED, 75 FAILED

Coverage data: Code Coverage:

Functional Coverage:

Note: I've read the vrefines files(core_ibex/waivers/*)

-- BR, Shailesh

[GregAC]

As I can't see the column headings for those screenshots I cannot comment as to the accuracy of those numbers.

Our latest nightly regression (sadly results not currently publicly available though we are planning to make them public) reports

93.26% PASS 1287 PASSED, 93 FAILED

With coverage summary report:

Legend: Metric* means cumulative e.g. Block* means Cumulative Block Coverage
name                                     Block* Covered       Branch* Covered      Statement* Covered   Expression* Covered  Toggle* Covered      Statement* Covered   Fsm* Covered         Assertion* Covered   CoverGroup* Covered  
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
uvm_pkg                                  n/a                  n/a                  n/a                  n/a                  n/a                  n/a                  n/a                  n/a                  100.00% (3/3)        
ibex_top                                 94.65% (12009/12688/1850) 90.55% (5498/6072/562) 94.62% (12407/13112/2294) 91.71% (4843/5281/4882) 98.55% (613/622/2402) 94.62% (12407/13112/2294) 100.00% (138/138)    98.08% (358/365/30)  41.54% (8189/19712)  

name                                     CoverGroup Average   CoverGroup Covered   
-----------------------------------------------------------------------------------
riscv_instr_pkg                          92.69%               92.76% (10511/11331) 
push_pull_agent_pkg                      100.00%              100.00% (3/3)        
core_ibex_fcov_if                        92.41%               50.00% (1253/2506)   
core_ibex_pmp_fcov_if.g_pmp_cgs_T        96.33%               91.86% (79/86)       
core_ibex_pmp_fcov_if.g_pmp_cgs_T.g_pmp_region_fcov 93.18%               75.99% (345/454)     

Note for the functional coverage the total bins covered is a little misleading as we have some very large crosses in there. We're still working on closing out coverage for these and there are likely large numbers of illegal bins we haven't had time to identify yet.

I have attached the full regression log so you can see the individual failures

regr.log