[edn/dv] edn_disable_auto_req_mode test sequence failing

[h-filali]

Description

edn_disable_auto_req_mode fails sometimes. It looks like this has something to do with the disablement of the EDN. If the EDN is disabled via backdoor during a TL transaction, the TL sequencer hangs and is thus killed automatically. To reproduce use this git hash f3af826941 and run util/dvsim/dvsim.py hw/ip/edn/dv/edn_sim_cfg.hjson -i edn_disable_auto_req_mode -t xcelium --fixed-seed 61070996226483181682937165685756564904249297810070954908397241823715099635498 Two more failing seeds I could find are: 83106884606859576866395536417688321183151628139158240429607179727476328981642 93668555437452971905965393254453301839962811678600888621534390713802600691300

A possible solution could involve something like killing and restarting the tl sequence before and after disabling the EDN.

Error message: UVM_ERROR @ 3783561 ps: (uvm_sequencer_base.svh:757) uvm_test_top.env.m_tl_agent_edn_reg_block.sequencer [SEQREQZMB] The task responsible for requesting a wait_for_grant on sequencer 'uvm_test_top.env.m_tl_agent_edn_reg_block.sequencer' for sequence 'm_tl_host_base_seq' has been killed, to avoid a deadlock the sequence will be removed from the arbitration queues

[vogelpi]

This still seems to be an issue. The edn_disable_auto_req_mode test has currently a pass rate of roughly 85%. This is not sufficient for V2(S). We need to investigate this for M4.

[vogelpi]

Update: #22752 fixed quite some test failures. The pass rate of edn_disable_auto_req_mode is now back up at 98%. The remaining failures have a predictable pattern though (see below). We should investigate what the issue is here.

Example failure pattern:

6.edn_disable_auto_req_mode.54433158439156122443312458540656012130830202329242750771247128625030336802488
Line 261, in log /container/opentitan-public/scratch/os_regression/edn-sim-vcs/6.edn_disable_auto_req_mode/latest/run.log

  UVM_ERROR @  35973176 ps: (edn_scoreboard.sv:313) [uvm_test_top.env.scoreboard] Check failed sw_cmd_sts == item.d_data (4 [0x4] vs 1 [0x1]) reg name: edn_reg_block.sw_cmd_sts
  UVM_INFO @  35973176 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
  --- UVM Report catcher Summary ---

[vogelpi]

Since failure may be systematic and may be related to the top-level issue we want to debug, I am raising priority here.

[h-filali]

#23004 should fix the initial problem mentioned in the initial description of this issue. Before I started working on this I didn't find any tests failing due to sw_cmd_sts responses. All I found were failures due to deadlocks. However after fixing the deadlocks I found issues due to sw_cmd_sts so this issue should stay open. These are the results of a full regressions worth of disable_auto_req_mode tests:

### Test Results
|  Stage  |  Name   | Tests                     |  Max Job Runtime  |  Simulated Time  |  Passing  |  Total  |  Pass Rate  |
|:-------:|:-------:|:--------------------------|:-----------------:|:----------------:|:---------:|:-------:|:-----------:|
|   V2    | disable | edn_disable_auto_req_mode |      8.000s       |     32.373us     |    49     |   50    |   98.00 %   |
|   V2    |         | **TOTAL**                 |                   |                  |    49     |   50    |   98.00 %   |
|         |         | **TOTAL**                 |                   |                  |    49     |   50    |   98.00 %   |

## Failure Buckets

* `UVM_ERROR (edn_scoreboard.sv:314) [scoreboard] Check failed sw_cmd_sts == item.d_data (* [*] vs * [*]) reg name: edn_reg_block.sw_cmd_sts` has 1 failures:
    * Test edn_disable_auto_req_mode has 1 failures.
        * 40.edn_disable_auto_req_mode.39320318562867595749477158689023392518502151318454332063452493068902464697048\
          Line 98, in log /home/dev/src/scratch/edn-fix-disable-deadlock/edn-sim-xcelium/40.edn_disable_auto_req_mode/latest/run.log

                UVM_ERROR @  10593981 ps: (edn_scoreboard.sv:314) [uvm_test_top.env.scoreboard] Check failed sw_cmd_sts == item.d_data (0 [0x0] vs 4 [0x4]) reg name: edn_reg_block.sw_cmd_sts
                UVM_INFO @  10593981 ps: (uvm_report_catcher.svh:705) [UVM/REPORT/CATCHER]
                --- UVM Report catcher Summary ---
                
                



INFO: [FlowCfg] [scratch_path]: [edn] [/home/dev/src/scratch/edn-fix-disable-deadlock/edn-sim-xcelium]
ERROR: [dvsim] Errors were encountered in this run.

          [   legend    ]: [Q: queued, D: dispatched, P: passed, F: failed, K: killed, T: total]                                                                                                               
00:00:23  [    build    ]: [Q: 00, D: 00, P: 01, F: 00, K: 00, T: 01] 100%                                                                                                                                     
00:00:45  [     run     ]: [Q: 00, D: 00, P: 49, F: 01, K: 00, T: 50] 100%