vunit
vunit copied to clipboard
VUnit
Verilog Vunit does not fail when there is an error
The first issue I encounter is because of line 138 of file vunit_pkg.sv, when I launch a test it doesn't end. I had to replace the $stop by $finish (Running with xcellium).
The second issue is that when the test is supposed to fail, it's not reported by the tool. For example, I'm running verilog_ams example from the repo. The test "Test that fail" should fail but from the output I get:
==== Summary ===================================== pass lib.tb_dut.Test that pass (1.3 seconds) pass lib.tb_dut.Test that fail (1.3 seconds)
pass 2 of 2
Total time was 2.6 seconds Elapsed time was 2.6 seconds
@vperrin59 Is there some option to Xcelium that let you define the simulator behavior on $stop?
I looked at the xrun (xcellium) options and I didn't see any knobs for exiting on $stop. I also checked on the web support and nothing came out.
Another thing that was buzzing me in the VUnit. I see that when check fails the verilog macro use $error command. However I didn't anywhere in the Vunit script where it parses for errors (*E). So in my case, even when $error is executed, the test was passing. That's not what I expected. I kind of fixed the issue by having a custom post_check callback in my config and parsing myself for the (*E). Could you point me where the errors are supposed to be parsed @LarsAsplund ?
@vperrin59 VUnit doesn't rely on log parsing for error detection. It fails when you use one of VUnit's error checking mechanisms or when a simulation ends prematurely. That's why it's important to configure the simulator (VUnit responsibility) to stop on errors unless there is a mechanism to read out error counters. In that case you can read those counters at the end of the simulation and fail with VUnit's mechanism if they are not zero (this is how you use VUnit with UVM). Parsing logs is something that you should only have to do when error reporting is done in a non-standard way. For example if someone uses $display to print an error message. In that case you can use post check. The benefit of this approach is that you can detect many types of unexpected errors like null pointer access and internal simulator exceptions without being dependent on simulator specific log formats. Also silent errors are detected (simulator crashes with no log output)
Thanks for the detailed explanation. Now I understand that Vunit is relying on simulator exit before the end of the test to detect the failure.
I have also run into this issue. Xcelium doesn't stop when an *E occurs and therefore when the simulation does end, via $finish, I also had to change $stop to $finish in vunit_pkg.sv to get vunit to run the full simulation and exit, no errors are detected by vunit even though the log definitely shows some errors occurred. I am using UVM and do a check for uvm errors in the TEST_SUITE_CLEANUP phase using the CHECK_EQUAL macro. An assertion error is triggered but the simulator doesn't stop prematurely and therefore vunit doesn't detect the error. I did change the CHECK_EQUAL(x,y) to 'if( x != y) $fatal("this failed");' and that allowed vunit to detect the error, but doesn't feel like the right solution. I've been digging around trying to figure out how vunit is supposed to detect that an error has occured, presumably from the simulator exiting, but haven't been able to figure out where that occurs. I did find where it occurs in the modelsim case, I have also tested with modelsim and have had success getting it to work, and wonder if there's a similar approach to xcelium or if it is a totally different beast. I have also been trying to get xcelium to quit on errors with no success so far. I am reaching out to Cadence to see if they have a solution for this as well.
I cannot try this myself right now, but maybe we could set the option -errormax 1? (Or was it -maxerrors 1?)
