cocotb
Introduce simulator support grades
We support a lot of simulators. Not all of them work all of the time. We should grade this support (school grades?) and make that available in the documentation. This will allow users who are interested in using cocotb to better select a simulator and implementation language for their design before going in. It will also set their expectations for when things inevitably fail on the simulators with lower grade, hopefully reducing their frustration.
The best way we have currently to judge this is how capable of completing the regression a simulator is. For example, Icarus can pass with no failures (A). Questa and Riviera both run, but there are a few things we have to skip to avoid failures (B). And finally GHDL and Verilator cannot complete the regression, both from severe crashes and exceptions and they fail many tests (C). And finally there is the theoretical grades of: fails to run successfully in any test (D) and completely unsupported (F).
If this is deemed appropriate, I can create a PR modifying the Simulator Support page.
I don't know either about the Synopsys and Cadence simulators to be able to grade them. Recommendations/analysis are welcome.
We could extend our existing cocotb.test() decorator with attributes like description and related_issue (and sim_name, sim_version) for each of expect_error, expect_fail, skip which would (a) document the test better and (b) allow us to programmatically extract this information into a matrix with simulator name/version and test name as axes.
This makes it easier for a user to judge whether a simulator is well-behaved in a certain functional area in cocotb or not (things like normalized number of expect_error could be shown in each simulator's subsection of "Simulator Support").
What remains to be solved is that
- some tests are not even started by the Makefile depending on
SIMandTOPLEVEL_LANGbecause the support is not there in general; and - within test code, we sometimes differentiate between simulators and their versions
