Add possibility to see 'skipped' when attribute-selected tests are ignored (suggestion)

[dejfson]

trafficstars

I'm using attributes to set, which tests to perform. Typically I have bunch of tests tagged by user attribute .disabled, and my automatic python script is running all the tests not having this attribute. All works flawlessly, however very quickly becomes cluttered when many testbench files exist and many disabled tests exist in those testbench files. The reason why I'm disabling them is, that either they are incomplete, or they are 'visual' (meaning, that I need them there because I use them as sanboxes for modelsim to look manually on how the design performs).

The issue I have with this system is, that when running the test suite, I have no clue which tests were actually skipped. Having one or two disabled one can remember them, but not when one has twenty-thirty of them scattered around the testbenches. At the end it pretty much causes, that I forget to repair the tests, because I don't know about them unless I re-run the suite with --with-attribute=.disabled. Would it be possible to display in addition to 'pass' and 'fail' as well 'skip' such, that when the suite is ran, it clearly states which tests were skipped?

I saw someone asking for that feature quite some time ago in issue #398. The solution with attributes is great for the tests exclusions/inclusions, however I miss the other part - clearly stating which of them were excluded in the final fail/pass list.

Probably the best way for me would be to add 'disabled' as a system attribute, as e.g. run_all_in_same_sim is, and just automatically skip those disabled, and report them in the final pass/fail list while keeping the functionality of attributes selection as is (i.e. not to report them in the fail/pass log)

Many thanks for consideration.

[eine]

We discussed showing skipped testbenches in January 16, 2020 7:01 PM:

Maybe we can have a list of found but ignored/skipped testbenches to be optionally shown in the summary?

Yes that is an alternative. But I think it is important that the reason for the skip is shown to not cause frustration

Yes. I was thinking about marking them as "invalid name", "no runner_cfg", etc.

So, overall, I agree that we might/should improve the information/usefulness of skipped tests to account for various use cases.

[crdavis12]

I would also get use out of this feature. One nice thing this would bring is integration into our CI system. Instead of simply showing passed/failed tests, this would allow skipped tests to be viewable in CI as well.