circt icon indicating copy to clipboard operation
circt copied to clipboard

[CI] List which tests were unsupported

Open mortbopet opened this issue 3 years ago • 3 comments

Currently, test runs in CI only shows a passed/failed/unsupported split (https://github.com/llvm/circt/actions/runs/3045345694/jobs/4906793030#step:7:68).
In case a test is picked up as unsupported, but one actually would've liked for CI to check it, it would be nice if the test run would explicitly list which tests were unsupported. This should help triaging as well as allow better finger-pointing when that set of unsupported integration tests inevitably grows.

mortbopet avatar Sep 13 '22 13:09 mortbopet

There is an llvm-lit flag to list out all the tests and their results.

teqdruid avatar Sep 13 '22 18:09 teqdruid

Assuming you refer to https://llvm.org/docs/CommandGuide/lit.html#cmdoption-lit-show-unsupported - that is obviously what we need. Stupid questions aside, the question is then how to pass that lit argument to the ninja check-circt-integration command.

mortbopet avatar Sep 13 '22 18:09 mortbopet

I haven't figured that out yet. I usually just run llvm-lit directly on the test directory in the build dir.

teqdruid avatar Sep 13 '22 18:09 teqdruid