Add --ignore-skip flag

[TheZoq2]

I have some long running tests which download files. I want to test them in CI but don't feel like remembering to add the -e flag every time I run runt

My initial idea was to add a exclude_by_default toml flag which would only run the tests if -i was used, but then you need to -i all tests which is also not what I want

This skip solution works for my use though it does feel like a bit of a hack

[rachitnigam]

Oh interesting! This is a bit funky because it'd require you to have both the .skip and .expect around for the same test. One style thing we use in our existing runt files (like the one for Calyx) is adding fake "tags" like "correctness" or "frontend" and then saying something like:

runt -i correctness

One possible workaround for your suite might be having such tags. It might be good to add a formal mechanism for those tags in runt as well.

[TheZoq2]

Oh interesting! This is a bit funky because it'd require you to have both the .skip and .expect around for the same test. This doesn't seem super weird to me, for example if I have a test that worked before, but I've now temporarily broken but still want to keep the old goldentest around, I would end up with .expect and .skip

Of course, in that situation, my --ignore-skip isn't very heplful though :sweat:

Tags would kind of solve the issue, but as far as I can tell, there is no mechanism to have a default set of tags, or to run all tests? Or have I missed some flag somewhere?

[rachitnigam]

You have not! Tags are entirely a naming discipline in runt and don’t really exist in any other form right now. Regarding your broken golden test example, the workflow we’ve used in the past is to simply rename the expect file to skip and go from there.

[rachitnigam]

@TheZoq2 just gauging what you think about the ad-hoc tag nonsense and if the skip-ignore flag still makes sense. If it does, let's merge it.

[TheZoq2]

For my use case, where I want to run all tests apart from long running ones, and explicitly requiring enabling those, it seems like the tag stuff doesn't work.

But using the .skip feature for that also feels like a bit of a hack imo