nodejs
keep up to date with flaky tags
we need to keep up to date with flaky tags as I fear that we are often marking modules flaky without doing much research and often then leaving them marked as flaky despite the fact that they may be fixed now. I propose we add a tag to citgm-all such as citgm-all --check-flaky which reports all the modules at the end that are passing but are marked as flaky. Then we can hopefully prevent this from being a problem.
I've done periodic checking of flakyness. We are about due for another.
@MylesBorins how have you currently been doing this? would it not be easier to have a tidy report at the end of the output?
Our output currently reports flaky modules. I've just been running CI with forcing things to fail for each release line and checking
I like the idea of always letting you know somehow (possibly at the end) what the results (pass/fail) were for the flaky tests. I don't think it needs to be a flag though, if it's only a couple of lines somewhere I can't see a reason you wouldn't want that info.
@gibfahn are you suggesting that we report passing "flaky" module differently than passing "non-flaky" modules?
No the way I view it is that at the end of the citgm run there will be an output summarising any modules that are marked as flaky but are actually passing
Yes I think we are actually, is there any harm to implementing this? It means that we catch fixed modules quicker