avocado icon indicating copy to clipboard operation
avocado copied to clipboard

Conditionally dismiss all warnings per users configuration

Open luckyh opened this issue 2 years ago • 1 comments

Is your feature request related to a problem? Please describe. Avocado has the test status WARN for "keeping track of noteworthy events that ultimately do not affect the test outcome" (see the description here), which is something sensible to me. However, there is a comment indicated that "some result plugins does not support this and report PASS instead" , which made me think about if we would better to make having the status or not configurable. Because, in case of we are going to present Avocado's test results to some result services, we may need to do the status replacement by having some external logic as the same reason. Therefore, I would like to request such a feature for conditionally dismissing all warnings by respecting users willingness.

Describe the solution you'd like For users who do not want the WARN status to be presented in their test results we can introduce a new option (e.g. maybe job.run.dismiss_warnings?) for replacing all the WARN with PASS when the option was set, and that means they understand about the risk but just like to ignore it for a better communication experience with the downstream result services of their auto pipelines.

Describe alternatives you've considered Maybe we could even let users define the test status model by providing the mapping of Avocado defined statuses to their own, for example:

Avocado defined statuses User defined statuses
WARN PASS
ERROR FAIL
SKIP (discard)
... ...

But this could be tricky in some way.

Additional information The original issue was, after the fix of https://github.com/autotest/tp-qemu/issues/3048, we found some tests report WARN instead of PASS due to we changed the logger of all the tp-qemu tests from the root logger to avocado.test and there is a particular rule (https://github.com/avocado-framework/avocado/pull/444) for handling all the warning messages. We indeed can fix this issue by demoting all the warnings, but just consider if it would be better by doing in the other way around.

luckyh avatar Mar 22 '22 11:03 luckyh

@PaulYuuu @fbq815 feel free to add your comments here, thanks!

luckyh avatar Mar 22 '22 11:03 luckyh