avocado
avocado copied to clipboard
Conditionally dismiss all warnings per users configuration
Is your feature request related to a problem? Please describe.
Avocado has the test status WARN
for "keeping track of noteworthy events that ultimately do not affect the test outcome" (see the description here), which is something sensible to me. However, there is a comment indicated that "some result plugins does not support this and report PASS
instead" , which made me think about if we would better to make having the status or not configurable. Because, in case of we are going to present Avocado's test results to some result services, we may need to do the status replacement by having some external logic as the same reason. Therefore, I would like to request such a feature for conditionally dismissing all warnings by respecting users willingness.
Describe the solution you'd like
For users who do not want the WARN
status to be presented in their test results we can introduce a new option (e.g. maybe job.run.dismiss_warnings
?) for replacing all the WARN
with PASS
when the option was set, and that means they understand about the risk but just like to ignore it for a better communication experience with the downstream result services of their auto pipelines.
Describe alternatives you've considered Maybe we could even let users define the test status model by providing the mapping of Avocado defined statuses to their own, for example:
Avocado defined statuses | User defined statuses |
---|---|
WARN | PASS |
ERROR | FAIL |
SKIP | (discard) |
... | ... |
But this could be tricky in some way.
Additional information
The original issue was, after the fix of https://github.com/autotest/tp-qemu/issues/3048, we found some tests report WARN
instead of PASS
due to we changed the logger of all the tp-qemu tests from the root logger to avocado.test
and there is a particular rule (https://github.com/avocado-framework/avocado/pull/444) for handling all the warning messages. We indeed can fix this issue by demoting all the warnings, but just consider if it would be better by doing in the other way around.
@PaulYuuu @fbq815 feel free to add your comments here, thanks!