dynamorio
dynamorio copied to clipboard
TEST REPORTING BUG: Wrong issue number can be reported for failing tests that are ignored
Describe the bug
Test runs can report the wrong issue number for ignored tests.
To Reproduce
See for example, https://github.com/DynamoRIO/dynamorio/actions/runs/7731301531/job/21078628227#step:7:84492
====> FAILURE in debug-internal-64 <====
debug-internal-64: 902 tests passed, **** 6 tests failed, of which 2 were flaky, but ignoring 3 for i#2941: ****
(ignore: i#2941) code_api|client.attach_test
code_api|api.static_maps_mixup_novars_FLAKY
(ignore: i#2941) code_api,opt_memory|common.loglevel
(ignore: i#2941) code_api,opt_speed|common.decode-stress
code_api|tool.drcachesim.invariants
code_api|api.static_sideline_FLAKY
WARNING: maximum warning/error limit hit for release-external-64!
Manually verify whether it succeeded.
Note i#2941 is being reported as the issue for the attach_test failure. We'll need to augment ignored test marking to let one specify the issue so that it makes it through to the test output.