dynamorio icon indicating copy to clipboard operation
dynamorio copied to clipboard

TEST REPORTING BUG: Wrong issue number can be reported for failing tests that are ignored

Open xdje42 opened this issue 1 year ago • 0 comments

Describe the bug

Test runs can report the wrong issue number for ignored tests.

To Reproduce

See for example, https://github.com/DynamoRIO/dynamorio/actions/runs/7731301531/job/21078628227#step:7:84492

====> FAILURE in debug-internal-64 <====
debug-internal-64: 902 tests passed, **** 6 tests failed, of which 2 were flaky, but ignoring 3 for i#2941: ****
	(ignore: i#2941) 	code_api|client.attach_test 
	code_api|api.static_maps_mixup_novars_FLAKY 
	(ignore: i#2941) 	code_api,opt_memory|common.loglevel 
	(ignore: i#2941) 	code_api,opt_speed|common.decode-stress 
	code_api|tool.drcachesim.invariants 
	code_api|api.static_sideline_FLAKY 
WARNING: maximum warning/error limit hit for release-external-64!
  Manually verify whether it succeeded.

Note i#2941 is being reported as the issue for the attach_test failure. We'll need to augment ignored test marking to let one specify the issue so that it makes it through to the test output.

xdje42 avatar Jan 31 '24 20:01 xdje42