aqa-tests icon indicating copy to clipboard operation
aqa-tests copied to clipboard

trapme: tests failing when a testimage is not specified on a `CUSTOMIZED_SDK_URL`

Open sxa opened this issue 10 months ago • 0 comments

I've been re-running the tests suites manually on some custom builds in order to test some modified compiler options on Linux as per https://github.com/adoptium/temurin-build/issues/3685

By default if you just specify the JDK tarball a number of the tests in the sanity suite will fail - for example this was the output from sanity.openjk JDk22 without the testimage specified:

18:39:09  	jdk_lang_0 - Test results: passed: 999; failed: 4; error: 7
18:39:09  	jdk_lang_1 - Test results: passed: 999; failed: 4; error: 7
18:39:09  	jdk_lang_native_0 - Test results: error: 1
18:39:09  	jdk_lang_native_1 - Test results: error: 1
18:39:09  	jdk_foreign_0 - Test results: passed: 56; failed: 40; error: 4
18:39:09  	jdk_foreign_1 - Test results: passed: 56; failed: 40; error: 4

(Note, I'm not 100% certain all of these are caused by the missing test image but many of them are, and a subsequent run with the fix had all of these passing)

Since this is "non-obvious" when you're just trying to run a test suite I think it would be useful to trap this situation or otherwise warn the user if a testimage has not been specified and the tests may require it. The above list may be adequate for determining which suites require it. Presumably these would also be a list of "expected failures" when using someone else's third party tarball which does not include a testimage download too and therefore would not be able to pass the tests.

The description of CUSTOMIZED_SDK_URL is also Customized SDK url, need to set when SDK_RESOURCE=customized and makes no mention of the special format to specify the testimage tarball (specify it after the main jdk tarball with a space character separating them) so it would be good to update the description to make that clear. I think chaging the behavior here would be a great usability improvement to stop people tripping up and having unexpected test case failures

sxa avatar Mar 26 '24 11:03 sxa