aqa-tests
aqa-tests copied to clipboard
trapme: tests failing when a testimage is not specified on a `CUSTOMIZED_SDK_URL`
I've been re-running the tests suites manually on some custom builds in order to test some modified compiler options on Linux as per https://github.com/adoptium/temurin-build/issues/3685
By default if you just specify the JDK tarball a number of the tests in the sanity suite will fail - for example this was the output from sanity.openjk JDk22 without the testimage specified:
18:39:09 jdk_lang_0 - Test results: passed: 999; failed: 4; error: 7
18:39:09 jdk_lang_1 - Test results: passed: 999; failed: 4; error: 7
18:39:09 jdk_lang_native_0 - Test results: error: 1
18:39:09 jdk_lang_native_1 - Test results: error: 1
18:39:09 jdk_foreign_0 - Test results: passed: 56; failed: 40; error: 4
18:39:09 jdk_foreign_1 - Test results: passed: 56; failed: 40; error: 4
(Note, I'm not 100% certain all of these are caused by the missing test image but many of them are, and a subsequent run with the fix had all of these passing)
Since this is "non-obvious" when you're just trying to run a test suite I think it would be useful to trap this situation or otherwise warn the user if a testimage has not been specified and the tests may require it. The above list may be adequate for determining which suites require it. Presumably these would also be a list of "expected failures" when using someone else's third party tarball which does not include a testimage download too and therefore would not be able to pass the tests.
The description of CUSTOMIZED_SDK_URL
is also Customized SDK url, need to set when SDK_RESOURCE=customized
and makes no mention of the special format to specify the testimage tarball (specify it after the main jdk tarball with a space character separating them) so it would be good to update the description to make that clear. I think chaging the behavior here would be a great usability improvement to stop people tripping up and having unexpected test case failures