aqa-tests icon indicating copy to clipboard operation
aqa-tests copied to clipboard

Published 20 hours ago • verified publisher icon adoptium

Using custom target for rerun jobs

Open sophia-guo opened this issue 1 year ago • 4 comments

Rerun job with failed targets might be costly. There might be a few test cases failures but the test target might include a few hundred of test cases. Also with failed targets some original passing test cases may fail as intermittent failure or fail on some agents.

For openjdk and jck tests rerun job can be updated to use custom targets to only run failed test cases instead of test targets.

One example of running time difference https://github.com/adoptium/aqa-tests/issues/5016#issuecomment-2048026555

If the set of failed test cases is too big as jenkins parameter to trigger the job a check can be added to run the failed target instead.

Part of https://github.com/adoptium/aqa-tests/issues/5016

sophia-guo avatar Apr 18 '24 19:04 sophia-guo

Openjdk has two custom targets, jck has four custom targets.

Currently each test build can only trigger one custom target. That is if more than one custom target is needed we need to trigger rerun build more than once correspondingly. Or can **_custom target can be triggered as TESTLIST so multiple custom target can be run? ( need to invesitgate).

The other thing is the limitation of the hppt url parameter length. There may be situations that a set of test cases fail ( due to setting issues or machine issues) so the string CUSTOM_TARGET is too long as the url parameter. For this case we may either divide the CUSTOM_TARGET to small sets and trigger the rerun multiple times or just fall back to failed targets. Falling back to failed targets might make more sense as this situation most likely some test cases setting issue or environment issues.

sophia-guo avatar Apr 25 '24 19:04 sophia-guo

I think it is fair to simply handle the simple case (which are frequent), logic:

if ((test case failure list belongs to only one type of custom list) && (failed test cases string is below the URI too long limit)) then { 
    trigger rerun as a custom target 
} else {
    trigger the failed targets to be  rerun // as we currently do
}

There are many cases that match this simple case, and it would help to reduce execution time without being too complicated.

smlambert avatar Apr 25 '24 19:04 smlambert

Not sure what is the maximum of Adoptiums jenkins header size. As Jenkins header size can be set/configured we may not need to know exactly the size. In most cases failed test cases should not be too long unless some environment issues or test setting issues, for which case we may just switch back to failed targets. I would suggest set the number of testcases as 50 for now. If not good can change later.

sophia-guo avatar Apr 26 '24 15:04 sophia-guo

I would suggest set the number of testcases as 50 for now. If not good can change later.

It would be somewhat easy to test this, by finding one of the machines that throw "no route to host" error and run jdk_net and jdk_nio targets or some such (can look at triage issues to find an example).

smlambert avatar Apr 26 '24 15:04 smlambert

jdk11 java/lang/invoke/ on linux-aarch64 has 20+ testcases in ProblemList.txt, see how that goes as the number of testcases for http link length. https://ci.adoptium.net/view/Test_grinder/job/Grinder/10329/ - all passed. might only fail on test-aws-rhel76-armv8-1? https://github.com/adoptium/infrastructure/issues/1118

rerun on test-aws-rhel76-armv8-1 https://ci.adoptium.net/view/Test_grinder/job/Grinder/10330/ - passed. rerun for jdk17,21,22,23 https://ci.adoptium.net/view/Test_grinder/job/Grinder/10331/ https://ci.adoptium.net/view/Test_grinder/job/Grinder/10332/ https://ci.adoptium.net/view/Test_grinder/job/Grinder/10333/ https://ci.adoptium.net/view/Test_grinder/job/Grinder/10334/

all passed except java/lang/invoke/LFCaching/LFGarbageCollectedTest.java, which is not in the current problem list.

sophia-guo avatar Jun 10 '24 20:06 sophia-guo

try aix jdk17 java/net, java/nio https://ci.adoptium.net/view/Test_grinder/job/Grinder/10342/

sophia-guo avatar Jun 11 '24 16:06 sophia-guo

Manually edit https://ci.adoptium.net/job/Grinder/parambuild/?SDK_RESOURCE=nightly&TARGET=jdk_custom&TEST_FLAG=&UPSTREAM_TEST_JOB_NAME=&DOCKER_REQUIRED=false&ACTIVE_NODE_TIMEOUT=&VENDOR_TEST_DIRS=&EXTRA_DOCKER_ARGS=-v+%24%7BTEST_JDK_HOME%7D%3A%2Fopt%2Fjava%2Fopenjdk&TKG_OWNER_BRANCH=AdoptOpenJDK%3Amaster&OPENJ9_SYSTEMTEST_OWNER_BRANCH=eclipse-openj9%3Amaster&PLATFORM=ppc64_aix&GENERATE_JOBS=false&KEEP_REPORTDIR=true&PERSONAL_BUILD=true&ADOPTOPENJDK_REPO=https%3A%2F%2Fgithub.com%2Fadoptium%2Faqa-tests.git&RERUN_ITERATIONS=0&LABEL=&EXTRA_OPTIONS=&CUSTOMIZED_SDK_URL=&BUILD_IDENTIFIER=&ADOPTOPENJDK_BRANCH=master&LIGHT_WEIGHT_CHECKOUT=false&NON_AQA_TEST_REPOS=&ARTIFACTORY_SERVER=&KEEP_WORKSPACE=false&USER_CREDENTIALS_ID=&JDK_VERSION=17&ITERATIONS=1&VENDOR_TEST_REPOS=&JDK_REPO=&OPENJ9_BRANCH=master&OPENJ9_SHA=&JCK_GIT_REPO=&VENDOR_TEST_BRANCHES=&OPENJ9_REPO=https%3A%2F%2Fgithub.com%2Feclipse%2Fopenj9.git&UPSTREAM_JOB_NAME=&CLOUD_PROVIDER=&PLATFORM_AND_MACHINE=&CUSTOM_TARGET=java%2Fbeans%2FIntrospector%2F8132566%2FOverridePropertyInfoTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverrideUserDefPropertyInfoTest.java+java%2Fnet%2FDatagramSocket%2FSendReceiveMaxSize.java+java%2Fnet%2FInet4Address%2FPingThis.java+java%2Fnet%2FMulticastSocket%2FB6427403.java+java%2Fnet%2FMulticastSocket%2FNoLoopbackPackets.java+java%2Fnet%2FMulticastSocket%2FPromiscuous.java+java%2Fnet%2FMulticastSocket%2FSetLoopbackMode.java+java%2Fnet%2FMulticastSocket%2FSetOutgoingIf.java+java%2Fnet%2FMulticastSocket%2FTest.java+java%2Fnet%2FSocketPermission%2FSocketPermissionTest.java+java%2Fnio%2Fchannels%2FDatagramChannel%2FAfterDisconnect.java+java%2Fnio%2Fchannels%2FDatagramChannel%2FSendReceiveMaxSize.java+java%2Fnio%2Fchannels%2FAsyncCloseAndInterrupt.java+sun%2Fmanagement%2Fjdp%2FJdpDefaultsTest.java+sun%2Fmanagement%2Fjdp%2FJdpJmxRemoteDynamicPortTest.java+sun%2Fmanagement%2Fjdp%2FJdpOffTest.java+sun%2Fmanagement%2Fjdp%2FJdpSpecificAddressTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverridePropertyInfoTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverrideUserDefPropertyInfoTest.java+java%2Fnet%2FDatagramSocket%2FSendReceiveMaxSize.java+java%2Fnet%2FInet4Address%2FPingThis.java+java%2Fnet%2FMulticastSocket%2FB6427403.java+java%2Fnet%2FMulticastSocket%2FNoLoopbackPackets.java+java%2Fnet%2FMulticastSocket%2FPromiscuous.java+java%2Fnet%2FMulticastSocket%2FSetLoopbackMode.java+java%2Fnet%2FMulticastSocket%2FSetOutgoingIf.java+java%2Fnet%2FMulticastSocket%2FTest.java+java%2Fnet%2FSocketPermission%2FSocketPermissionTest.java+java%2Fnio%2Fchannels%2FDatagramChannel%2FAfterDisconnect.java+java%2Fnio%2Fchannels%2FDatagramChannel%2FSendReceiveMaxSize.java+java%2Fnio%2Fchannels%2FAsyncCloseAndInterrupt.java+sun%2Fmanagement%2Fjdp%2FJdpDefaultsTest.java+sun%2Fmanagement%2Fjdp%2FJdpJmxRemoteDynamicPortTest.java+sun%2Fmanagement%2Fjdp%2FJdpOffTest.java+sun%2Fmanagement%2Fjdp%2FJdpSpecificAddressTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverridePropertyInfoTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverrideUserDefPropertyInfoTest.java+java%2Fnet%2FDatagramSocket%2FSendReceiveMaxSize.java+java%2Fnet%2FInet4Address%2FPingThis.java+java%2Fnet%2FMulticastSocket%2FB6427403.java+java%2Fnet%2FMulticastSocket%2FNoLoopbackPackets.java+java%2Fnet%2FMulticastSocket%2FPromiscuous.java+java%2Fnet%2FMulticastSocket%2FSetLoopbackMode.java+java%2Fnet%2FMulticastSocket%2FSetOutgoingIf.java+java%2Fnet%2FMulticastSocket%2FTest.java+java%2Fnet%2FSocketPermission%2FSocketPermissionTest.java+java%2Fnio%2Fchannels%2FDatagramChannel%2FAfterDisconnect.java+java%2Fnio%2Fchannels%2FDatagramChannel%2FSendReceiveMaxSize.java+java%2Fnio%2Fchannels%2FAsyncCloseAndInterrupt.java+sun%2Fmanagement%2Fjdp%2FJdpDefaultsTest.java+sun%2Fmanagement%2Fjdp%2FJdpJmxRemoteDynamicPortTest.java+sun%2Fmanagement%2Fjdp%2FJdpOffTest.java+sun%2Fmanagement%2Fjdp%2FJdpSpecificAddressTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverridePropertyInfoTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverrideUserDefPropertyInfoTest.java+java%2Fnet%2FDatagramSocket%2FSendReceiveMaxSize.java+java%2Fnet%2FInet4Address%2FPingThis.java+java%2Fnet%2FMulticastSocket%2FB6427403.java+java%2Fnet%2FMulticastSocket%2FNoLoopbackPackets.java+java%2Fnet%2FMulticastSocket%2FPromiscuous.java+java%2Fnet%2FMulticastSocket%2FSetLoopbackMode.java+java%2Fnet%2FMulticastSocket%2FSetOutgoingIf.java+java%2Fnet%2FMulticastSocket%2FTest.java+java%2Fnet%2FSocketPermission%2FSocketPermissionTest.java+java%2Fnio%2Fchannels%2FDatagramChannel%2FAfterDisconnect.java+java%2Fnio%2Fchannels%2FDatagramChannel%2FSendReceiveMaxSize.java+java%2Fnio%2Fchannels%2FAsyncCloseAndInterrupt.java+sun%2Fmanagement%2Fjdp%2FJdpDefaultsTest.java+sun%2Fmanagement%2Fjdp%2FJdpJmxRemoteDynamicPortTest.java+sun%2Fmanagement%2Fjdp%2FJdpOffTest.java+sun%2Fmanagement%2Fjdp%2FJdpSpecificAddressTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverridePropertyInfoTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverrideUserDefPropertyInfoTest.java+java%2Fnet%2FDatagramSocket%2FSendReceiveMaxSize.java+java%2Fnet%2FInet4Address%2FPingThis.java+java%2Fnet%2FMulticastSocket%2FB6427403.java+java%2Fnet%2FMulticastSocket%2FNoLoopbackPackets.java+java%2Fnet%2FMulticastSocket%2FPromiscuous.java+java%2Fnet%2FMulticastSocket%2FSetLoopbackMode.java+java%2Fnet%2FMulticastSocket%2FSetOutgoingIf.java+java%2Fnet%2FMulticastSocket%2FTest.java+java%2Fnet%2FSocketPermission%2FSocketPermissionTest.java+java%2Fnio%2Fchannels%2FDatagramChannel%2FAfterDisconnect.java+java%2Fnio%2Fchannels%2FDatagramChannel%2FSendReceiveMaxSize.java+java%2Fnio%2Fchannels%2FAsyncCloseAndInterrupt.java+sun%2Fmanagement%2Fjdp%2FJdpDefaultsTest.java+sun%2Fmanagement%2Fjdp%2FJdpJmxRemoteDynamicPortTest.java+sun%2Fmanagement%2Fjdp%2FJdpOffTest.java+sun%2Fmanagement%2Fjdp%2FJdpSpecificAddressTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverridePropertyInfoTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverrideUserDefPropertyInfoTest.java+java%2Fnet%2FDatagramSocket%2FSendReceiveMaxSize.java+java%2Fnet%2FInet4Address%2FPingThis.java+java%2Fnet%2FMulticastSocket%2FB6427403.java+java%2Fnet%2FMulticastSocket%2FNoLoopbackPackets.java+java%2Fnet%2FMulticastSocket%2FPromiscuous.java&VENDOR_TEST_SHAS=&JDK_BRANCH=&TEST_IMAGES_REQUIRED=true&LABEL_ADDITION=&ARTIFACTORY_REPO=&ARTIFACTORY_ROOT_DIR=&UPSTREAM_TEST_JOB_NUMBER=&DOCKERIMAGE_TAG=&JDK_IMPL=hotspot&SSH_AGENT_CREDENTIAL=&AUTO_DETECT=true&DYNAMIC_COMPILE=false&ADOPTOPENJDK_SYSTEMTEST_OWNER_BRANCH=adoptium%3Amaster&APPLICATION_OPTIONS=&CUSTOMIZED_SDK_URL_CREDENTIAL_ID=&ARCHIVE_TEST_RESULTS=false&OPENJDK_SHA=&NUM_MACHINES=&BUILD_LIST=openjdk&USE_TESTENV_PROPERTIES=false&UPSTREAM_JOB_NUMBER=&STF_OWNER_BRANCH=adoptium%3Amaster&TIME_LIMIT=10&JVM_OPTIONS=&PARALLEL=None works ( testcases 97) add one more testcases java%2Fnet%2FMulticastSocket%2FNoLoopbackPackets.java will exceed the maximum length. Considered some test case length may be longer and we really don't need to be that accurate as the failure testcases number is rarely large in normal case. Probably the test case number 80 is big enough.

sophia-guo avatar Jun 12 '24 01:06 sophia-guo