DaaLoadTest exits unexpectedly

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk24_j9_extended.system_x86-64_mac_Nightly_testList_1/53 - mac14-x86-2 DaaLoadTest_all_5m_0 -Xjit -Xgcpolicy:gencon -Xnocompressedrefs

00:45:42  DLT 00:45:39.075 - Completed 6.7%. Number of tests started=1816
00:46:00  DLT 00:45:59.161 - Completed 13.4%. Number of tests started=4435 (+2619)
00:46:19  DLT 00:46:19.104 - Completed 20.0%. Number of tests started=7441 (+3006)
00:46:41  DLT 00:46:39.120 - Completed 26.7%. Number of tests started=10747 (+3306)
00:47:00  DLT 00:46:59.145 - Completed 33.4%. Number of tests started=13746 (+2999)
00:47:00  STF 00:47:00.121 - **FAILED** Process DLT ended with exit code (1) and not the expected exit code/s (0)

No diagnostics.

[babsingh]

@pshipton @JasonFengJ9 Have you seen this failure recently?

@hzongaro This failure is related to Dynamic Loop Transfer (DLT).

fyi @tajila

[JasonFengJ9]

This failure wasn't seen in recent JDK24 runs. Also note that DaaLoadTest variation tests failed before such as:

• https://github.com/eclipse-openj9/openj9/issues/17037
• https://github.com/eclipse-openj9/openj9/issues/15508

[tajila]

based on the comment above Ill move this out of the JDK24 milestone

[pshipton]

This doesn't just occur on jdk24 but I'm seeing it on other versions as well. One example: https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_special.system_aarch64_mac_Personal_testList_1/123 DaaLoadTest_all_special_5m_4

10:12:24  DLT 00:12:16.889 - Completed 46.8%. Number of tests started=10424 (+3592)
10:12:30  STF 00:12:29.567 - **FAILED** Process DLT ended with exit code (1) and not the expected exit code/s (0)

[hzongaro]

No luck yet trying to reproduce this in grinder runs.

[pshipton]

Haven't seen this recently.

[pshipton]

Another one.

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_special.system_ppc64le_linux_Personal_testList_0/135/

13:14:47  DLT 17:14:42.948 - Completed 53.4%. Number of tests started=17136 (+2595)
13:14:49  STF 17:14:49.075 - **FAILED** Process DLT ended with exit code (1) and not the expected exit code/s (0)

[hzongaro]

This failure is related to Dynamic Loop Transfer (DLT).

Actually, is the problem related to Dynamic Loop Transfer, or is DLT an abbreviation of DaaLoadTest in this context?

[hzongaro]

Unfortunately, there doesn't seem to be any useful information captured in the system_test_output.tar.gz about the nature of the failure reported above.

@llxia, I spent a bit of time looking through some of the System Test Framework. It looks like the message "Process DLT ended with exit code (1) and not the expected exit code. . ." is reported by ProcessMgmt.pm. I see there are some additional trace and debuglines in that file that I think are connected to the verbose and super-verbose options from to https://github.com/adoptium/STF/blob/7534c9296a7effe8499d9c236e0cc12d5bc97826/stf.core/config/stf.properties#L146-L148. Is there an easy way that I could set those options in a grinder run, or would I have to supply my own fork of STF.git that overrides those settings?

[hzongaro]

I haven't had any luck reproducing this failure in grinder runs, and we don't have much information to go on. I'm going to move this out to the Future release for now. If we see more frequent failures, we can move it back in to a scheduled release.

[llxia]

Is there an easy way that I could set those options in a grinder run, or would I have to supply my own fork of STF.git that overrides those settings?

Unfortunately, you cannot set those options in a grinder run at the moment. You can either do it in your personal STF or do it in a personal aqa test repo (set the option in command. It will be passed into stf.pl https://github.com/adoptium/aqa-tests/blob/4d9585fe185cfabceaf49b6688290db2fb90b0ee/system/daaLoadTest/playlist.xml#L45)

[hzongaro]

or do it in a personal aqa test repo (set the option in command. It will be passed into stf.pl

Thanks! I'll give that a try.

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_special.system_aarch64_mac_Personal_testList_1/66 DaaLoadTest_all_special_5m_1 -Xgcpolicy:optthruput -Xjit:count=0 -Xnocompressedrefs

[pshipton]

@babsingh question for you in https://github.com/eclipse-openj9/openj9/issues/21652#issuecomment-3197470045

[babsingh]

is the problem related to Dynamic Loop Transfer, or is DLT an abbreviation of DaaLoadTest in this context?

Based on @hzongaro’s recent comments, in this context DLT most likely refers to DaaLoadTest. Earlier, I couldn't find the test documentation that mapped DLT to DaaLoadTest. So, I had read DLT as Dynamic Loop Transfer based on the abbreviations use in the OpenJ9 repo.

[hzongaro]

I had read DLT as Dynamic Loop Transfer

Yup - me as well!

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_special.system_ppc64_aix_Personal_testList_2/173 DaaLoadTest_all_special_5m_21 -Xcompressedrefs -Xgcpolicy:gencon -Xquickstart -Xjit

[pshipton]

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_extended.system_x86-64_mac_Nightly_testList_1/882/ DaaLoadTest_all_5m_1 -Xcompressedrefs -Xjit -Xgcpolicy:gencon

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_special.system_aarch64_mac_Personal_testList_2/146 DaaLoadTest_daa1_special_5m_4 -Xgcpolicy:gencon -Xjit:count=0 -Xnocompressedrefs

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_extended.system_x86-64_mac_Nightly_testList_2/1003 DaaLoadTest_all_5m_0 -Xjit -Xgcpolicy:gencon -Xnocompressedrefs