openj9 icon indicating copy to clipboard operation
openj9 copied to clipboard

aarch64 mac any test, "Invalid JIT return address" after completing test

Open pshipton opened this issue 3 years ago • 32 comments

https://openj9-jenkins.osuosl.org/job/Test_openjdknext_j9_sanity.system_aarch64_mac_Nightly_testList_2/9 NioLoadTest_5m_1 -XX:-UseCompressedOops

No diagnostics. https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdknext_j9_sanity.system_aarch64_mac_Nightly_testList_2/9/system_test_output.tar.gz

NLT 16:02:54.294 - Thread completed. Suite=0 thread=0
NLT 16:02:55.289 - Thread completed. Suite=0 thread=2
NLT 16:02:58.632 - Thread completed. Suite=0 thread=3
NLT 16:02:59.498 - Thread completed. Suite=0 thread=5
NLT 16:03:01.156 - Thread completed. Suite=0 thread=4
NLT 16:03:02.560 - Thread completed. Suite=0 thread=1
NLT 16:03:02.571 - Test stopped due to reaching runtime limit
NLT 16:03:02.572 - Load test completed
NLT 16:03:02.572 -   Ran     : 1072
NLT 16:03:02.573 -   Passed  : 1072
NLT 16:03:02.573 -   Failed  : 0
NLT 16:03:02.574 -   Result  : PASSED
NLT stderr 
NLT stderr 
NLT stderr *** Invalid JIT return address 000000037E8F2670 in 000000016F1977D8
NLT stderr 
STF 16:03:02.852 - **FAILED** Process NLT ended with exit code (255) and not the expected exit code/s (0)

@knn-k Similar to https://github.com/eclipse-openj9/openj9/issues/14635

pshipton avatar Mar 15 '22 13:03 pshipton

No failures in 20x Grinder job: https://openj9-jenkins.osuosl.org/job/Grinder/712/

knn-k avatar Mar 16 '22 01:03 knn-k

Another successful Grinder job (20x): https://openj9-jenkins.osuosl.org/job/Grinder/713/

knn-k avatar Mar 16 '22 06:03 knn-k

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.system_aarch64_mac_Nightly_testList_0/22 LambdaLoadTest_CS_5m_0 '-Xgcpolicy:gencon -Xgc:concurrentScavenge'

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_sanity.system_aarch64_mac_Nightly_testList_0/22/ - no diagnostics

LT  14:04:33.475 - Completed 93.4%. Number of tests started=43496 (+3171)
STF 14:04:52.123 - Heartbeat: Process LT  is still running
LT  14:04:53.354 - Thread completed. Suite=0 thread=2
LT  14:04:53.354 - Thread completed. Suite=0 thread=5
LT  14:04:53.356 - Thread completed. Suite=0 thread=1
LT  14:04:53.356 - Thread completed. Suite=0 thread=3
LT  14:04:53.356 - Thread completed. Suite=0 thread=0
LT  14:04:53.357 - Thread completed. Suite=0 thread=4
LT  14:04:53.383 - Test stopped due to reaching runtime limit
LT  14:04:53.383 - Load test completed
LT  14:04:53.384 -   Ran     : 46613
LT  14:04:53.384 -   Passed  : 46613
LT  14:04:53.384 -   Failed  : 0
LT  14:04:53.384 -   Result  : PASSED
LT  stderr 
LT  stderr 
LT  stderr *** Invalid JIT return address 000000013B882D10 in 000000016D61F1F8
LT  stderr 
STF 14:04:53.683 - **FAILED** Process LT  ended with exit code (255) and not the expected exit code/s (0)

pshipton avatar Mar 30 '22 00:03 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_special.system_aarch64_mac_Personal_testList_0/1 - mac11-aarch64-2 LambdaLoadTest_special_J9_5m_8 -Xgcpolicy:gencon -Xshareclasses -Xjit -Xnocompressedrefs

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_special.system_aarch64_mac_Personal_testList_0/1/system_test_output.tar.gz

LT  01:03:46.390 - Completed 100.0%. Number of tests started=45183 (+3008)
LT  01:03:46.391 - Thread completed. Suite=0 thread=5
LT  01:03:46.393 - Thread completed. Suite=0 thread=2
LT  01:03:46.396 - Thread completed. Suite=0 thread=0
LT  01:03:46.399 - Thread completed. Suite=0 thread=1
LT  01:03:46.399 - Thread completed. Suite=0 thread=4
LT  01:03:46.407 - Thread completed. Suite=0 thread=3
LT  01:03:46.501 - Test stopped due to reaching runtime limit
LT  01:03:46.501 - Load test completed
LT  01:03:46.502 -   Ran     : 45183
LT  01:03:46.502 -   Passed  : 45183
LT  01:03:46.502 -   Failed  : 0
LT  01:03:46.502 -   Result  : PASSED
LT  stderr 
LT  stderr 
LT  stderr *** Invalid JIT return address 000000037D2BE818 in 000000016F2F31F8
LT  stderr 
STF 01:03:46.864 - **FAILED** Process LT  ended with exit code (255) and not the expected exit code/s (0)

pshipton avatar Apr 11 '22 12:04 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.system_aarch64_mac_Nightly_testList_2/32 - mac11-aarch64-2 DBBLoadTest_5m_1 -XX:-UseCompressedOops

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk11_j9_extended.system_aarch64_mac_Nightly_testList_2/32/system_test_output.tar.gz

DBLT 15:28:23.295 - Test stopped due to reaching runtime limit
DBLT 15:28:23.295 - Load test completed
DBLT 15:28:23.296 -   Ran     : 1201
DBLT 15:28:23.297 -   Passed  : 1201
DBLT 15:28:23.297 -   Failed  : 0
DBLT 15:28:23.298 -   Result  : PASSED
DBLT stderr 
DBLT stderr 
DBLT stderr *** Invalid JIT return address 000000014D754EA0 in 000000017023BDB8

pshipton avatar Apr 12 '22 12:04 pshipton

Is it better to exclude the test for the time being?

(Edit) Ah, it is not a specific test that fail with "Invalid JIT return address".

knn-k avatar Apr 12 '22 12:04 knn-k

It's not one particular test, every failure listed here is a different suite. I think they should keep running.

pshipton avatar Apr 12 '22 12:04 pshipton

It's interesting the failure only seems to occur after the test load is finished and passed.

pshipton avatar Apr 12 '22 12:04 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdk18_j9_extended.functional_aarch64_mac_Nightly_testList_1/33 - mac11-aarch64-6 threadMXBeanTestSuite2_0 -Xjit -Xgcpolicy:gencon -Xnocompressedrefs

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk18_j9_extended.functional_aarch64_mac_Nightly_testList_1/33/functional_test_output.tar.gz

GeneralTest suite
Total tests run: 18, Failures: 0, Skips: 0
===============================================



*** Invalid JIT return address 000000037FB06C60 in 000000016D803D98

pshipton avatar Apr 13 '22 12:04 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.system_aarch64_mac_Nightly_testList_0/37 LambdaLoadTest_CS_5m_0 -Xgcpolicy:gencon -Xgc:concurrentScavenge

LT  16:28:46.032 - Test stopped due to reaching runtime limit
LT  16:28:46.032 - Load test completed
LT  16:28:46.032 -   Ran     : 52304
LT  16:28:46.032 -   Passed  : 52304
LT  16:28:46.033 -   Failed  : 0
LT  16:28:46.033 -   Result  : PASSED
LT  stderr 
LT  stderr 
LT  stderr *** Invalid JIT return address 000000037A38DCA0 in 000000016FAE3218

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.system_aarch64_mac_Nightly_testList_1/37 MiniMix_5m_1 -XX:-UseCompressedOops

LT  16:07:29.658 - Test stopped due to reaching runtime limit
LT  16:07:29.658 - Load test completed
LT  16:07:29.658 -   Ran     : 60695
LT  16:07:29.658 -   Passed  : 60695
LT  16:07:29.658 -   Failed  : 0
LT  16:07:29.658 -   Result  : PASSED
LT  stderr 
LT  stderr 
LT  stderr *** Invalid JIT return address 000000013C818C70 in 000000016C145AD8

pshipton avatar Apr 19 '22 12:04 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdknext_j9_extended.functional_aarch64_mac_Nightly_testList_0/41 threadMXBeanTestSuite2_0 -Xjit -Xgcpolicy:gencon -Xnocompressedrefs

GeneralTest suite
Total tests run: 18, Failures: 0, Skips: 0
===============================================

*** Invalid JIT return address 000000037FFA65C0 in 000000016D9CFD98

pshipton avatar May 02 '22 14:05 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.system_aarch64_mac_Nightly_testList_1/49 ConcurrentLoadTest_5m_1

LT  16:10:49.961 -   Result  : PASSED
LT  stderr 
LT  stderr 
LT  stderr *** Invalid JIT return address 000000037FB081A8 in 000000016B396678

pshipton avatar May 06 '22 11:05 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_aarch64_mac_Nightly/66/ jdk_lang_VarHandleTest_j9_1 -Xjit:enableAggressiveLiveness -XX:-JITServerTechPreviewMessage -Xjit:scratchSpaceFactorWhenJSR292Workload=1 -XX:-UseCompressedOops

19:47:40  Test results: passed: 38
19:47:40  Report written to /Users/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_aarch64_mac_Nightly/jvmtest/openjdk/report/html/report.html
19:47:40  Results written to /Users/jenkins/workspace/Test_openjdk17_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/output_16539541205245/jdk_lang_VarHandleTest_j9_1/work
19:47:40  
19:47:40  
19:47:40  *** Invalid JIT return address 0000000124F5DAA0 in 000000016D553D98
19:47:40  
19:47:40  
19:47:40  jdk_lang_VarHandleTest_j9_1_FAILED

pshipton avatar May 31 '22 13:05 pshipton

DDR is enabled in the nightly build at https://openj9-jenkins.osuosl.org/job/Build_JDK17_aarch64_mac_Nightly/68/, but core file is unavailable for this case.

knn-k avatar Jun 01 '22 03:06 knn-k

https://openj9-jenkins.osuosl.org/job/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/72/ jdk_lang_1 java/lang/ProcessBuilder/Basic.java#id0

20:38:39  STDOUT:
20:38:39  Testing ASCII environment values
20:38:39  Testing Latin1 environment values
20:38:39  Testing Unicode environment values
20:38:39  Testing directory .
20:38:39  Testing directory ..
20:38:39  Testing directory /
20:38:39  Testing directory /bin
20:38:39  cmdp
20:38:39  cmdp 0: /Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/openjdkbinary/j2sdk-image/bin/java
20:38:39  cmdp 1: -XX:+DisplayVMOutputToStderr
20:38:39  cmdp 2: -classpath
20:38:39  cmdp 3: /Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/output_16543029449355/jdk_lang_1/work/classes/1/java/lang/ProcessBuilder/Basic_id0.d:/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/lang/ProcessBuilder:/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/output_16543029449355/jdk_lang_1/work/classes/1/test/lib:/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/openjdk/openjdk-jdk/test/lib:/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/jvmtest/openjdk/jtreg/lib/javatest.jar:/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/jvmtest/openjdk/jtreg/lib/jtreg.jar
20:38:39  cmdp 4: Basic$JavaChild
20:38:39  cmdp 5: System.getenv()
20:38:39  envp
20:38:39  envp 0: LC_ALL=C
20:38:39  envp 1: FO=BR
20:38:39   waitFor process: delta: 21875
20:38:39  
20:38:39  Passed = 9050, failed = 2
20:38:39  
20:38:39  STDERR:
20:38:39  >'255'< not equal to '0'
20:38:39  java.lang.Throwable
20:38:39  	at java.base/java.lang.Thread.dumpStack(Thread.java:525)
20:38:39  	at Basic.fail(Basic.java:2847)
20:38:39  	at Basic.fail(Basic.java:2848)
20:38:39  	at Basic.equal(Basic.java:2854)
20:38:39  	at Basic.testIORedirection(Basic.java:1213)
20:38:39  	at Basic.realMain(Basic.java:1326)
20:38:39  	at Basic.main(Basic.java:2857)
20:38:39  	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
20:38:39  	at java.base/java.lang.reflect.Method.invoke(Method.java:577)
20:38:39  	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
20:38:39  	at java.base/java.lang.Thread.run(Thread.java:862)
20:38:39  >'standard error
20:38:39  
20:38:39  *** Invalid JIT return address 000000037FE2CE38 in 000000016B6837C8
20:38:39  
20:38:39  '< not equal to 'standard error'
20:38:39  java.lang.Throwable
20:38:39  	at java.base/java.lang.Thread.dumpStack(Thread.java:525)
20:38:39  	at Basic.fail(Basic.java:2847)
20:38:39  	at Basic.fail(Basic.java:2848)
20:38:39  	at Basic.equal(Basic.java:2854)
20:38:39  	at Basic.testIORedirection(Basic.java:1215)
20:38:39  	at Basic.realMain(Basic.java:1326)
20:38:39  	at Basic.main(Basic.java:2857)
20:38:39  	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
20:38:39  	at java.base/java.lang.reflect.Method.invoke(Method.java:577)
20:38:39  	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
20:38:39  	at java.base/java.lang.Thread.run(Thread.java:862)
20:38:39  WARNING: A terminally deprecated method in java.lang.System has been called
20:38:39  WARNING: System::setSecurityManager has been called by Basic (file:/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/output_16543029449355/jdk_lang_1/work/classes/1/java/lang/ProcessBuilder/Basic_id0.d/)
20:38:39  WARNING: Please consider reporting this to the maintainers of Basic
20:38:39  WARNING: System::setSecurityManager will be removed in a future release
20:38:39  java.lang.AssertionError: Some tests failed
20:38:39  	at Basic.main(Basic.java:2859)
20:38:39  	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
20:38:39  	at java.base/java.lang.reflect.Method.invoke(Method.java:577)
20:38:39  	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
20:38:39  	at java.base/java.lang.Thread.run(Thread.java:862)

pshipton avatar Jun 06 '22 12:06 pshipton

No core file is available for java/lang/ProcessBuilder/Basic.java#id0 in the artifact of Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/72/ above. There is a core file in aqa-tests/TKG/output_16543029449355/jdk_lang_1/work/java/lang/invoke/VarHandles/VarHandleTestMethodTypeInt/ instead. VarHandleTestMethodTypeInt test passed.

knn-k avatar Jun 07 '22 04:06 knn-k

I don't think any of these failures cause a core file. It may be best to modify the code that prints "Invalid JIT return address" to cause a core file on amac.

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/74

22:31:57  ===============================================
22:31:57  Running test jdk_lang_1 ...
22:31:57  ===============================================
22:31:57  jdk_lang_1 Start Time: Thu Jun  9 12:31:57 2022 Epoch Time (ms): 1654741917258
22:31:57  "/Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/openjdkbinary/j2sdk-image/bin/java" -Xshareclasses:destroyAll; "/Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/openjdkbinary/j2sdk-image/bin/java" -Xshareclasses:groupAccess,destroyAll; echo "cache cleanup done";
22:31:57  
22:31:57  Attempting to destroy all caches in cacheDir /Users/jenkins/javasharedresources/
22:31:57  
22:31:57  JVMSHRC807I Non-compressed references persistent shared cache "sharedcc_jenkins" has been destroyed. Use option -Xcompressedrefs if you want to destroy a compressed references cache.
22:31:58  JVMSHRC005I No shared class caches available
22:31:58  cache cleanup done
22:31:58  variation: -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage Mode650
22:31:58  JVM_OPTIONS:  -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -XX:-UseCompressedOops 
22:31:58  { itercnt=1; \
22:31:58  	mkdir -p "/Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/../TKG/output_16547419164137/jdk_lang_1"; \
22:31:58  	cd "/Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/../TKG/output_16547419164137/jdk_lang_1"; \
22:31:58  	"/Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/openjdkbinary/j2sdk-image/bin/java" -Xmx512m -jar "/Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/../../jvmtest/openjdk/jtreg/lib/jtreg.jar" \
22:31:58  	-agentvm -a -ea -esa -v:fail,error,time,nopass -retain:fail,error,*.dmp,javacore.*,heapdump.*,*.trc -ignore:quiet -timeoutFactor:8 -xml:verify -concurrency:4  -nativepath:"/Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/openjdkbinary/openjdk-test-image/jdk/jtreg/native" -vmoptions:"-Xmx512m  -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -XX:-UseCompressedOops  " \
22:31:58  	-timeoutHandler:jtreg.openj9.CoreDumpTimeoutHandler -timeoutHandlerDir:"/Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/../TKG/lib/openj9jtregtimeouthandler.jar" \
22:31:58  	-w ""/Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/../TKG/output_16547419164137/jdk_lang_1"/work" \
22:31:58  	-r "/Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/../../jvmtest/openjdk/report" \
22:31:58  	-jdk:"/Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/openjdkbinary/j2sdk-image" \
22:31:58  	-exclude:"/Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/../openjdk/openjdk-jdk/test/jdk/ProblemList.txt" \
22:31:58  	-exclude:"/Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/../../jvmtest/openjdk/excludes/ProblemList_openjdk11-openj9.txt" \
22:31:58  	 \
22:31:58  	 \
22:31:58  	"/Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/../openjdk/openjdk-jdk/test/jdk:jdk_lang"; \
22:31:58  	if [ $? -eq 0 ] ; then echo ""; echo "jdk_lang_1""_PASSED"; echo ""; cd /Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/..;  else echo ""; echo "jdk_lang_1""_FAILED"; echo ""; fi; } 2>&1 | tee -a "/Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/../TKG/output_16547419164137/TestTargetResult";
22:31:58  Directory "/Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/../TKG/output_16547419164137/jdk_lang_1/work" not found: creating
22:31:58  Directory "/Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/../../jvmtest/openjdk/report" not found: creating
22:32:01  XML output with verification to /Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/output_16547419164137/jdk_lang_1/work
22:36:09  Test results: passed: 727
22:36:09  Report written to /Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/jvmtest/openjdk/report/html/report.html
22:36:09  Results written to /Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/output_16547419164137/jdk_lang_1/work
22:36:09  
22:36:09  
22:36:09  *** Invalid JIT return address 000000014A6423D8 in 000000016F2177E8

pshipton avatar Jun 09 '22 15:06 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdk18_j9_sanity.system_aarch64_mac_Nightly_testList_2/74 NioLoadTest_5m_1

pshipton avatar Jun 13 '22 13:06 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.system_aarch64_mac_Nightly_testList_2/78 ConcurrentLoadTest_5m_1

LT  stderr *** Invalid JIT return address 000000037FC08138 in 000000016B586678

knn-k avatar Jun 15 '22 04:06 knn-k

https://openj9-jenkins.osuosl.org/job/Test_openjdk18_j9_extended.system_aarch64_mac_Nightly_testList_1/78 SharedClassesWorkload_0

STF 13:01:24.332 - **FAILED** Process SCL2 ended with exit code (255) and not the expected exit code/s (0)

15.SCL2.stderr

*** Invalid JIT return address 000000037F1A13B0 in 000000016BAB31F8

pshipton avatar Jun 15 '22 15:06 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/87 jdk_lang_1

21:20:24  Test results: passed: 727
21:20:24  Report written to /Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/jvmtest/openjdk/report/html/report.html
21:20:24  Results written to /Users/jenkins/workspace/Test_openjdk11_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/output_16564653751219/jdk_lang_1/work
21:20:24  
21:20:24  
21:20:24  *** Invalid JIT return address 000000014CCE2350 in 000000016D9C77E8

pshipton avatar Jun 29 '22 13:06 pshipton

I wonder if the following is this issue. The tests passed, but there was a hang.

https://openj9-jenkins.osuosl.org/job/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/89 jdk_util_1 java/util/HashMap/TreeBinAssert.java

21:22:57  rerun:
21:22:57  cd /Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/output_16564644159337/jdk_util_1/work/scratch/3 && \
21:22:57  HOME=/Users/jenkins \
21:22:57  PATH=/bin:/usr/bin:/usr/sbin \
21:22:57      /Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/openjdkbinary/j2sdk-image/bin/javac \
21:22:57          -J-ea \
21:22:57          -J-esa \
21:22:57          -J-Xmx1540m \
21:22:57          -J-Xdump:system:none \
21:22:57          -J-Xdump:heap:none \
21:22:57          -J-Xdump:system:events=gpf+abort+traceassert+corruptcache \
21:22:57          -J-XX:-UseCompressedOops \
21:22:57          -J-Djava.library.path=/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/openjdkbinary/openjdk-test-image/jdk/jtreg/native \
21:22:57          -J-Dtest.vm.opts='-ea -esa -Xmx1540m -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-UseCompressedOops' \
21:22:57          -J-Dtest.tool.vm.opts='-J-ea -J-esa -J-Xmx1540m -J-Xdump:system:none -J-Xdump:heap:none -J-Xdump:system:events=gpf+abort+traceassert+corruptcache -J-XX:-UseCompressedOops' \
21:22:57          -J-Dtest.compiler.opts= \
21:22:57          -J-Dtest.java.opts= \
21:22:57          -J-Dtest.jdk=/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/openjdkbinary/j2sdk-image \
21:22:57          -J-Dcompile.jdk=/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/openjdkbinary/j2sdk-image \
21:22:57          -J-Dtest.timeout.factor=8.0 \
21:22:57          -J-Dtest.nativepath=/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/openjdkbinary/openjdk-test-image/jdk/jtreg/native \
21:22:57          -J-Dtest.root=/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk \
21:22:57          -J-Dtest.name=java/util/HashMap/TreeBinAssert.java \
21:22:57          -J-Dtest.file=/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/util/HashMap/TreeBinAssert.java \
21:22:57          -J-Dtest.src=/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/util/HashMap \
21:22:57          -J-Dtest.src.path=/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/util/HashMap \
21:22:57          -J-Dtest.classes=/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/output_16564644159337/jdk_util_1/work/classes/2/java/util/HashMap/TreeBinAssert.d \
21:22:57          -J-Dtest.class.path=/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/output_16564644159337/jdk_util_1/work/classes/2/java/util/HashMap/TreeBinAssert.d \
21:22:57          -J-Dtest.class.path.prefix=/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/output_16564644159337/jdk_util_1/work/classes/2/java/util/HashMap/TreeBinAssert.d:/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/util/HashMap \
21:22:57          -d /Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/output_16564644159337/jdk_util_1/work/classes/2/java/util/HashMap/TreeBinAssert.d \
21:22:57          -sourcepath /Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/util/HashMap \
21:22:57          -classpath /Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/util/HashMap:/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/TKG/output_16564644159337/jdk_util_1/work/classes/2/java/util/HashMap/TreeBinAssert.d:/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/jvmtest/openjdk/jtreg/lib/testng.jar:/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/jvmtest/openjdk/jtreg/lib/jcommander.jar:/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/jvmtest/openjdk/jtreg/lib/guice.jar /Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/util/HashMap/TreeBinAssert.java
21:22:57  direct:
21:22:57  Note: /Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/util/HashMap/TreeBinAssert.java uses unchecked or unsafe operations.
21:22:57  Note: Recompile with -Xlint:unchecked for details.
21:22:57  
21:22:57  ACTION: testng -- Error. Program `/Users/jenkins/workspace/Test_openjdk18_j9_sanity.openjdk_aarch64_mac_Nightly/openjdkbinary/j2sdk-image/bin/java' timed out (timeout set to 960000ms, elapsed time including timeout handling was 960983ms).
21:22:57  REASON: User specified action: run testng/othervm -esa TreeBinAssert 
21:22:57  TIME:   960.985 seconds
21:22:57  messages:
21:22:57  command: testng -esa TreeBinAssert
21:22:57  reason: User specified action: run testng/othervm -esa TreeBinAssert 
21:22:57  Mode: othervm [/othervm specified]
21:22:57  Timeout information:
21:22:57  Running jcmd on process 45033
21:22:57  Error getting data from 45033: Exception connecting to 45033
21:22:57  Usage : jcmd <vmid> <arguments>
21:22:57  
21:22:57     -J : supply arguments to the Java VM running jcmd
21:22:57     -l : list JVM processes on the local machine
21:22:57     -h : print this help message
21:22:57  
21:22:57     <vmid> : Attach API VM ID as shown in jps or other Attach API-based tools
21:22:57  
21:22:57     arguments:
21:22:57        help : print the list of diagnostic commands
21:22:57        help <command> : print help for the specific command
21:22:57        <command> [command arguments] : command from the list returned by "help"
21:22:57  
21:22:57  list JVM processes on the local machine. Default behavior when no options are specified.
21:22:57  
21:22:57  NOTE: this utility might significantly affect the performance of the target JVM.
21:22:57      The available diagnostic commands are determined by
21:22:57      the target VM and may vary between VMs.
21:22:57  Error getting data from 45033: Exception connecting to 45033
21:22:57  Usage : jcmd <vmid> <arguments>
21:22:57  
21:22:57     -J : supply arguments to the Java VM running jcmd
21:22:57     -l : list JVM processes on the local machine
21:22:57     -h : print this help message
21:22:57  
21:22:57     <vmid> : Attach API VM ID as shown in jps or other Attach API-based tools
21:22:57  
21:22:57     arguments:
21:22:57        help : print the list of diagnostic commands
21:22:57        help <command> : print help for the specific command
21:22:57        <command> [command arguments] : command from the list returned by "help"
21:22:57  
21:22:57  list JVM processes on the local machine. Default behavior when no options are specified.
21:22:57  
21:22:57  NOTE: this utility might significantly affect the performance of the target JVM.
21:22:57      The available diagnostic commands are determined by
21:22:57      the target VM and may vary between VMs.
21:22:57  Running jstack on process 45033
21:22:57  2022-06-29T11:22:54.151100
21:22:57  Error getting data from 45033: Exception connecting to 45033
21:22:57  --- Timeout information end.
21:22:57  elapsed time (seconds): 960.985
21:22:57  configuration:
21:22:57  STDOUT:
21:22:57  config TreeBinAssert.checkAssertionStatus(): success
21:22:57  test TreeBinAssert.testMap(524288, [I@2d5aac16): success
21:22:57  test TreeBinAssert.testMap(1048576, [I@7d0d3e9): success
21:22:57  test TreeBinAssert.testSet(524288, [I@1252ffeb): success
21:22:57  test TreeBinAssert.testSet(1048576, [I@486c8197): success
21:22:57  
21:22:57  ===============================================
21:22:57  java/util/HashMap/TreeBinAssert.java
21:22:57  Total tests run: 4, Passes: 4, Failures: 0, Skips: 0
21:22:57  ===============================================

pshipton avatar Jun 29 '22 14:06 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_aarch64_mac_Nightly_testList_1/98 JCL_Test_none_SCC_1

pshipton avatar Jul 14 '22 12:07 pshipton

I ran JCL_Test_none_SCC_1 in my local environment, but I could not reproduce the failure.

Not reproduced in https://openj9-jenkins.osuosl.org/job/Grinder/1135/ (30x runs), either.

knn-k avatar Jul 15 '22 02:07 knn-k

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.system_aarch64_mac_Nightly_testList_2/101 NioLoadTest_5m_1

pshipton avatar Jul 19 '22 13:07 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_extended.functional_aarch64_mac_aot_Personal_testList_1/30 threadMXBeanTestSuite2_0

~Looks like no diagnostics for this one (the file is too small), but updated the previous comment with the diagnostics link.~ No diagnostics for any of these. https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk11_j9_extended.functional_aarch64_mac_aot_Personal_testList_1/30/functional_test_output.tar.gz

pshipton avatar Jul 19 '22 13:07 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_aarch64_mac_Release/12 jdk_util_1

16:18:15  *** Invalid JIT return address 0000000127DADA80 in 000000016D353D98
16:18:15  
16:18:15  
16:18:15  jdk_util_1_FAILED

pshipton avatar Jul 19 '22 20:07 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.system_aarch64_mac_Nightly_testList_0/104 LambdaLoadTest_CS_5m_0

pshipton avatar Jul 21 '22 13:07 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.system_aarch64_mac_Nightly_testList_0/104 DBBLoadTest_5m_1

I don't think there is any point in continuing to report these. They seemingly occur after any test, and there are no diagnostics produced to investigate further.

pshipton avatar Jul 22 '22 11:07 pshipton

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.system_aarch64_mac_Nightly_testList_2/113 NioLoadTest_5m_1

https://openj9-jenkins.osuosl.org/job/Test_openjdk18_j9_extended.system_aarch64_mac_Nightly_testList_0/113 DBBLoadTest_5m_1

pshipton avatar Aug 04 '22 16:08 pshipton