openj9 icon indicating copy to clipboard operation
openj9 copied to clipboard

Published 20 hours ago • verified publisher icon eclipse-openj9

aarch64 mac segfault in j9gc_createJavaLangString

Open pshipton opened this issue 3 years ago • 5 comments

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.system_aarch64_mac_Nightly_testList_0/96 SharedClasses.SCM23.MultiCL_0 -Xjit -Xgcpolicy:gencon -Xnocompressedrefs

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_extended.system_aarch64_mac_Nightly_testList_0/96/system_test_output.tar.gz

MCL5 10:15:25 >> Loaded 19000 classes...
STF 10:15:27.320 - Found dump at: /Users/jenkins/workspace/Test_openjdk17_j9_extended.system_aarch64_mac_Nightly_testList_0/aqa-tests/TKG/output_16575830392423/SharedClasses.SCM23.MultiCL_0/20220712-100821-SharedClasses/results/core.19700101.000000.45317.0001.dmp
MCL1 10:15:27 >> Loaded 20000 classes...
MCL1 10:15:27 >> Total classes loaded = 20001
MCL1 stderr Unhandled exception
MCL1 stderr Type=Segmentation error vmState=0x00000000
MCL1 stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
MCL1 stderr Handler1=0000000104EF22BC Handler2=0000000105222E14 InaccessibleAddress=0000000000006348
MCL1 stderr x0=000000012F809300 x1=0000000109F37A41 x2=0000000000000006 x3=0000000000000000
MCL1 stderr x4=000000016B6CBE90 x5=000000016B6CBED8 x6=000000016B6CBED0 x7=000000016B6CBEC8
MCL1 stderr x8=0000000000000000 x9=0000000104E9EDE8 x10=000000012F818B88 x11=000000012F818BB2
MCL1 stderr x12=000000016B6CBEE8 x13=000000016B6CBEC0 x14=000000037F7D6E4C x15=000000016B6CC2A0
MCL1 stderr x16=0000000188D6BF60 x17=00000001F77A6F28 x18=000000037F7D6DC0 x19=000000012F809300
MCL1 stderr x20=0000000000000000 x21=0000000109F37A41 x22=0000000109F37A41 x23=0000000000000006
MCL1 stderr x24=000000010501630C x25=000000010A5BA2A8 x26=000000016B6CC210 x27=000000012F813E20
MCL1 stderr x28=000000012F824A68 x29(FP)=000000016B6CBE80 x30(LR)=0000000104EE9680 x31(SP)=000000016B6CBDD0
MCL1 stderr PC=000000010A5BA2E8 SP=000000016B6CBDD0
MCL1 stderr v0 07ffffffffffffff (f: 4294967296.000000, d: 3.785767e-270)
MCL1 stderr v1 0000000000000007 (f: 7.000000, d: 3.458460e-323)
MCL1 stderr v2 0706050403020100 (f: 50462976.000000, d: 7.949929e-275)
MCL1 stderr v3 3fd686c85e9b14cf (f: 1587221760.000000, d: 3.519765e-01)
MCL1 stderr v4 0000000100000000 (f: 0.000000, d: 2.121996e-314)
MCL1 stderr v5 0000000100000000 (f: 0.000000, d: 2.121996e-314)
MCL1 stderr v6 0000000000000001 (f: 1.000000, d: 4.940656e-324)
MCL1 stderr v7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
MCL1 stderr v8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
MCL1 stderr v9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
MCL1 stderr v10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
MCL1 stderr v11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
MCL1 stderr v12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
MCL1 stderr v13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
MCL1 stderr v14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
MCL1 stderr v15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
MCL1 stderr v16 bfd0000000000000 (f: 0.000000, d: -2.500000e-01)
MCL1 stderr v17 3fd56906a0555555 (f: 2689946880.000000, d: 3.345353e-01)
MCL1 stderr v18 bf73bd735a15802f (f: 1511358464.000000, d: -4.819346e-03)
MCL1 stderr v19 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
MCL1 stderr v20 00000000ffffffff (f: 4294967296.000000, d: 2.121996e-314)
MCL1 stderr v21 00000000ffffffff (f: 4294967296.000000, d: 2.121996e-314)
MCL1 stderr v22 00000000ffffffff (f: 4294967296.000000, d: 2.121996e-314)
MCL1 stderr v23 ffffffffffffffff (f: 4294967296.000000, d: nan)
MCL1 stderr v24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
MCL1 stderr v25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
MCL1 stderr v26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
MCL1 stderr v27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
MCL1 stderr v28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
MCL1 stderr v29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
MCL1 stderr v30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
MCL1 stderr v31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
MCL1 stderr Module=/Users/jenkins/workspace/Test_openjdk17_j9_extended.system_aarch64_mac_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9gc29.dylib
MCL1 stderr Module_base_address=000000010A4F8000 Symbol=j9gc_createJavaLangString
MCL1 stderr Symbol_address=000000010A5BA2A8
MCL1 stderr Target=2_90_20220712_99 (Mac OS X 11.5.2)
MCL1 stderr CPU=aarch64 (8 logical CPUs) (0x400000000 RAM)
MCL1 stderr ----------- Stack Backtrace -----------
MCL1 stderr ---------------------------------------
...
More segfaults follow

@knn-k @dmitripivkine fyi

pshipton avatar Jul 12 '22 12:07 pshipton

There is observation GC Check returns thousands of errors class pointer not in a class segment. There are many j9 classes look correct (and not unloaded) but have not returned by !allclasses There are examples of such classes:

java/lang/invoke/DirectMethodHandle
jdk/internal/misc/InnocuousThread
java/lang/Thread
java/lang/Class
jdk/internal/loader/ClassLoaders$PlatformClassLoader
...

I am not sure is it tool reporting issue or real problem. I am still looking to it.

j9gc_createJavaLangString is more VM than GC code really, adding comp:vm label as well. @tajila FYI

dmitripivkine avatar Jul 12 '22 14:07 dmitripivkine

please note core in artifacts is generated for:

> !gpinfo
Failing Thread: !j9vmthread 0x10f809d00
Failing Thread ID: 0xa019a34 (167877172)
gpInfo:
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
Handler1=0000000104EF22BC Handler2=0000000105222E14 InaccessibleAddress=0000000000006798
x0=000000010F809D00 x1=000000010F809D00 x2=000000037FD75A90 x3=0000000000000001
x4=000000010A5A4204 x5=0000000000000000 x6=0000000000000009 x7=00000000000002E0
x8=0000000000000020 x9=0000000000000000 x10=0000000110FD3900 x11=00000000021E0001
x12=00000000021E0002 x13=0000000110F64000 x14=0000000110FD3710 x15=0000000000000009
x16=0000000188D1E2A0 x17=0000000000000023 x18=0000000000000000 x19=0000000000000006
x20=00000001382061A0 x21=0000000118000000 x22=000000037FD75A90 x23=0000000000000018
x24=0000000000000001 x25=000000012F813E20 x26=0000000000000020 x27=0000000109E7E364
x28=000000016B2485D8 x29(FP)=000000016B24D8E0 x30(LR)=000000010989BB98 x31(SP)=000000016B247770
PC=000000010A5A421C SP=000000016B247770
v0 00000000000000ff (f: 255.000000, d: 1.259867e-321)
v1 ffffffffffffffff (f: 4294967296.000000, d: nan)
v2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v3 0706050403020100 (f: 50462976.000000, d: 7.949929e-275)
v4 00000000ffffffff (f: 4294967296.000000, d: 2.121996e-314)
v5 0000000000000002 (f: 2.000000, d: 9.881313e-324)
v6 0000080000000800 (f: 2048.000000, d: 4.345847e-311)
v7 000000000000000d (f: 13.000000, d: 6.422853e-323)
v8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v16 0000000000000001 (f: 1.000000, d: 4.940656e-324)
v17 0000000000000001 (f: 1.000000, d: 4.940656e-324)
v18 0000000000000001 (f: 1.000000, d: 4.940656e-324)
v19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v21 ffffffffffffffff (f: 4294967296.000000, d: nan)
v22 ffffffffffffffff (f: 4294967296.000000, d: nan)
v23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/Users/jenkins/workspace/Test_openjdk17_j9_extended.system_aarch64_mac_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9gc29.dylib
Module_base_address=000000010A4F8000 Symbol=j9gc_objaccess_mixedObjectReadObject <-------
Symbol_address=000000010A5A4204

Method_being_compiled=net/openj9/sc/classes/TestClass_9668.stringOperations(III)V <-------

dmitripivkine avatar Jul 12 '22 14:07 dmitripivkine

The !j9object 0x37FD75A90 mentioned in registers is java/lang/invoke/DirectMethodHandle

x2=000000037FD75A90

> !j9object 0x37FD75A90
!J9Object 0x000000037FD75A90 {
	struct J9Class* clazz = !j9class 0x110FD3900 // java/lang/invoke/DirectMethodHandle
	Object flags = 0x00000020;
	Ljava/lang/invoke/MethodType; type = !fj9object 0x280308578 (offset = 0) (java/lang/invoke/MethodHandle)
	Ljava/lang/invoke/LambdaForm; form = !fj9object 0x280305cc0 (offset = 8) (java/lang/invoke/MethodHandle)
	Ljava/lang/invoke/MethodHandle; asTypeCache = !fj9object 0x0 (offset = 16) (java/lang/invoke/MethodHandle)
	B customizationCount = 0x00000000 (offset = 32) (java/lang/invoke/MethodHandle)
	Z updateInProgress = 0x00000000 (offset = 36) (java/lang/invoke/MethodHandle)
	Ljava/lang/invoke/MemberName; jitVMEntryKeepAlive = !fj9object 0x0 (offset = 24) (java/lang/invoke/MethodHandle) <hidden>
	Ljava/lang/invoke/MemberName; member = !fj9object 0x2803085b0 (offset = 48) (java/lang/invoke/DirectMethodHandle)
	Z crackable = 0x00000000 (offset = 56) (java/lang/invoke/DirectMethodHandle)
	J lockword = 0x0000000000000000 (offset = 40) (java/lang/invoke/DirectMethodHandle) <hidden>
}

dmitripivkine avatar Jul 12 '22 14:07 dmitripivkine

Looking to Snap traces: looks like JVM was in Shutdown mode deeply when recorded crash occur. I am not sure was JVM shutdown triggered by another crash occur earlier or the reason for crashes is partial shutdown. For example Garbage Collector has it structures teared down partially

dmitripivkine avatar Jul 12 '22 18:07 dmitripivkine

This is a failure with SharedClasses.SCM23.MultiCL_0. Let's see if it happens or not after PR #15907 is applied.

knn-k avatar Sep 19 '22 10:09 knn-k