[BUG] Service Commander crashes (Java dump) after move to Power10

[chrjorgensen]

Describe the bug

We've just finished moving some of our LPAR's from a Power9 server to Power10, and now Service Commander creates a Java dump.

When running the command sc list we get the following output:

→ sc list
Unhandled exception
Type=Illegal instruction vmState=0x00000000
J9Generic_Signal_Number=00000048 Signal_Number=00000004 Error_Value=00000000 Signal_Code=0000001e
Handler1=08001000A003AC08 Handler2=08001000A0011ED0
R0=0000000000000028 R1=000000018005B7E0 R2=0000000000000010 R3=00000007FFF00258
R4=00000007FFF078D8 R5=0000000000000038 R6=0000000000000000 R7=7FFFFFFF00000000
R8=0700000025BE78E8 R9=0000000008000000 R10=0000000000000001 R11=0000000030027230
R12=0000000024008842 R13=0000000180067800 R14=0000000030027700 R15=0000000030027B00
R16=0000000000000000 R17=FFFFFFFFFFFFFFFF R18=090000000AD23A70 R19=000000018005BC08
R20=000000018005BA40 R21=09001000A095AB10 R22=090000000AA9CD04 R23=09001000A095AB10
R24=000000018005BB00 R25=09001000A095AB10 R26=00000001835DD658 R27=000000018005BBC8
R28=00000001800DEC10 R29=0000000000000000 R30=00000007FFF078D8 R31=000000000000000A
IAR=0000000183D2A910 LR=0000000183B2C8BC MSR=800000002288F032 CTR=0000000183B2C6CC
CR=0000000004000000 FPSCR=8200000000000000 XER=0400000082000000
FPR0 0000000082000000 (f: 2181038080.000000, d: 1.077576e-314)
FPR1 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
FPR2 41cdcd6500000000 (f: 0.000000, d: 1.000000e+09)
FPR3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR5 c3e0000000000000 (f: 0.000000, d: -9.223372e+18)
FPR6 bfe0000000000000 (f: 0.000000, d: -5.000000e-01)
FPR7 412e848000000000 (f: 0.000000, d: 1.000000e+06)
FPR8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR9 4530000000000000 (f: 0.000000, d: 1.934281e+25)
FPR10 412e848000000000 (f: 0.000000, d: 1.000000e+06)
FPR11 43300000000f4240 (f: 1000000.000000, d: 4.503600e+15)
FPR12 4530000000000000 (f: 0.000000, d: 1.934281e+25)
FPR13 0000000000000001 (f: 1.000000, d: 4.940656e-324)
FPR14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR16 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR17 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR18 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR19 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR21 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR22 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
FPR31 0000000000000000 (f: 0.000000, d: 0.000000e+00)

Compiled_method=unknown (In JIT code segment 0000000183A27938 but no method found)
Target=2_90_20220209_53 (OS400 7.3)
CPU=ppc64 (48 logical CPUs) (0x12100000000 RAM)
----------- Stack Backtrace -----------
initializeAttachedThreadImpl+0x3d4 (0x090000000927B1D8 [libj9vm29.so+0x6b1d8])
initializeAttachedThread+0x98 (0x0900000009281D9C [libj9vm29.so+0x71d9c])
standardInit+0x714 (0x090000000B1522F8 [libjclse29.so+0x462f8])
J9VMDllMain+0x33c (0x090000000B19F560 [libjclse29.so+0x93560])
runJ9VMDllMain@AF84_52+0x108 (0x0900000009227A6C [libj9vm29.so+0x17a6c])
pool_do+0x108 (0x090000000923698C [libj9vm29.so+0x2698c])
IPRA.$runInitializationStage+0xd8 (0x09000000092246DC [libj9vm29.so+0x146dc])
protectedInitializeJavaVM+0xe60 (0x090000000921AEE4 [libj9vm29.so+0xaee4])
omrsig_protect+0x4a0 (0x0900000009522584 [libj9prt29.so+0x57584])
initializeJavaVM+0x21c (0x0900000009219AA0 [libj9vm29.so+0x9aa0])
J9_CreateJavaVM+0xb8 (0x090000000923C33C [libj9vm29.so+0x2c33c])
JNI_CreateJavaVM_impl+0x948 (0x09000000091D200C [libjvm.so+0x2800c])
JNI_CreateJavaVM+0x1c4 (0x0900000009197E88 [libjvm.so+0x1ce88])
JNI_CreateJavaVM+0x1c4 (0x0900000009168E88 [libjvm.so+0x1ce88])
JavaMain+0xec (0x00000001000067D0 [java+0x67d0])
ThreadJavaMain+0xc (0x000000010000D750 [java+0xd750])
_pthread_body+0x100 (0x0900000000328E24 [libpthreads.a+0x3e24])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2023/03/18 18:23:14 - please wait.
JVMDUMP032I JVM requested System dump using '/home/<user>/core.20230318.182314.862.0001.dmp' in response to an event
Note: "Enable full CORE dump" in smit is set to FALSE and as a result there will be limited threading information in core file.
JVMDUMP010I System dump written to /home/<user>/core.20230318.182314.862.0001.dmp
JVMDUMP032I JVM requested Java dump using '/home/<user>/javacore.20230318.182314.862.0002.txt' in response to an event
JVMDUMP010I Java dump written to /home/<user>/javacore.20230318.182314.862.0002.txt
JVMDUMP032I JVM requested Snap dump using '/home/<user>/Snap.20230318.182314.862.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /home/<user>/Snap.20230318.182314.862.0003.trc
JVMDUMP032I JVM requested JIT dump using '/home/<user>/jitdump.20230318.182314.862.0004.dmp' in response to an event
JVMDUMP051I JIT dump occurred in 'main' thread 0x0000000030027B00
JVMDUMP010I JIT dump written to /home/<user>/jitdump.20230318.182314.862.0004.dmp
JVMDUMP013I Processed dump event "gpf", detail "".

Uninstalling the OpenJDK 11 EA from yum was not an option, since this also removed Service Commander, which has OpenJDK as a requirement.

The server move was done by creating a VM on Power10 to use the same storage as the VM on Power9. So the storage was not moved, only the CPU (and instruction set) was changed.

Please advice - I've never seen this before!

To Reproduce

Expected behavior

Verbose output Run the command with the -v option and share the output here.

Additional context Add any other context about the problem here.