aarch64 mac Unhandled exception in signal handler. Protected function: generateDiagnosticFiles (0x0)

[pshipton]

As noticed in https://github.com/eclipse-openj9/openj9/issues/15833#issuecomment-1245750025, and seen in other amac issues as well (https://github.com/eclipse-openj9/openj9/issues/15518, https://github.com/eclipse-openj9/openj9/issues/14998). There were likely other occurrences but the text didn't get copied into the issue.

This seems to be a problem which is preventing the creation of diagnostic files.

[pshipton]

@tajila fyi

[tajila]

@mikezhang1234567890 can you please take a look at this as well

[mikezhang1234567890]

Looking at the core files that were created in https://github.com/eclipse-openj9/openj9/issues/15833 for test failures for SharedClasses.SCM23.SingleCL_0 and SharedClasses.SCM23.MultiCL_0, the errors occur during VM shutdown.

`lldb` stack trace for one of the failing threads:

  thread #5
    frame #0: 0x00000001a223fddc libsystem_kernel.dylib`__wait4 + 8
    frame #1: 0x0000000104a04304 libj9prt29.dylib`omrdump_create(portLibrary=<unavailable>, filename="/Users/jenkins/workspace/Grinder/aqa-tests/TKG/output_16637145475961/SharedClasses.SCM23.MultiCL_0/20220920-185605-SharedClasses/results/core.20220920.190337.69077.0001.dmp", dumpType=<unavailable>, userData=<unavailable>) at omrosdump.c:549:3 [opt]
    frame #2: 0x0000000104bdebfc libj9dmp29.dylib`doSystemDump(agent=0x0000000155204930, label="/Users/jenkins/workspace/Grinder/aqa-tests/TKG/output_16637145475961/SharedClasses.SCM23.MultiCL_0/20220920-185605-SharedClasses/results/core.20220920.190337.69077.0001.dmp", context=0x000000016b90cbe8) at dmpagent.c:756:12 [opt]
    frame #3: 0x0000000104be3980 libj9dmp29.dylib`protectedDumpFunction(portLibrary=0x00000001047dde28, userData=0x000000016b90c700) at dmpagent.c:2852:16 [opt]
    frame #4: 0x0000000104a058f8 libj9prt29.dylib`omrsig_protect(portLibrary=0x00000001047dde28, fn=(libj9dmp29.dylib`protectedDumpFunction at dmpagent.c:2849), fn_arg=0x000000016b90c700, handler=(libj9dmp29.dylib`signalHandler at dmpagent.c:2861:2), handler_arg=0x0000000000000000, flags=505, result=<unavailable>) at omrsignal.c:425:12 [opt]
    frame #5: 0x0000000104be3664 libj9dmp29.dylib`runDumpAgent [inlined] runDumpFunction(agent=0x0000000155204930, label="/Users/jenkins/workspace/Grinder/aqa-tests/TKG/output_16637145475961/SharedClasses.SCM23.MultiCL_0/20220920-185605-SharedClasses/results/core.20220920.190337.69077.0001.dmp", context=0x000000016b90cbe8) at dmpagent.c:2830:21 [opt]
    frame #6: 0x0000000104be3630 libj9dmp29.dylib`runDumpAgent(vm=0x0000000155811a20, agent=0x0000000155204930, context=0x000000016b90cbe8, state=0x000000016b90cc20, detail="", timeNow=<unavailable>) at dmpagent.c:2760:13 [opt]
    frame #7: 0x0000000104bf89b4 libj9dmp29.dylib`triggerDumpAgents(vm=<unavailable>, self=<unavailable>, eventFlags=8192, eventData=<unavailable>) at trigger.c:1010:7 [opt]
    frame #8: 0x0000000104835730 libj9vm29.dylib`generateDiagnosticFiles(portLibrary=<unavailable>, userData=<unavailable>) at gphandle.c:1158:3 [opt]
    frame #9: 0x0000000104a058f8 libj9prt29.dylib`omrsig_protect(portLibrary=0x00000001047dde28, fn=(libj9vm29.dylib`generateDiagnosticFiles at gphandle.c:1136), fn_arg=0x000000016b90d528, handler=(libj9vm29.dylib`recursiveCrashHandler at gphandle.c:1278), handler_arg=0x000000016b90d510, flags=505, result=<unavailable>) at omrsignal.c:425:12 [opt]
    frame #10: 0x0000000104834af0 libj9vm29.dylib`structuredSignalHandler(portLibrary=0x00000001047dde28, gpType=24, gpInfo=<unavailable>, userData=<unavailable>) at gphandle.c:833:2 [opt]
    frame #11: 0x0000000104a06ed4 libj9prt29.dylib`mainSynchSignalHandler(signal=11, sigInfo=0x000000016b90e4c0, contextInfo=0x000000016b90e528) at omrsignal.c:1066:14 [opt]
    frame #12: 0x00000001a22bec44 libsystem_platform.dylib`_sigtramp + 56
    frame #13: 0x0000000104c4c514 libj9trc29.dylib`threadStop(thr=0x0000000000000010) at trcmain.c:0:4 [opt]
    frame #14: 0x0000000104c4c514 libj9trc29.dylib`threadStop(thr=0x00000001448825e0) at trcmain.c:624:4 [opt]
    frame #15: 0x0000000104c462a4 libj9trc29.dylib`hookThreadEnd [inlined] reportTraceEvent(vm=<unavailable>, self=0x0000000144881b00, eventFlags=2) at trcengine.c:817:25 [opt]
    frame #16: 0x0000000104c46240 libj9trc29.dylib`hookThreadEnd(hook=<unavailable>, eventNum=<unavailable>, eventData=<unavailable>, userData=<unavailable>) at trcengine.c:928:3 [opt]
    frame #17: 0x00000001049d9994 libj9hookable29.dylib`J9HookDispatch(hookInterface=0x0000000155813c78, taggedEventNum=<unavailable>, eventData=0x000000016b90eb50) at hookable.cpp:235:5 [opt]
    frame #18: 0x000000010487953c libj9vm29.dylib`::threadCleanup(vmThread=0x0000000144881b00, forkedByVM=0) at vmthread.cpp:428:2 [opt]
    frame #19: 0x0000000104840afc libj9vm29.dylib`protectedDetachCurrentThread(portLibrary=0x00000001047dde28, userData=<unavailable>) at jniinv.c:587:2 [opt]
    frame #20: 0x0000000104a058f8 libj9prt29.dylib`omrsig_protect(portLibrary=0x00000001047dde28, fn=(libj9vm29.dylib`protectedDetachCurrentThread at jniinv.c:581), fn_arg=0x0000000144881b00, handler=(libj9vm29.dylib`structuredSignalHandler at gphandle.c:698), handler_arg=0x0000000144881b00, flags=506, result=<unavailable>) at omrsignal.c:425:12 [opt]
    frame #21: 0x00000001048402a8 libj9vm29.dylib`DetachCurrentThread(javaVM=<unavailable>) at jniinv.c:619:7 [opt]
    frame #22: 0x000000010904c28c libj9jit29.dylib`protectedCompilationThreadProc((null)=<unavailable>, compInfoPT=0x0000000138981030) at CompilationThread.cpp:3966:7 [opt]
    frame #23: 0x0000000104a058f8 libj9prt29.dylib`omrsig_protect(portLibrary=0x00000001047dde28, fn=(libj9jit29.dylib`protectedCompilationThreadProc(J9PortLibrary*, TR::CompilationInfoPerThread*) at CompilationThread.cpp:3861), fn_arg=0x0000000138981030, handler=(libj9vm29.dylib`structuredSignalHandler at gphandle.c:698), handler_arg=0x0000000144881b00, flags=506, result=<unavailable>) at omrsignal.c:425:12 [opt]
    frame #24: 0x000000010904a680 libj9jit29.dylib`compilationThreadProc(entryarg=0x0000000138981030) at CompilationThread.cpp:3839:25 [opt]
    frame #25: 0x0000000104a8263c libj9thr29.dylib`thread_wrapper(arg=0x000000015580c060) at omrthread.c:1733:2 [opt]
    frame #26: 0x00000001a2273878 libsystem_pthread.dylib`_pthread_start + 320

lldb gives p/x vm->runtimeFlags as (U_32) $2 = 0x00e000ee, which means J9_RUNTIME_SHUTDOWN_STARTED and J9_RUNTIME_EXIT_STARTED are set, indicating the JVM is shutting down.

From the above, the JIT thread receives a signal due to not shutting down cleanly, which invokes vmSignalHandler, which then invokes generateDiagnosticFiles. However, the VM is freeing resources during JVM shutdown so it is likely that the generateDiagnosticFiles call will further segfault and prevent the core from being generated.

The errors all occur with JIT threads during shutdown, and the https://github.com/eclipse-openj9/openj9/pull/15907 fixing the underlying issue probably also fixes this one.

[pshipton]

I heard talk of a "work around". Is there any benefit in considering that, in case there are future problems with JIT threads during shutdown?

[babsingh]

Potential workaround: -XX:allowvmshutdown:false (https://www.eclipse.org/openj9/docs/xxallowvmshutdown). @mikezhang1234567890 Did your grinder pass with the above option? It should use the JDK, which causes the failure.

[mikezhang1234567890]

No, even with -XX:allowvmshutdown:false we still get failiures. Internal grinder here. There is also an instance of the core files failing to be written in that grinder.