Avoid unsteady states during JVMTI Suspend and Resume All VThreads

[babsingh]

For VThreads, the mount and unmount phases are unsteady states since a J9VMThread is modified during these phases.

JVMTI [Suspend|Resume]AllVirtualThreads acquire exclusive VM access. This can pause the virtual threads or carrier threads in the middle of the mount and unmount phases.

Effects of the above issue:

• Suspended threads might still run;
• Resuming threads might encounter threads that are not suspended; and
• JVMTI events might trigger for the suspended threads.

The above issue hasn't been reproduced due to the small gap in the mount and unmount phases.

[babsingh]

#18511 takes the CRIU approach from criusupport.cpp to address this issue.

@fengxue-IS proposed an approach where the J9VMThread is stored in VirtualThread.cont.state at virtualThreadMountBegin. With this approach,

• we will have access to the J9VMThread during the unsteady states, and it can used in JVMTI to suspend and resume virtual threads during the unsteady states
• logic for JVMTI GetThreadState will need to be updated
• API in ContinuationHelpers will need to be rewritten (e.g. the check to determine if a virtual thread is fully mounted will change)
• this is a better approach in terms of perf and will avoid timeouts which are possible in the CRIU approach, but it will take more time to implement and verify functional correctness since all major components (VM, GC and JIT) will be impacted

[babsingh]

approach where the J9VMThread is stored in VirtualThread.cont.state at virtualThreadMountBegin

Setting a Continuation field in the VirtualThread API will be a bad approach from an API design perspective since the Continuation operations will no longer be encapsulated in the Continuation class. Continuation will no longer be independent and will have to depend on VirtualThread to function correctly.

To resolve this issue, the J9VMThread address can be stored in VirtualThread.isSuspendedInternal during virtualThreadMountBegin and unset in virtualThreadMountEnd. This will achieve similar results, but it will avoid the dependency between Continuation and VirtualThread and reduce the complexity of re-working the API in ContinuationHelpers.

[babsingh]

fyi @tajila @fengxue-IS

[babsingh]

The better solution proposed above will also need to update:

• getVMThread: vthreads can be suspended in unsteady states. to perform jvmti operations after such a suspend, another code path needs to be added in getVMThread to avoid deadlocks with acquireVThreadInspector.
• getVMThread: targetThread will need to be carefully evaluated to correctly execute jvmti operations. in case of resumeThread, targetThread might be NULL, but the halt flag for the J9VMThread cached in VirtualThread.isSuspendedInternal will need to be cleared.
• we might need a new API which relies on GC CAS operations such as inlineMixedObjectCompareAndSwapU64 to prevent side-effects of concurrently updating VirtualThread.isSuspendedInternal.

[babsingh]

A fix for this issue might also fix https://github.com/eclipse-openj9/openj9/issues/18675.

[babsingh]

@fengxue-IS Any updates on this issue?

[babsingh]

A fix for this issue might also fix https://github.com/eclipse-openj9/openj9/issues/18735.

[pshipton]

We've hit the M2 date for 0.44, moving out to Java 22. We can consider if we want to make the change in 0.44 once it's ready.