ivmai
GC_suspend can hang attempting to suspend DllMain thread
It seems that in some cases, the SuspendThread call within GC_suspend may hang and never return if it is trying to suspend the thread that is running DllMain to detach itself: https://github.com/ivmai/bdwgc/blob/b435e76287dbf275d922a0a3c432efc913c8460e/win32_threads.c#L457
This seems related to the previously fixed issue: #704. Again I am running bdwgc in the neko vm on Windows with DllMain enabled, with this commit: 5658d91c8dd3d9d7cb19bc447d5b0a9f4d4563f2, modified with a few debug print statements.
I was previously debugging by adding printf statements, however, I have now run a proper debugger on the program and removed the print statements to avoid their potential side effects. It seems that without modification, gc.dll actually hangs on GC_win32_MessageBoxA, which it attempts to run due to ResumeThread failed in suspend loop. This error happens because it runs ResumeThread on a NULL handle. It looks like between the SuspendThread and the ResumeThread, the thread handle has started closing the thread and set it to NULL.
I'm not sure why MessageBoxA hangs since the message box isn't actually showing, but I guess GC_suspend shouldn't abort on the ResumeThread failure if it is caused by a NULL handle.
Should be fixed now (but introducing a potential resource leak as mentioned in https://github.com/ivmai/bdwgc/pull/712#issuecomment-2725616155).
Let me keep it open as the current solution has some drawback and need to be reworked.
On behalf of @tobil4sk (copied from PR #712): It seems that in these cases where we suspend a thread running DllMain and lose the handle so are unable to resume, it can prevent new threads from being able to enter DllMain at all, so attempting to create a new thread doesn't work and the thread never starts running.
On behalf of @tobil4sk (copied from PR #712): Alternatively, maybe DllMain shouldn't run CloseHandle at all, but instead mark the thread with some flag so that it gets closed by the gc thread the next time it iterates through the thread table?