Callback function causes crash on latest RubyInstaller (e.g. 2.7.5)

[ashie]

Currently a test is omitted due to a crash bug on latest RubyInstaller such as v2.7.5:

https://github.com/cosmo0920/win32-api/runs/4948175686?check_suite_focus=true#step:5:63

2022-01-26T07:36:27.1610716Z -- C level backtrace information -------------------------------------------
2022-01-26T07:36:27.1669065Z C:\Windows\SYSTEM32\ntdll.dll(NtWaitForSingleObject+0x14) [0x00007ffdee1ffa74]
2022-01-26T07:36:27.1697069Z C:\Windows\System32\KERNELBASE.dll(WaitForSingleObjectEx+0x93) [0x00007ffdead985c3]
2022-01-26T07:36:27.1698753Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_vm_bugreport+0x34b) [0x00007ffdb8bc78bb]
2022-01-26T07:36:27.1703138Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_bug_for_fatal_signal+0x7f) [0x00007ffdb89e04cf]
2022-01-26T07:36:27.1708470Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(ruby_sysinit+0x790) [0x00007ffdb8b214c0]
2022-01-26T07:36:27.1725023Z  [0x00007ff6f3d71f32]
2022-01-26T07:36:27.1726769Z C:\Windows\System32\msvcrt.dll(_C_specific_handler+0x98) [0x00007ffdedd08068]
2022-01-26T07:36:27.1727500Z C:\Windows\SYSTEM32\ntdll.dll(_chkstk+0x11f) [0x00007ffdee204a2f]
2022-01-26T07:36:27.1728861Z C:\Windows\SYSTEM32\ntdll.dll(RtlWalkFrameChain+0x14bf) [0x00007ffdee164cef]
2022-01-26T07:36:27.1730164Z C:\Windows\SYSTEM32\ntdll.dll(KiUserExceptionDispatcher+0x2e) [0x00007ffdee20379e]
2022-01-26T07:36:27.1731281Z C:\Windows\System32\msvcrt.dll(strlen+0x31) [0x00007ffdedd3d721]
2022-01-26T07:36:27.1738352Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_str_new_cstr+0x1b) [0x00007ffdb8b3c74b]
2022-01-26T07:36:27.1738817Z  [0x00007ffdb94e36c5]
2022-01-26T07:36:27.1752100Z  [0x00007ffdb94e4b74]
2022-01-26T07:36:27.1754341Z C:\Windows\System32\USER32.dll(EnumChildWindows+0x18d) [0x00007ffded53082d]
2022-01-26T07:36:27.1755107Z C:\Windows\System32\USER32.dll(EnumWindows+0x25) [0x00007ffded530695]
2022-01-26T07:36:27.1755517Z  [0x00007ffdb94e21d3]
2022-01-26T07:36:27.1757371Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_error_arity+0x353) [0x00007ffdb8b9fd33]
2022-01-26T07:36:27.1759110Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_add_method+0x359) [0x00007ffdb8bb61c9]
2022-01-26T07:36:27.1760986Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_vm_exec+0x231) [0x00007ffdb8baa8d1]
2022-01-26T07:36:27.1762926Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_yield_force_blockarg+0x68d) [0x00007ffdb8bae40d]
2022-01-26T07:36:27.1764557Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_block_call_kw+0x2f8) [0x00007ffdb8ba5248]
2022-01-26T07:36:27.1766193Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_catch+0xe4) [0x00007ffdb8ba54d4]
2022-01-26T07:36:27.1768201Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_error_arity+0x353) [0x00007ffdb8b9fd33]
2022-01-26T07:36:27.1769841Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_add_method+0x359) [0x00007ffdb8bb61c9]
2022-01-26T07:36:27.1771793Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_vm_exec+0x231) [0x00007ffdb8baa8d1]
2022-01-26T07:36:27.1773569Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_yield_force_blockarg+0x68d) [0x00007ffdb8bae40d]
2022-01-26T07:36:27.1775181Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_block_call_kw+0x2f8) [0x00007ffdb8ba5248]
2022-01-26T07:36:27.1776703Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_catch+0xe4) [0x00007ffdb8ba54d4]
2022-01-26T07:36:27.1778605Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_error_arity+0x353) [0x00007ffdb8b9fd33]
2022-01-26T07:36:27.1780251Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_add_method+0x359) [0x00007ffdb8bb61c9]
2022-01-26T07:36:27.1781922Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_vm_exec+0x231) [0x00007ffdb8baa8d1]
2022-01-26T07:36:27.1783646Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_proc_call+0x81) [0x00007ffdb8aca3f1]
2022-01-26T07:36:27.1785256Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_ec_error_print+0x620) [0x00007ffdb89e8700]
2022-01-26T07:36:27.1786843Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_ec_error_print+0x9e7) [0x00007ffdb89e8ac7]
2022-01-26T07:36:27.1788117Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(rb_ec_error_print+0xca0) [0x00007ffdb89e8d80]
2022-01-26T07:36:27.1789805Z C:\hostedtoolcache\windows\Ruby\2.7.5\x64\bin\x64-msvcrt-ruby270.dll(ruby_run_node+0x9f) [0x00007ffdb89e980f]
2022-01-26T07:36:27.1790336Z  [0x00007ff6f3d727f9]
2022-01-26T07:36:27.1790745Z  [0x00007ff6f3d713b1]
2022-01-26T07:36:27.1808565Z  [0x00007ff6f3d714e6]
2022-01-26T07:36:27.1810018Z C:\Windows\System32\KERNEL32.DLL(BaseThreadInitThunk+0x14) [0x00007ffdeb5e7974]
2022-01-26T07:36:27.1817396Z 

I'm investigating it.

[ashie]

It doesn't crash when I use RubyInstaller 2.7.2.

[ashie]

The crash is occurred when a callback tries to access to the second argument of EnumWindowsProc.

https://github.com/cosmo0920/win32-api/blob/9886db6d7f4d917864b9b544055c04993eac694e/test/test_win32_api_callback.rb#L17

https://github.com/cosmo0920/win32-api/blob/9886db6d7f4d917864b9b544055c04993eac694e/ext/win32/api.c#L626-L629

https://github.com/cosmo0920/win32-api/blob/9886db6d7f4d917864b9b544055c04993eac694e/test/test_win32_api_callback.rb#L56

The pointer is valid when calling EnumWIndows: https://github.com/cosmo0920/win32-api/blob/9886db6d7f4d917864b9b544055c04993eac694e/ext/win32/api.c#L900-L903

but become invalid on calling the callback function: https://github.com/cosmo0920/win32-api/blob/9886db6d7f4d917864b9b544055c04993eac694e/ext/win32/api.c#L602-L603

[ashie]

It seems that one of the reason is that mismatched signature of callback function. The signature of EnumWindowProc is BOOL EnumWindowsProc(HWND, lParam) but defined signature in win32-api is DWORD func(DWORD, DWORD) https://github.com/cosmo0920/win32-api/blob/9886db6d7f4d917864b9b544055c04993eac694e/ext/win32/api.c#L688-L689

When I use same signature with EnumWindowsProc, the test passed.

[ashie]

I confirmed that it occurs from RubyInstaller 2.7.3 (for 2.7 series).

[ashie]

Since win32-api cannot specify signature of callback function, it seems hard to fix. Migrating to Fiddle might be better solution than adding such feature to win32-api.

[ashie]

For Fluentd, probably callback feature isn't used so that it isn't affected immediately.