GammaRay on Windows randomly fails to load process list, burns CPU and memory

[nyanpasu64]

I built GammaRay 08a246f7986b4f024d25a81af76f225502ae4b4f on Windows using KDE's Craft (it's tricky to enable both release and debug probes, though craft --set buildReleaseAndDebug=true libs/qt5/qtbase and rebuilding qtbase/Craft seems to have worked). However I've run into an issue. Around 40% of the time when I launch GammaRay, the contents of the startup Attach tab never loads (though the UI isn't hung):

At this point, the process probing thread is stuck in an endless loop, burning a CPU core and causing system RAM usage to fluctuate unpredictably.

I traced it down to a loop calling CreateToolhelp32Snapshot() and retrying when GetLastError() == ERROR_BAD_LENGTH:

https://github.com/KDAB/GammaRay/blob/490f80a069c3e3b7f1efd4e3fef67247549151d8/launcher/core/probeabidetector_win.cpp#L201-L208

Unfortunately, the function never does succeed. When I pause at this point in Visual Studio's debugger and press "Step Out", the function never returns.

By opening this source file, then repeatedly restarting the process and breakpointing this loop in the source (maybe disassembly works too) after the hang occurred, I could sample the pids causing the hang. It seems to generally be pid 0.

Interestingly, by making the breakpoint conditional on pid == 0, I found that every time I start the program, the loop is called once with pid 0. On some program runs, the loop terminates immediately (and gets called again and terminates immediately once per second or so). On other program runs, the loop lasts forever, and snapshot is set to 0xffffffffffffffff on the first iteration, and then small multiples of 4 on subsequent iterations.

Maybe you shouldn't try probing PID 0 (System Idle Process). Reading the Microsoft docs, maybe you should call SetLastError(0) on every loop iteration, because it seems CreateToolhelp32Snapshot doesn't set the error code to 0 when it succeeds. (Maybe you should implement both changes, after verifying each one by itself fixes the hang.)

(I disabled the Windows Firewall prompt asking for GammaRay to open a port. However I don't think it's related to this bug.)

[nyanpasu64]

Interestingly, when I build GammaRay from source in Qt Creator rather than Craft, using the same compiler (MSVC x64), it never hangs. It seems CreateToolhelp32Snapshot and GetLastError() operate differently. So I'm not sure exactly what's the problem.