Decompilation of a large function (a WindowProc with many switch case statements) is stuck in an infinite loop in 11.2 but not in 11.1.2

[gynt]

Describe the bug Decompilation of a large function (a WindowProc with many switch case statements) is stuck in an infinite loop in 11.2 but not in 11.1.2. Ever since the upgrade to 11.2 I cannot decompile this function anymore. 11.3.1 doesn't solve the issue.

To Reproduce Steps to reproduce the behavior:

1. Go to the WindowProc function in the listing
2. Watch the decompiler get stuck forever

Expected behavior I expect the function to decompile in roughly 30 seconds like it did in 11.1.2

Screenshots I don't think that would help

Attachments I cannot attach the software itself.

Environment (please complete the following information):

• OS: Windows 11
• Java Version: 21
• Ghidra Version: 11.2
• Ghidra Origin: official GitHub distro's

Additional context

1. I can copy paste the decompile.exe and sleigh.exe from version 11.1.2 to 11.2's folder and then the decompiler works again. So the issue really is in either of these .exe files. (Note: using 11.1.2's exe's in 11.3.1 doesn't work: Low-level Error: Unknown metatype: enum_uint)
2. For some colleagues, the Task Manager reports 0% CPU usage, and the ghidra GUI stops showing the "Decompiling" waiting GUI visual.

If I can help figure out the issue that would be great, such as debugging the decompiler itself? I wouldn't know where to start right now.

[gynt]

Okay, so, walking through every decompiler related commit from 11.1.2 to 11.2, I found that commit aac3e5ad1ce70c5db136e17fd50ca17aff4e47cd breaks the decompilation of my function, in other words, commit a31c4033a8d5224328d5196dcdb8a084cf81c2a9 is the last one that works.

My issue seems therefore be caused by: GP-4782_PtrsubUndo

[gynt]

I also just ran two VerySleepy analyses (https://github.com/VerySleepy/verysleepy/releases/tag/v0.91) and I attach them here. One for each git sha. Note that I had to start the analysis manually, reacting to the ghidra GUI showing decompilation, because I know no better way. So the two analyses aren't exact matches in any case. Maybe it still helps.

ghidra-7997-01.zip

[fmagin]

You can use the "Debug Function Decompilation" Action in the decompiler component menu to dump an XML file that allows reproducing the decompilation without the Java part of Ghidra being in any way involved. This simplifies profiling (especially with IDE integration).

If you can share that others can also reproduce it without needing the binary.

[gynt]

@fmagin Unfortunately, "Debug Function Decompilation" is not available while the GUI is trying to decompile the function.

[gynt]

Since 11.2 introduced recursive functions, I am inclined to call this issue an "infinite loop" situation of the decompiler. I hope it is accurate.

[gynt]

This is a Debug Function Decompilation xml file created with a31c4033a8d5224328d5196dcdb8a084cf81c2a9 WindowMsgProcessingFunc-Debug-Function-Decompilation-a31c4033.zip

How do I continue solving this issue from here?

[axd1x8a]

Same problem happens on 11.3.1 and 11.3.2, the decompiler gets stuck in an infinite loop when decompiling a 3kb function after setting the first argument to the correct type, it's also quite large with a size of 148kb so that might be part of the problem too. Here is the result of “Debug Function Decompilation”:

debug.zip

[gynt]

Can you try to see if reverting to https://github.com/NationalSecurityAgency/ghidra/commit/a31c4033a8d5224328d5196dcdb8a084cf81c2a9 solves it for you? The way I did this was:

1. get 11.2 from the Github releases page, use this version to launch Ghidra 11.2. let's call this A
2. checkout the ghidra github repo at the github sha a31c403. Let's call this B
3. go to the Ghidra/Features/Decompiler folder in B and copy the src/ folder to the same location in A.
4. In A, run the gradle build command
5. Move the newly compiled files from the build directory into the corresponding folder in the os/ directory
6. Use folder A to launch ghidra and go to the function in question. Hopefully it decompiles.

[gynt]

Is there anything we can do to help solve this issue? It is pretty annoying to not be able to decompile a function!

[dt-12345]

Here's an example I have that's similar to what FeeeeK brought up where setting the correct parameter type causes the decompiler to get stuck in a seemingly infinite loop: inf_loop.zip (this is the output when setting the param_3 to void* - setting it to the correct type of Scene* triggers the infinite loop). I'm on 11.3 so using any earlier version of the decompiler gives me an unknown metatype error so I can't easily test if the older versions work properly.

[aboood40091]

Is there anything we can do to help solve this issue? It is pretty annoying to not be able to decompile a function!

Seconded. It's seriously more common of an issue than it should be. I'm not able to decompile many, many functions due to this bug.

[pr0me]

hey y'all, could you check out https://github.com/NationalSecurityAgency/ghidra/pull/8277 and see if that fixes things for you (while preserving the types etc. that you'd expect)?

[gynt]

I can confirm my issue has been solved in 11.4.1