Vector35
Function taints `rbp` for unknown reason causing bad analysis
In the following screenshot, we can see the variable at rbp-0x8 is NOT annotated with var_10, which is the stack variable associated with the offset.
Looking at the MLIL, I see the variable is translated to
6 @ 00401a00 int32_t rax_1 = [rbp_1 - 8].d
which eventually leads to the awkward HLIL that uses rbp_1[-1]:
Related file: mazeware.zip
Credit: https://ctfd.nusgreyhats.org/, source code: https://github.com/NUSGreyhats/greyctf24-challs-public/tree/main/quals/rev/mazeware/src-please-do-not-read-this
the problem here is that rbp is being tainted by sub_4017cd its unknown if that is intentional or its a bug. This can be fixed by removing rbp from the clobbered registers in sub_4017cd
If we look at the prologue of sub_4017cd, we can see that we created a variable var_8 for the saved rbp. Which is quite unusual, since we usually just treat it as __saved_rbp.
I do not know why this happens, but chances are this is related to how the function is returning:
Clearly, the function is playing some tricks, and this is not the normal function epilog