__usercall and __userpurge failing with the stack

[Gemini-Loboto3]

I'm currently working on an injection dll that uses clang with your extension, but I think I found a pretty huge bug with how you need to define functions that require stack pushes. These are the prototypes:

EXTERN_C void __userpurge sub_4B8340(DWORD* a1@<edi>, int frame);
EXTERN_C void __usercall sub_4C2A60(int a1@<eax>, DWORD a2, DWORD a3, VERTC* a4, float a5);

Basically you'd expect them to assign the first parameter to a register, then the rest as pushes on the stack, but for whatever reason the extension just assigns them as stack variables and fails to recognize stack adjustments for __usercall after it's called, breaking the call stack whenever I go back to the original code. Am I missing something that I need to specify in order to make those other parameters are pushable? I'm currently using inlined assembly to make sure the calls behave properly, but it's not optimal:

	__asm
	{
		mov eax, dword ptr[frame]
		push eax
		mov edi, dword ptr[dword_6F6CBC]
		call sub_4B8340
	}

	__asm
	{
		lea eax, dword ptr[z]
		push eax
		lea eax, dword ptr[v]
		push eax
		mov eax, dword ptr[dword_6F6CC8]
		push eax
		push 0x2000
		mov eax, 1
		call sub_4C2A60
		add esp, 0x10
	}

This is what the extension produces:

 mov         eax,dword ptr ds:[006F6CB4h]  
 mov         ecx,dword ptr [eax+0Ch]  
 mov         edi,dword ptr ds:[6F6CBCh]  
 mov         eax,esp  
 mov         dword ptr [eax],ecx  
 call        sub_4B8340@8 (0F9C100Ah)

 fld         dword ptr [z]  
 lea         edx,[v]  
 mov         ecx,dword ptr ds:[6F6CC8h]  
 mov         eax,esp  
 fstp        dword ptr [eax+0Ch]  
 mov         dword ptr [eax+8],edx  
 mov         dword ptr [eax+4],ecx  
 mov         dword ptr [eax],2000h  
 mov         eax,1  
 call        sub_4C2A60@20 (0FF91014h)  
 sub         esp,10h 

NOTE: the code doesn't cause any issue if I'm overriding a function that takes mixed registers and stack as parameters, it's when I call the original code that it causes trouble and ultimately fails.

EDIT: nevermind, it seems to fail with all functions that declare something to be accessed on the stack. Just tried with this and it can't read the float parameters:

void __usercall Set_fog_mode(float a4, float a5, int a1@<eax>, int a2@<edx>, int a3@<ebx>)
{
}

a4 and a5 have completely random values.

[widberg]

Thanks for opening an issue; sorry I took so long getting to this, I haven't checked the repository in a while and GitHub didn't feel like sending me a notification. In case you are still using the project or would like to try it again I'll answer despite the long delay. Without the full context, it is hard to see what may be causing the problems with the extension output for you, but the generated assembly appears to be putting everything in the right place and cleaning the stack up.

From the assembly output you posted the EXTERN_C void __userpurge sub_4B8340(DWORD* a1@<edi>, int frame); is marked __userpurge but your hand-rolled assembly and extension output do not pop the arguments off the stack after the return. The opposite is true for the EXTERN_C void __usercall sub_4C2A60(int a1@<eax>, int a2, int a3, VERTC* a4, float a5); function. The prototype and assembly seem to have the stack cleaning conventions swapped, this may be where the confusion is coming from.

I have created a Godbolt sample with the function prototypes you provided and annotated the output below: https://godbolt.org/z/ohj1ET1s6

First the call to EXTERN_C void __userpurge sub_4B8340(DWORD* a1@<edi>, int frame);

mov     dword ptr [esp + 4], 1 ; store a1 on the stack so we can pass its address to the function
sub     esp, 12 ; align the stack pointer to 16 bytes (we will push a2 later so save 4 bytes for that)
lea     edi, [esp + 16] ; load the address of a1
push    2 ; push the stack variables right to left, just frame
call    sub_4B8340@PLT ; actually call the function, &a1 is in edi and frame is on the stack
add     esp, 16 ; clean up the stack because it is __userpurge 

Second the call to EXTERN_C void __usercall sub_4C2A60(int a1@<eax>, int a2, int a3, VERTC* a4, float a5);

mov     dword ptr [esp + 16], 1082130432 ; initialize the VERTC struct on the stack
mov     dword ptr [esp + 12], 1082130432
mov     dword ptr [esp + 8], 1082130432
lea     ecx, [esp + 8] ; load the address of the VERTC struct into ecx to be pushed later
mov     eax, 1 ; set the value of eax to 1 because the first parameter is passed in eax and the first argument is 1
push    1084227584 ; push the stack variables right to left starting with a5
push    ecx ; push the address of the VERTC struct
push    3 ; push a 3 for a3
push    2 ; finally push a 2 for a2
call    sub_4C2A60@PLT ; actually call the function
; do not clean up the stack because it is __usercall

This output seems to match your hand-rolled assembly, but with the stack cleaning conventions matching the prototypes. If I have misunderstood something or you have further questions please let me know, I will leave this issue open until we sort this out. Also, if you have a collection of these calling conventions in a project you are working on, I would love to see it and add them to the tests so they don't break in the future.