Unstable ABI in x86_64-unknown-linux for function with aggregate as fifth parameter

[hunterbridges]

We have this function in our C API:

	Vector3 RSDK_Vector3_CalculateBezierPoint_2(float t, Vector3 start, Vector3 ctrl1, Vector3 ctrl2, Vector3 end)
	{
		return Vector3::CalculateBezierPoint(t, start, ctrl1, ctrl2, end);
	}

Exposed in Beef as such:

[LinkName("RSDK_Vector3_CalculateBezierPoint_2"), CallingConvention(.Cdecl)] extern public static Vector3 CalculateBezierPoint(float t, Vector3 start, Vector3 ctrl1, Vector3 ctrl2, Vector3 end);

When the function is called from Beef in the x86_64-unknown-linux target triple, the end parameter is not making it across the language boundary. In the receiving frame on the C side, parameters t, start, ctrl1 and ctrl2 are intact, but end is garbage data.

According to x64 calling convention, the fifth parameter in this case should be pushed onto the stack as a pointer. By the looks of the Beef disassembly, it appears to be not properly pushed onto the stack.

---

Vector3 declaration in C++:

	struct Vector3
	{
		union
		{
			struct
			{
				float x;
				float y;
				float z;
			};

			struct
			{
				float r;
				float g;
				float b;
			};

			float data[3];
			Vector2 xy;
		};
	}

Vector3 declaration in Beef:

	[CRepr, Align(4)]
	public struct Vector3
	{
		public float x;
		public float y;
		public float z;
	}

The x86_64-unknown-linux assembly being generated around the invocation looks like this:

				position = Vector3.CalculateBezierPoint(tween,
00000000008DFCD6  mov rax,qword ptr [rsp+0A8h] 
00000000008DFCDE  mov qword ptr [rsp+8],rax 
00000000008DFCE3  movss xmm0,dword ptr [rsp+0A4h] 
00000000008DFCEC  movsd xmm1,qword ptr [rax+4C0h] 
00000000008DFCF4  movss xmm2,dword ptr [rax+4C8h] 
00000000008DFCFC  movsd xmm3,qword ptr [rax+4CCh] 
00000000008DFD04  movss xmm4,dword ptr [rax+4D4h] 
00000000008DFD0C  movsd xmm5,qword ptr [rax+4D8h] 
00000000008DFD14  movss xmm6,dword ptr [rax+4E0h] 
00000000008DFD1C  movsd xmm7,qword ptr [rax+4E4h] 
00000000008DFD24  movss xmm8,dword ptr [rax+4ECh] 
00000000008DFD2D  mov rax,rsp 
00000000008DFD30  movss dword ptr [rax],xmm8 
00000000008DFD35  call RSDK_Vector3_CalculateBezierPoint_2 (00000000007C0860h) 
00000000008DFD3A  mov rax,qword ptr [rsp+40h] 
00000000008DFD3F  movss dword ptr [rsp+0A0h],xmm1 
00000000008DFD48  movlpd qword ptr [rsp+98h],xmm0 
00000000008DFD51  movss xmm0,dword ptr [rsp+98h] 
00000000008DFD5A  movss xmm1,dword ptr [rsp+9Ch] 
00000000008DFD63  movss xmm2,dword ptr [rsp+0A0h] 
00000000008DFD6C  movss dword ptr [rax+28h],xmm2 
00000000008DFD71  movss dword ptr [rax+24h],xmm1 
00000000008DFD76  movss dword ptr [rax+20h],xmm0 
					boss.attackArcPos[0], boss.attackArcPos[1], boss.attackArcPos[2], boss.attackArcPos[3]);

Here's a look at an invocation of the same function in C++ llvm targeting x86_64-unknown-linux on Godbolt: https://godbolt.org/z/bYE5e6bbe

Let me know if you need any more information