[MojoShader] Preshaders with bool outputs aren't supported

[rds1983]

See the attached sample: BranchIfTest.zip

It has very simple shader that draws a shape in color, which is set in the following HLSL(see TestEffect.fx):

	float3 color = Color1;

	[branch]
	if (Choose > 0.5)
	{
		color = Color2;
	}

The Color1, Color2 are blue and green. The Choose is switched between 0.0f and 1.0 when Space is pressed. So basically the sample should draw the shape in blue or green color. However, for some reason, it draws in blue and pink:

https://github.com/FNA-XNA/FNA/assets/1057289/47fdb4f0-5d3c-4e13-89ee-131df1f45e3a

If the [branch] is removed or replaced with [flatten], then it works correctly. Similarly it works correctly in both MonoGame versions(DirectX and OpenGL):

https://github.com/FNA-XNA/FNA/assets/1057289/2794ac68-8fb0-4f4c-9822-f82db5b5fe70

[flibitijibibo]

Looks like it's preshaders:

technique Default
{
    pass
    {
        vertexshader =
            asm {
            //
            // Generated by Microsoft (R) HLSL Shader Compiler 9.29.952.3111
            //
            // Parameters:
            //
            //   float4x4 MatrixTransform;
            //
            //
            // Registers:
            //
            //   Name            Reg   Size
            //   --------------- ----- ----
            //   MatrixTransform c0       4
            //
            
                vs_3_0
                dcl_position v0
                dcl_texcoord v1
                dcl_texcoord o0.xy
                dcl_position o1
                dp4 o1.x, v0, c0
                dp4 o1.y, v0, c1
                dp4 o1.z, v0, c2
                dp4 o1.w, v0, c3
                mov o0.xy, v1
            
            // approximately 5 instruction slots used
            };

        pixelshader =
            asm {
            //
            // Generated by Microsoft (R) HLSL Shader Compiler 9.29.952.3111
            //
            // Parameters:
            //
            //   float Choose;
            //
            //
            // Registers:
            //
            //   Name         Reg   Size
            //   ------------ ----- ----
            //   Choose       c0       1
            //
            
                preshader
                neg r0.x, c0.x
                add r1.x, r0.x, (0.5)
                cmp ob0.x, r1.x, (0), (1)

            // approximately 3 instructions used
            //
            // Generated by Microsoft (R) HLSL Shader Compiler 9.29.952.3111
            //
            // Parameters:
            //
            //   float3 Color1;
            //   float3 Color2;
            //
            //
            // Registers:
            //
            //   Name         Reg   Size
            //   ------------ ----- ----
            //   Color1       c0       1
            //   Color2       c1       1
            //
            
                ps_3_0
                def c2, 1, 0, 0, 0
                if b0
                  mov r0.xyz, c1
                else
                  mov r0.xyz, c0
                endif
                mov oC0.xyz, r0
                mov oC0.w, c2.x
            
            // approximately 9 instruction slots used
            };
    }
}

Compiling with /Op will likely work around this.

[rds1983]

Yeah, compiling with that option solved the problem.

[flibitijibibo]

Looking at the asm, the issue is probably the output register - the VM currently assumes output goes to the float registers, it's probably parsed somewhere but I'm not sure where...

Preshader parser: https://github.com/icculus/mojoshader/blob/main/mojoshader.c#L3957

VM runner: https://github.com/icculus/mojoshader/blob/main/mojoshader_effects.c#L1889-L1914