GS/HW: Add ROV based blending/colclip/fbmask/afail for DX11/DX12

[stenzek]

Description of Changes

Not ready for widespread use, here for testing.

• [x] Make dx11/dx12 not break if the gpu doesn't support rov
• [x] Port across to vulkan renderer with fragment shader interlock
• [ ] Optimization and address all the TODOs I left in comments

Rationale behind Changes

ROV can be pretty fast in some games. Plus AFAIL handling isn't double pass.

Suggested Testing Steps

None yet.

Fixes Hot Shots FORE Golf/Everybody's Golf menus, Fixes #1499 Fixes #3876 Fixes #3350

[JordanTheToaster]

Fixes the shadows part of #6839

[refractionpcsx2]

Makes Hot Shots Golf worse, I think the condition/problem has actually been inverted, as you can see from the screenshot, the "blotches" which shouldn't be there, are all that's there lol

DX12:

Vulkan:

SW:

Dump: Hot Shots Golf FORE! ate depth.zip

[refractionpcsx2]

This problem from The Suffering was said to be an Alpha test/Depth problem but I can't remember the condition to what's happening, but for prosperity that doesn't work either

DX12:

SW:

Dump: SufferingATE.zip

[refractionpcsx2]

Slight update, I need to do "Maximum" blending to get Hot shots to work, and it is fixed like that

[GitHub-Khi]

Slight update, I need to do "Maximum" blending to get Hot shots to work, and it is fixed like that

Can you please check the PAL version of Hot Shots Golf Fore! as well?

Single Frame GS-Dump: Everybody's Golf_SCES-52582_20221222213239.zip

[refractionpcsx2]

[DonelBueno]

This is huge for games like SSX and Black, which need Full blending for proper emulation in HW.

On Vulkan, the fastest renderer for both games in master, SSX goes from 46 to 64 fps (6x native) and Black from 51 to 91 fps (6x native).

[lightningterror]

DX12 crashes with sharpening enabled (non rov gpu), also noticed this in the logs. [ 46.0960] C:\GitHub\master-pcsx2\pcsx2\bin\0(248,32): warning X4000: use of potentially uninitialized variable (FxaaPixelShader)

[bigol83]

These are several gsdumps from MGS3 intro first few minutes at default settings showing graphic glitches in Hardware rendering, even at native resolution.

Metal Gear Solid 3 - Snake Eater_SLES-82024_20230513010045.gs.zip Metal Gear Solid 3 - Snake Eater_SLES-82024_20230513010147.gs.zip Metal Gear Solid 3 - Snake Eater_SLES-82024_20230513010203.gs.zip Metal Gear Solid 3 - Snake Eater_SLES-82024_20230513010301.gs.zip Metal Gear Solid 3 - Snake Eater_SLES-82024_20230513010806.gs.zip Metal Gear Solid 3 - Snake Eater_SLES-82024_20230513010937.gs.zip Metal Gear Solid 3 - Snake Eater_SLES-82024_20230513010951.gs.zip

I just tested Vulkan, dx11 and dx12 and they all seem to have the same issues. I took the dumps with Vulkan renderer and switched to software to see the difference.

[Uzarkis]

@stenzek - Do you need more tests with the golf games?

[stenzek]

No, and please stop posting/bumping PRs.