GS/VK/GL/DX12/DX11: Depth feedback loops and accurate AFAIL.

[TJnotJT]

Status

Draft until

1. Dump runs are completed for all renderers: ~~VK~~, GL, DX12, DX11.
2. Ported to Metal.

Description of Changes

1. Add support for depth feedback loops either via texture barriers or copies.
2. Use the depth feedback for accurate AFAIL (enabled via INI setting, though maybe we could make it the default if has better/equal performance to the 2 pass system).

Follow-up for separate PRs

1. In certain cases we don't need full barriers for AFAIL. The cases are detected with flags ate_RGBA_then_Z and ate_RGBA_then_Z in GSRendererHW::DrawPrims() for the current AFAIL system.
2. ~~Add texture barriers to DX12 instead of copies.~~ Nvm, I just found out that DX12 doesn't support this for feedback loops.

Rationale behind Changes

1. Fixes some effects that are not accurately rendered by the current AFAIL emulation. Also reduces draw calls in some cases.
2. In the future, the depth feedback will also be useful for AA1 emulation (see discussion: https://github.com/PCSX2/pcsx2/pull/13617).

Suggested Testing Steps

Please put the following in the INI:

[EmuCore/GS]
...
HWAFAILFeedback = 1

Then use one of the renderers: VK, GL, DX12, DX11. Performance testing would be a big help.

Did you use AI to help find, test, or implement this issue or feature?

Yes, to reference graphics API functions/semantics.

Comparisons

There are many dumps with significant difference, but for the most part it is not obvious that they are fixes (i.e., random pixels in foliage). However, there are some clear cases below.

"everybody_golf_shadow.gs.xz" (see loading bar at bottom)

Master VK

PR VK

"Hot Shots Golf FORE!_SCUS-97401_20221204040422.gs.xz" (see character portraits)

Master VK

PR VK

"Minna no Tennis_SCPS-15113_20230922202011.gs.xz"

Master VK

PR VK

[JordanTheToaster]

Barriers on the OSD do not seem to display properly now only showing 0 on Vulkan. The setting itself seems to do nothing for me currently.

[TJnotJT]

Barriers on the OSD do not seem to display properly now only showing 0 on Vulkan. The setting itself seems to do nothing for me currently.

Oops, I put the wrong setting name - it should be HWAFAILFeedback=1. For the OSD issue, could you enable debug device and post the log, please?

[TJnotJT]

Fixed barrier counts in VK and deleted some old commented code.

[TJnotJT]

Added a new commit to refactor alpha test selection so that it is contained in a single function. Some minor changes to shaders, etc.

[TJnotJT]

Latest push rebases and adds a temporary commit that for depth feedback via copy in OpenGL (based on the DX11 method). Based on a a few tests we have reason to believe that texture barriers for a depth attachment bound as a shader resource doesn't always work, so this is for testing until we can determine exactly.

[TJnotJT]

Lastest commit rebases and adds a temporary commit to try a different depth feedback method for DX12 and OpenGL: convert Z to a temp color texture and attach it as a second RT (allows using recent barriers addition to DX12 (https://github.com/PCSX2/pcsx2/pull/13653) and barriers/FB fetch in OpenGL).

[wxvu]

God.of.War_SCUS-97399_20251220193624.gs.zst.zip

PCSX2-windows-Qt-x64-cmake-clang-pr[13631]-sha[34f3357]-title[GSVKGLDX12DX11 Depth feedback loops and accurate AFAIL] (Disable Shader Cache has no effect.)

HW Renderer (Max Blend)

SW Renderer

[TJnotJT]

God.of.War_SCUS-97399_20251220193624.gs.zst.zip

PCSX2-windows-Qt-x64-cmake-clang-pr[13631]-sha[34f3357]-title[GSVKGLDX12DX11 Depth feedback loops and accurate AFAIL] (Disable Shader Cache has no effect.)

Thanks for the comparison. I don't notice any visual difference in the two images, does this mean the issue is fixed on your end?

[wxvu]

I don't notice any visual difference in the two images, does this mean the issue is fixed on your end?

The severe ordering issue is resolved, yes. HW Renderer on the PR now appears to match Master. (There are visual differences remaining between SW and HW, but these are not related to the PR and exist on Master.)

[TJnotJT]

The severe ordering issue is resolved, yes. HW Renderer on the PR now appears to match Master. (There are visual differences remaining between SW and HW, but these are not related to the PR and exist on Master.)

Got it, thanks for clarifying.