pcsx2
pcsx2 copied to clipboard
PCSX2
GS: Fix huge ST coordinates in input vertices.
Description of Changes
In the vertex trace, clamp floating point UV coordinates to valid UV ranges [-2047, 2047]. Add a pass to GSState::FlushPrims() that iterates through the primitivies about to be drawn and culls any that may be impossible to draw correctly (e.g., NaN values in ST coordinates) or replaces primitives with huge ST coordinates with a new primitive with clamped coordinates.
Rationale behind Changes
The input texture ST coordinates can sometimes be huge or NaN (for reasons not totally clear, it might be emulation or game bugs) so they cause issues with calculating the vertex trace values, determining texture vounding boxes, etc. This can cause graphical glitches downstream. Also, based on hardware tests it was determined that for truly large floating points values in ST (after divided by Q), they appeart to be clamped to +/-2047 when converting to UV (before the clamping/wrapping determined by the CLAMP register operates).
Suggested Testing Steps
Testing on GS dumps/games that are known to have huge ST coordinates as described above. I found a few examples of graphical issues that are caused in the SW renderer because of how the huge ST coords get clamped (first example below) . They were previously being masked because of the way the SW renderer it partially loads textures or cast floating point to integers in the rasterizer, which was being thrown off by the huge ST coordinates (so two problems were canceling each other out). Any help testing on more games/dumps or feedback on the changes would be greatly appreciated.
Currently these have been tested thouroughly on a large pack of GS dumps (provided by LightningTerror), with ta few of the more visible differences shown below. Some basic profiling was done on some of the GS dumps and the performance impact is similar to the GSVertexTraceFMM::FindMinMax(), since this is called once ever draw. It will have a larger impact on draws that have a large number of huge/infinite ST coordinates, which hopefully are not much.
In the example, before is left and after is right.
Should be a slight blurring effect. Unfortunately, fixing this issue causes another issue with the black patches. This should be "accurate" on the GS side but the input coordinates from the EE may not be in this dump. (Counter_Terrorist_Special_Forces_-_Fire_for_Effect_SLES-53046_20230830195729.gs.xz).
Should not have double eyebrows (Bakugan_Battle_Brawlers_shadows.gs.xz).
Should have a faint star-like patterns (Psychonauts_SLUS-21120_20231013005852.gs.xz).
Should have character's shadow (Tomb_Raider_-_Legend_SLUS-21203_20230905130255.gs.xz).
Should not have dark road (testdriveswminmax.gs.xz).
TODO: On the dumps that I saw a difference should be checked on a ps2 to compare how they should look like.
TODO: On the dumps that I saw a difference should be checked on a ps2 to compare how they should look like.
Lightning, would you be able to provide the dump name (if it is in gspack-10-july-2024)? I will try to work on this further given the new info from the hardware tests Ziemas ran.
007 Agent Under Fire - door blend 187 - Ride or Die_SLUS-21116_In_Game 2002 FIFA World Cup 50 Cent - Bulletproof_SLUS-21315_20230329231035 Ace Combat 04 - Shattered Skies_SLUS-20152_20230511225232 Ace_Combat_04_-_Shattered_Skies_SLUS-20152_20230502010138 AFL Premiership 2007_SCES-54639 Alfa Romeo Ape Escape 2_SLUS-20685_20220914200448 Dave_Mirra_Freestyle_BMX_2_SLES-50217_20230213220044
There's more but the list is huge, ace combat 4, ape escape 2 dave mirra seem broken.
I've fixed a few issues and the dump run seems to look good now on my end. Edit: Actually, there was a big error in the last force push that I didn't test. Should be fixed now hopefully.
What's the status on this, is it still worked on or ready for merge?
What's the status on this, is it still worked on or ready for merge?
@lightningterror Thanks for checking, I actually noticed a glitch with Counter_Terrorist_Special_Forces_-_Fire_for_Effect_SLES-53046_20230830195729.gs.xz. This is because we currently don't have handling of infinite STs in the rasterizer (the incorrect texture load was masking the issue at the expense of the missing haze effect; see image below). Unfortunately, even if we emulate this accurately, I think we would end up with the same or similar image. The issue may be further upstream with incorrect floating point calculations in VU or EE.
The area outlined has a flat patch from sampling the corner of the texture.
So the current status is I need to add infinite ST handling to the rasterizer and see if it fixes the above glitch.
Also, I thought you might want to run a dump run on your end to double check me since you may have additional dumps in your collection. If you have the time that would be super helpful, or if you think the testing I'm doing is sufficient, that's ok also.
@lightningterror This may be ready for review and/or merging. I updated the original post to some of the drawbacks of the fix, since it causes a graphical bug in at least a couple GS dumps in SW, but I believe these are problems with the input that are (hopefully) being emulated accurately on the GS. I also did a bit of profiling and found that the performance impact was not bad.
Thanks for the reviews lightningterror. The Oni dump had NaNs in the ST coordinates that were causing the triangle to be culled. I changed the handling to instead replace NaN ST coordinates with 0 (I think this was TellowKrinkle's suggestion at some point). I will run a full dump run to make sure nothing got broken and get back to you.
Thanks for catching this. That Galerian dump is strange; it has some NaN ST coordinates but I haven't found yet where the lines come from. I will need to investigate this further and maybe figure out how to do a hardware test.
@lightningterror I went on a large detour with this. I finally got my PS2 set up for HW tests and used the gs4ps2 code to make a dump runner that can be run from USB to test the Galerian dump. I'm unfortunately having some issues w/ the Vsync timing in the dump runner, but I may be able to run test it anyway. I have a feeling that its just a bad dump and the issues will show up on the PS2 also, but let's see.
TLDR: It needs some more time.
Apologies for the long delay in handling this. I tested the PR again with the dump and I'm not getting the red lines anymore. I tried it on 2.2 and got the red lines. Am I missing something?
Testing on a PS2 shows no red lines so I guess the dump itself is not corrupted.
I added a temporary workaround, might want to revert that when testing https://github.com/PCSX2/pcsx2/pull/12505
Thanks, I reverted it in this branch and tested more carefully (I was using HW instead of SW renderer by mistake earlier). Now I see that the lines in the Galerian dump show up always (this branch with temp fix reverted, current master, current master with temp fix reverted). Assuming that nothing has broken since a few months ago, I think this PR might be ready. The Galerian issue may require a better understanding of how GS handles NaNs.
Correction: The PS2 doesn't have NaNs. I meant that it might require more accurate floating point emulation.
I should note that I put in a small fix to gsrunner command line options. Let me know if that's ok or if I should make a separate PR for it.