[BUG]: GS: blend_mix inaccuracies causing color banding

[Zombeaver]

Describe the Bug

I'm putting issue in per @refractionpcsx2 There is an issue with color banding in some games when any blending mode other than Minimum or Maximum is used. Maximum looks better between the two (higher contrast) but in some games (like Grandia 3 and Shadow Man) have a massive performance impact. Instances that I encountered were:

Fatal Frame - Flashlights and darkness Fatal Frame 3 - Particularly visible on rounded surfaces (like the main character's butt) Grandia 3 - Same as Fatal Frame 3 Shadow Man: 2econd Coming - very visible on the blur and fire effects seen on the title screen, this is probably the most obvious case of them all Final Fantasy XII - Blur effect when Reks is waking up at the beginning of the game

Musashi - Samurai Legend has a similar issue as well, though it can be corrected with blending set to medium on GL or by using Auto Flush.

Reproduction Steps

Play any of the affected games with blending set to anything between Basic and Full.

Expected Behavior

For there not to be color banding, and for it to look similar to either Minimum or Maximum blending modes.

PCSX2 Revision

v1.7.3146

Operating System

Windows 10 (64bit)

If Linux - Specify Distro

No response

CPU

Ryzen 9 5900X

GPU

RTX 3080

GS Settings

No response

Emulation Settings

No response

GS Window Screenshots

Examples:

Shadow Man - Minimum:

Shadow Man - Basic:

Shadow Man - Maximum:

Fatal Frame 3 - Minimum:

Fatal Frame 3 - Basic:

Fatal Frame 3 - Maximum:

Logs & Dumps

Shadow Man - 2econd Coming_SLUS-20413_20220731164949.gs.zst.zip Fatal Frame 3 - The Tormented_SLUS-21244_20220731165200.gs.zst.zip Musashi Samurai Legend_SLUS-20983_20220731154957.zip

[refractionpcsx2]

Can you provide the Musashi dump here too, I think that was a clear example.

[Zombeaver]

Yep, just added it.

[lightningterror]

The games don't appear to use dithering effect, issue is blend mix isn't accurate and requires sw blend which is maximum in this case.

With that said I was working on something to further improve blend mix, maybe it will help here but need to figure some stuff out how to best handle it.

[refractionpcsx2]

Musashi 100% uses dithering, because I tested forcing dithering on and the colour banding went.

[lightningterror]

Musashi 100% uses dithering, because I tested forcing dithering on and the colour banding went.

I'm unable to trigger the dithering log from the dumps.

[Zombeaver]

The games don't appear to use dithering effect, issue is blend mix isn't accurate and requires sw blend which is maximum in this case.

So then why does it look correct (or more correct anyway) at minimum? The banding is only present in basic through full but not at either extreme of minimum or maximum. Minimum basically just looks like it's slightly lower contrast, but without the banding. Musashi is fixed at medium (or with Auto Flush), but it's an exception among the other test cases.

If there's anything you need me to send or test for any of these games just let me know what you need.

[Zombeaver]

Encountered another instance of this with a dust cloud in Legaia 2 (it's difficult to see, but there was a difference - banding was present with basic blending but not on minimum or medium+). It seems like in most cases where I'm seeing it it involves some kind of semi-transparent effect overlaid on top of something else.

[refractionpcsx2]

If it's fine on Medium, that's probably not going to change.

[Zombeaver]

Even though it's not present in minimum either?

[refractionpcsx2]

Uhh, maybe... :P I guess we can look at it.

Can you provide GS Dumps?

[Zombeaver]

I can, but you're gonna think I'm a loon when you look at it lol. It's the least obvious case of any of these by far. I'm going to keep playing and see if I can find a better example later.

https://drive.google.com/file/d/1R4Y1UVDhrckOS3xuGGGK0AlklW1vh2DF/view?usp=sharing

[Zombeaver]

Here's a pretty good one for Primal. Correct at anything other than Basic, including Minimum.

Primal - Civilization Is Only Skin Deep_SCUS-97142_20220805173843.gs.zst.zip

[lightningterror]

See if https://github.com/PCSX2/pcsx2/pull/6784 helps Fatal Frame, other than that you'll need to use Maximum blending as that is the accurate option, nothing else we can do atm.

[Zombeaver]

I can test #6784 but again this isn't addressing the elephant in the room that the issue isn't present with minimum. There's something missing here.

[lightningterror]

I can test #6784 but again this isn't addressing the elephant in the room that the issue isn't present with minimum. There's something missing here.

It's because of truncation/rounding differences between hw and sw blending, blend mix will never be accurate, and some games may be affected negative by it, real solution is to use higher blending modes.

[Zombeaver]

Got an interesting case that may or may not be the related to the others mentioned previously.

Contra: Shattered Soldier has some fairly ugly full screen dithering that, if disabled, causes color banding during certain screen effects (a blur in this case) at anything other than minimum - including maximum. No issue with any of them including basic if dithering is enabled, and no issue with disabling dithering so long as minimum is used.

Contra - Shattered Soldier_SLUS-20306_20220808145918.gs.zst.zip

[refractionpcsx2]

That's the whole point in dithering, that's not a bug.

[Zombeaver]

I understand that the purpose of dithering is to prevent banding, yes. What I'm not understanding here is why the issue isn't presenting itself at minimum blending. There's no color banding there, with dithering off.

[refractionpcsx2]

That's probably a blending thing, but after we had some discussion, using blend_mix (where we both do software and hardware blending) is always going to be inaccurate, there's not much we can do about that, so the solution is really to use higher blending.

But it would be nice (personally) to have the software blending be purely software and not blend mix in some of the slightly higher levels, maybe at "high" or something, since there will be a performance impact.

[Zombeaver]

I guess I'm just kindof confused by the fact that pretty consistently here, in all cases other than Musashi in fact, there's no issue at minimum - does minimum actually use pure software blending and everything in between it and maximum uses mix? If so, could pure software be used across the board (or maybe just at a slightly lower level than maximum like you said)? It doesn't seem to have any impact on performance at minimum if it's being used there. Maximum has a huge hit on performance in most cases, though I don't know that it's specifically because of software vs mix (I assume it's a combination of things).

Sorry if I'm asking something stupid, just trying to understand.

[TellowKrinkle]

There's three possible ways to do blending: Pure Software (Same as PS2): floor(SrcColor * Alpha + DstColor * (1 - Alpha)) Pure Hardware: round(SrcColor * Alpha + DstColor * (1 - Alpha)) Mix: round(floor(SrcColor * Alpha) + DstColor * (1 - Alpha))

This is because PC hardware blend units always perform the equation listed under "Pure Hardware", while shaders don't have easy access to DstColor. (Mix is done by calculating floor(SrcColor * Alpha) in the shader, then sending that to the hardware blend unit configured to use the equation round(SrcColor + DstColor * (1 - Alpha)).)

Of all possible inputs (SrcColor in 0-255, DstColor in 0-255, Alpha in 0-128), pure hardware is correct 52% of the time and too high 48% of the time, while mix is too low 12.5% of the time, correct 75% of the time, and too high 12.5% of the time.

This gives the mix the benefit of both being correct more often, and averaging to the correct value, but it does give banding when two gradients are blended together with a fixed alpha due to the way it rounds partial results. In the end, we've selected hardware for minimum, mix for basic and above, and software is applied at varying amounts because it's slow, but reaches 100% of the time in Ultra blending.

Comparison program if anyone wants to run it for themselves

(Run with Swift)

var hwError = [Int](repeating: 0, count: 9)
var mixError = [Int](repeating: 0, count: 9)

func ToRGBA8(_ float: Float) -> Int {
	return Int((float * 255).rounded(.toNearestOrEven))
}

for srcI in 0...255 {
	for dstI in 0...255 {
		for alphaI in 0...128 {
			let srcF = Float(srcI) / 255
			let dstF = Float(dstI) / 255
			let alphaF = Float(alphaI) / 128
			let ps2 = (((srcI - dstI) * alphaI) >> 7) + dstI;
			let hw = ToRGBA8(srcF * alphaF + dstF * (1 - alphaF))
			let mix = ToRGBA8(Float((srcI * alphaI) >> 7) / 255 + dstF * (1 - alphaF))
			hwError[hw - ps2 + 5] += 1
			mixError[mix - ps2 + 5] += 1
		}
	}
}

func printResults(_ results: [Int], name: String) {
	print("\(name):")
	for (i, result) in results.enumerated() where result > 0 {
		print("\t\(i - 5): \(result)")
	}
}

printResults(hwError, name: "HW")
printResults(mixError, name: "Mix")

[Zombeaver]

Thank you! That's exactly what I wanted to know, and actually makes sense based on my testing.

So that means that in all of my problem cases other than Musashi, pure hardware is doing a better job than mix and yielding a result similar to pure software, albeit with generally lower overall contrast, which I've been offsetting with minor shade boost adjustments.

I guess my next question is, is there any way to make it at all "adaptive" based on the situation? The reason I ask is that the behavior I'm seeing happens pretty consistently in very specific similar circumstances - in all cases it involves some semi-transparent overlaid effect or blur. Is there any way to do any intelligent swapping based on the scene demands? I don't know if that's feasible or not, just asking. Either that or some way to make mix better suited to deal with that specific situation?

[TellowKrinkle]

Actually, I just thought of a way to be more accurate in blend mix