Implemented bicubic downscaling

[ruanformigoni]

This PR implements a bicubic filter to remedy issue #692.

Update 10-mar-2023

Nier Automata (2160p)

Current gamescope result:

Proposal:

Need for Speed ProStreet (2160p)

Current gamescope result:

Proposal:

GTA IV (4320p)

Current gamescope result:

Proposal:

Dark Souls II (4320p)

Current gamescope result:

Proposal:

[vic-bay]

Downsampling from 2160p to 1080p looks okay, but everything lower looks weird. Here is comparison between 1440p and 1080p with FXAA. 1440p looks a bit aliased and pixelated. In my opinion forced FSR downsampling still looks better.

[EndlesslyFlowering]

you should probably look into Catmull-Rom (Bicubic with b=0 and c=0.5, rather than than b=0.33 and c=0.33) for downscaling rather than using normal bicubic. it's sharper and preserves edges the same as the default.

[rKsanu2MMYvypWePtQWM]

I can confirm that resolutions that aren't double the monitor resolution look weird.

[ruanformigoni]

Hey @EndlesslyFlowering , I've included the Catmull-Rom algorithm, thanks for bringing it to my attention. Thanks for the heads up @rKsanu2MMYvypWePtQWM, I've fixed this in the latest commits.

Here's the ranges of are shown below, the numbers represent the resolution multiplier from the monitor resolution and the render resolution, e.g., render at 2160p and display in 1080p == 2160/1080 == 2.

• Range of [1,2) uses FSR Bilinear
• Range of [2,3) uses Catmull-Rom with a mean of 4 samples
• Range of [3,4) uses Catmull-Rom with a mean of 9 samples
• Range of (4,inf) uses Catmull-Rom with a mean of 16 samples

You can toggle the downscaling effect with super + K

[DisplayTalk]

Definitely making really great progress. I think to cap it all off you should add the ability for user to adjust the bicubic parameters of B and C. Catmull-rom has ringing, which increases perceived sharpness but I want to be able to adjust it.

Examples of different bicubic setups with the 2 parameters:

B-Spline(blur only): B1 C0

Hermite(blocking only): B0 C0

Mitchell(balanced minor amounts of blocking, aliasing, and ringing): B0.33 C0.33

Catmull-Rom(balance of ringing and blocking): B0 C0.5

Explanation of different scalers: https://artoriuz.github.io/blog/mpv_upscaling.html https://en.wikipedia.org/wiki/Mitchell%E2%80%93Netravali_filters

Image showing the effects of different bicubic parameters on scaled quality: https://artoriuz.github.io/blog/images/mpv_upscaling/mitchell_survey.png

Also, if it's not too far out of scope, these scalers(especially the ringy ones like catmull-rom or lanzcos) work better when you convert gamma to sigmoid light before scaling, and back to the native gamma afterwards.

Pic showing that with upscaling: https://artoriuz.github.io/blog/images/mpv_upscaling/kanao/kanao_comparison.png

This applies also to the downscaling(your PR can be used as both an upscaler or downscaler, if it still works like the first iteration)

[ruanformigoni]

@Displaysguy thanks a ton for the explanation and resources! I'll study how to implement this properly and look into the possible method for argument passing you mentioned here.

[DisplayTalk]

Another tidbit, this one is specifically on sigmoidization:

"You may decrease halos and increase perceptual sharpness by increasing the sigmoidal contrast (up to 11.5, say). Higher contrasts are especially recommended with greyscale images (even “false RGB greyscale” that have three proportional color channels). The downside of sigmoidization is that it sometimes produces “color bleed” artefacts that look a bit like cheap flexographic (”gummidruck”) printing or chromatic aberration. In addition, sigmoidization’s “fattening” of extreme light and dark values may not work for your image content. If such artefacts are obvious, push the contrast value down from 7.5 (to 5, for example, or even lower). Setting the contrast to 0 is equivalent to enlarging through linear RGB. (Robidoux, 2012)"

From this(and the MPV scaling article) we know that sigmoidization can lower both light and dark overshoot(overshoot=ringing) with ringy scalers on both upscaling and downscaling. It's always superior to linearization because linearization only accounts for light overshoot and not dark overshoot.

However, it is not without downside. It has negative effects on the image and in certain situations(even where using it makes sense, which is only when using ringy scaling configurations) might not even be a net benefit. Because of this I wouldn't make adding sigmoidization a priority, and if you do add it make sure it is user-adustable and can be disabled(and is not enabled by default).

https://guide.encode.moe/encoding/resampling.html (ctrl-f sigmoid to find the quote)

[specfreq]

I've had a chance to test this out a bit. Here's some test screenshots.

[ruanformigoni]

Hi @specfreq , could you also include a screenshot with the downsampling algorithm disabled?

[specfreq]

@ruanformigoni Here's some new test screenshots. One thing I'm doing is changing the desktop resolution to match my target scaled resolution (1024x768 in this case) and forcing on vsync through the OS.

Launch options: gamescope -D -f -W 1024 -H 768 -w 4096 -h 3072 -- %command%

Let me know if I don't have it set up correctly.

[ruanformigoni]

Thanks for the followup @specfreq. I did the same as you (change the desktop resolution and downscale) and got way better results, here is a comparison. Did you set the in-game render resolution to 4096x3072 in the resolution settings?

[specfreq]

The resolution is set to 4096x3072

Here's my git pull:

[ruanformigoni]

Are you able to toggle it with super + k?

[specfreq]

I am not able to toggle it with super + k. I'm pretty new to using Git, should I just delete it and reinstall?

[ruanformigoni]

That should be fine, make sure that lines 176-179 of the file src/sdlwindow.cpp are these:

case KEY_K:
	g_wantedUpscaleFilter = (g_wantedUpscaleFilter == GamescopeUpscaleFilter::BICUBIC) ?
		GamescopeUpscaleFilter::LINEAR : GamescopeUpscaleFilter::BICUBIC;
	break;

In your launch command gamescope -D -f -W 1024 -H 768 -w 4096 -h 3072 -- %command%, do could you point it directly to where it is compiled? E.g.: ~/Repositories/gamescope/build/gamescope -D -f -W 1024 -H 768 -w 4096 -h 3072 -- %command%.

Edit: You can also try with other keys, maybe change KEY_K to KEY_P , recompile, and try with super + p. By your screenshots the bicubic filter is not enabled.

[specfreq]

I just rebuilt it and it is working now.

[specfreq]

Here's a test video: https://youtu.be/mbMsGK45p-U

And high quality: https://youtu.be/6RVYbmluInk

I'd love to be able to use this with the integer scaling.

[rKsanu2MMYvypWePtQWM]

any news? would love to see this finally merged...

[Filoppi]

I highly suggest to take a look at this more advanced downsampling method: https://github.com/dolphin-emu/dolphin/pull/11999/commits/ca93a5191fef40947a6fac935ce814fca8a2a295 It's heavier, but it looks damn fantastic when scaling from a relatively higher resolution. On older games, this would be perfect and very temporally stable.

[misyltoad]

I don't understand why you are still running the whole RCAS pass, surely we don't want sharpening after doing downscaling?

What's the rationale behind doing FSR for bilinear at certain ranges? Does it look better? I have my doubts there.

I think it also makes sense to have a g_wantedDownscaleFilter and g_downscaleFilter. Shouldn't affect upscaling at all. I would be happy to default downscaling method.

Then do like:

				bool needsUpScaling = frameInfo.layers[0].scale.x < 0.999f && frameInfo.layers[0].scale.y < 0.999f;
                bool needsDownScaling = frameInfo.layers[0].scale.x > 1.001f && frameInfo.layers[0].scale.y > 1.001f;
                frameInfo.useBICUBICLayer0 = g_downscaleFilter == GamescopeDownscaleFilter::BICUBIC && needsDownScaling;
				frameInfo.useFSRLayer0 = g_upscaleFilter == GamescopeUpscaleFilter::FSR && needsUpScaling;
				frameInfo.useNISLayer0 = g_upscaleFilter == GamescopeUpscaleFilter::NIS && needsUpScaling;

[ruanformigoni]

Hi @Joshua-Ashton , it was because I had issues running without it. I seem to have done so now, it uses blit instead, is that how it is supposed to work? I tried to only use a single cmdBuffer dispatch, but that didn't work well.

FSR (or its bilinear filter) looks better when the resolution is lower than 2X. I can try to experiment with bilinear in this range or leave as bicubic (since this PR is for a bicubic filter, using anything other would be misleading), to confirm this I've implemented another more computationally expensive bicubic method:

vec4 textureBicubic( sampler2D textureSampler, vec2 texCoord )
{
	AF1 fWidth = AF1_AU1(c0.z);
	AF1 fHeight = AF1_AU1(c0.w);

    // Size of one texel 
	AF1 texelSizeX = ARcpF1(fWidth);
	AF1 texelSizeY = ARcpF1(fHeight);

    // Result
	vec4 nSum = vec4( 0.0, 0.0, 0.0, 0.0 );
	vec4 nDenom = vec4( 0.0, 0.0, 0.0, 0.0 );

    // Get the decimal part
	AF1 a = fract( texCoord.x * fWidth );
	AF1 b = fract( texCoord.y * fHeight );

    // B & C parameters
	float B = AF1_AU1(c2.x)/AF1_(100);
	float C = AF1_AU1(c2.y)/AF1_(100);

    // Bicubic calculation
	for( int m = -1; m <= 2; m++ )
	{
        for( int n = -1; n <= 2; n++)
        {
            vec4 vecData = texture(textureSampler, texCoord + vec2(texelSizeX * float( m ), texelSizeY * float( n )));
            float f1 = catMullRom( B, C, float( m ) - a );
            float f2 = catMullRom( B, C, -( float( n ) - b ) );
            vec4 vecCooef1 = vec4( f1,f1,f1,f1 );
            vec4 vecCoeef2 = vec4( f2, f2, f2, f2 );
            nSum = nSum + ( vecData * vecCoeef2 * vecCooef1  );
            nDenom = nDenom + (( vecCoeef2 * vecCooef1 ));
        }
	}
	return nSum / nDenom;
}

The results were almost identical to the current bicubic implementation, which is way faster. What do you think of an option such as --force-needs-scaling (not on this PR), this way one could use fsr or other filter independently for downscaling or upscaling.

Also, what do you thing about the passing of the B and C parameters? Is this -B 0.0,0.5 and --bicubic=0.0,0.5 ok or would you like to have only fixed values?

I have implemented the requested g_downscaleFilter and g_wantedDownscaleFilter

[misyltoad]

I am so sorry, I didn't see this until now as it was posted when I was on holiday.

I think it's best to just keep downscale filter separate

[ruanformigoni]

Hi @Joshua-Ashton , no problem :smile: . I'll try to improve the filter when possible, there is still this issue pointed out by @DisplayTalk .

[gapys-krzysztof]

Hey @ruanformigoni , any updates about this? :slightly_smiling_face: Seems it conflicts with the master branch

Also, is there going to be a way to use the bilinear filter for downscaling? For some people the current implementation of bicubic seeems a bit blurry in some cases, and those configurable B and C params (no clue what they do) seem to not affect the actual game on my end no matter what values I force them to, including small fractional and high integer. (6880x2880 down to 3440x1440)

One could technically hack FSR to also downscale (doesn't allow it by default), and it kinda works, but it wasn't designed for it and it's probably not a good idea.

[dustContributor]

I'd love to have something like this on gamescope. It's really nice for older games because instead of having the GPU sit at 30% usage doing mostly nothing, you can push it a bit and get very nice image quality in return.

[LethalManBoob]

Progress update?

[Comptr-user]

Is there a way to get this to work? I tried following the steps for building, but it refused to build. Using FSR for downsampling did build and run, however it was unusable.