Scaling quality difference between Windows ram vs vram encoding path

[psyke83]

Is there an existing issue for this?

• [X] I have searched the existing issues

Is your issue described in the documentation?

• [X] I have read the documentation

Is your issue present in the nightly release?

• [X] This issue is present in the nightly release

Describe the Bug

Host configuration: Windows 10, RX 6600 running at 1440p. Client configuration: Ubuntu 22.10, Moonlight-QT 4.3.1 (flatpak), 1366x768 @ 90MBps.

amdvce encoder on latest nightly:

amdvce encoder patched[1] to use ram software rendering path:

[1] Patch:

diff --git a/src/video.cpp b/src/video.cpp
index f6e0904..80a13c9 100644
--- a/src/video.cpp
+++ b/src/video.cpp
@@ -532,8 +532,8 @@ static encoder_t quicksync {

 static encoder_t amdvce {
   "amdvce"sv,
-  AV_HWDEVICE_TYPE_D3D11VA, AV_HWDEVICE_TYPE_NONE,
-  AV_PIX_FMT_D3D11,
+  AV_HWDEVICE_TYPE_NONE, AV_HWDEVICE_TYPE_NONE,
+  AV_PIX_FMT_NONE,
   AV_PIX_FMT_NV12, AV_PIX_FMT_P010,
   {
     // Common options
@@ -571,7 +571,7 @@ static encoder_t amdvce {
     "h264_amf"s,
   },
   PARALLEL_ENCODING,
-  dxgi_make_hwdevice_ctx
+  nullptr
 };
 #endif

Expected Behavior

Images should appear mostly identical, but the ram encoding path has better filtering of scaled image and therefore much better legibility of the fonts of the screenshot in particular.

Additional Context

The libx264 encoder also has subjectively "good" scaling, but I provided a screenshot with the the amdvce encoder patched to use the software rendering path to eliminate the variable of AMD's encoding being poorer than libx264 to explain the issue (it seems not).

There is also an image corruption issue for a few dozen pixels along the entire right-hand side of the screen that occurs when Moonlight is set to capture at 1366x768, but does not occur at more standard resolutions such as 1080p. That is probably not related to the scaling quality issue.

Host Operating System

Windows

Operating System Version

10 22H2 19045.2364

Architecture

64 bit

Sunshine commit or version

7770b5f708675854674e8372af9227ee91ce5882

Package

Windows - installer

GPU Type

AMD

GPU Model

RX 6600

GPU Driver/Mesa Version

22.11.2

Capture Method (Linux Only)

No response

Relevant log output

No response

[psyke83]

I'm not certain if this is merely a scaling issue or if it's also a bug with colour conversion. See the Cyberpunk 2077 loading screen:

amdvce via nightly:

amdvce patched to use software path:

This should be a red loading bar, but the default vram conversion adds blue/cyan fringing.

[0xEXPUNGED]

I'm having the same problem on 1366x768 but 1080p is fine. Host: Windows 11, GTX 1060 3GB @ 1440p Client: Arch Linux, moonlight-qt 4.3.1 @ 1366x768

[LizardByte-bot]

This issue is stale because it has been open for 90 days with no activity. Comment or remove the stale label, otherwise this will be closed in 10 days.

[MaxLebled]

This is a problem on the current non-nightly version (v0.23.1) & Nvidia GPUs as well.

Software encoding:

Hardware encoding:

Host config is Windows 10, 4090, 4K — Client is Steam Deck (1280x800)

The very same problem exists in the official GeForce Experience / Nvidia apps when recording at non-native resolutions...

[ns6089]

Sunshine doesn't really have an interpolating downscaler currently, only... pixel count reduction thing that doesn't look completely horrible.

Adding a proper two-pass downscaler would require two more draw calls per frame (only one more if we get creative and incorporate rgb to yuv conversion into one of the passes). And these draw calls can be preempted by gpu scheduler killing the latency. Unless we move the whole pipeline to compute shaders (which is completely different can of worms). So yeah, possible but very niche and with significant drawbacks. Maybe someone will be interested in picking it up though, who knows.