ares icon indicating copy to clipboard operation
ares copied to clipboard

ruby: Add source frame scaling via `screen` functions

Open jcm93 opened this issue 2 months ago • 2 comments

(Marked as draft pending some further testing and any feedback)

This PR exposes several methods in screen to the video backend in ruby, in order to resolve certain libretro shader scaling issues, and downscale source frame buffers as appropriate to improve shader performance.

Background

ares's current rendering strategy, broadly, is to render frames on the CPU one at a time before sending them to the video backend. The frames sent to the video backend can have many duplicated pixels present, as a shortcut to account for cases where frames contain scanlines with different pixel clocks. Unless we scale these buffers down to the appropriate size before sending them to librashader, we can introduce artifacts, cause incorrect shader behavior, or even crash if we cause a shader to create an overly large texture.

By allowing screen's setScale, setInterlace and setProgressive methods to call into the video backend, we can signal that ruby should scale down the framebuffers appropriately before sending them to shaders. This is (in theory of course) more efficient than performing this scaling on the CPU, and more streamlined logically in terms of making the video backend responsible for the final frame buffer creation.

Proof of concept

To test this approach, I added rudimentary implementations of these functions for the Metal backend. These methods basically do the following in a separate render pass, prior to shader passes:

  • In terms of width, simply always scale the framebuffer to the largest pixel width.
  • In terms of height, if we are in the progressive mode and the framebuffer is doubling identical lines, scale down.
  • If we're in the interlaced mode and lines are doubled, don't scale down, because a frame larger in height than a deinterlaced frame often acts as a signal for shaders that we are outputting interlaced video.

This approach is basic and may need refinement, but is a solid improvement on existing behavior in my testing.

Future work

(Disclaimer; not trying to speak for maintainers here; just thinking out loud):

Per conversations on the ares Discord, it seems as though ares may in the future move toward a renderer that leans more heavily on the video backend, perhaps similar to the approach taken by something like CLK, where the core generates a video signal, and the display backend is entirely responsible for synthesizing that signal into frames. This could have numerous advantages in terms of accuracy, performance and latency.

While not terribly significant, this PR could be considered a small step in this direction, as it moves some of the work generating the final source frame from the CPU to the GPU/display backend.

This does, however, also mean that for other video backends to achieve parity with Metal in terms of framebuffer scaling, they will need to implement setScale, setInterlace and setProgressive themselves in some fashion.

Examples

crt/crt-maximus-royale.slangp used here, not because it is my favorite shader but because it seems to detect interlacing changes on the fly, and also creates very large textures.

SNES 240p, first with PR then without PR:

Screenshot 2024-05-29 at 12 51 36 AM Screenshot 2024-05-29 at 12 54 21 AM

Sega 32x at combined 320/256 pixel width, first with PR then without PR:

Screenshot 2024-05-29 at 12 51 36 AM Screenshot 2024-05-29 at 12 54 21 AM

Sonic 2 switching from interlaced to progressive during runtime (works both ways) (crashes without PR):

Screenshot 2024-05-29 at 12 54 21 AM

jcm93 avatar May 29 '24 06:05 jcm93