Idea: Screen space effects for overlays

[BOLL7708]

Hello! Long time no issue-posting! This is just a loose idea though, not a real issue.

Ever since I made my own overlay application, I felt as if the overlays themselves stood out a lot, they did not look like they belonged in the rest of the VR environment. I was looking for some way to get the average color of the scene to tint my texture before submission, but did not find anything immediately available.

I have not stopped thinking about it though, and while the original inspiration for this post was the blur effect that exists for some Windows 10 settings windows, where it blurs what is behind it, funny enough Apple is now doing the same in their Vision OS. And that is basically what this suggestion/wish is about. The ability to use screen space effects in overlays, basically to access the pixels beneath it when rendered. It would be a convenient way to make overlays that do not look out of place, I've attached some shots from Vision OS.

I do understand that Vision OS is one unified 3D engine, and that the overlay system in OpenVR might not be in the pipe for any substantial updates, especially if OpenVR is moving towards OpenXR, although from what I've heard OpenXR is kind of lacking overlays at all so perhaps some innovation could be pushed on that front.

In any case, I just wanted to leave this here so it has any chance of happening at all 😅 Good job with SteamVR 2.x, I've already been enjoying playing games in the new game theater 🙂

[BOLL7708]

After some covert discussion on SadlyItsBradley's Discord server, a few additional points and thoughts came forward.

1. This could in theory be done by third party applications by acquiring the frame-buffer, but that is likely way more taxing than doing it natively in the existing composer.
2. This could be applied either as a chroma-key effect by including a specific color in an image and possibly a flag when displaying the overlay, or as a separate image that is used in addition to the RGBA data.
3. These shaders could be custom, or a range of preset shaders, like different levels of blur.
4. It could even be just one single feature/effect, where the whole image that is rendered also exists as a very blurry version, which can be masked in for overlays that provide the appropriate flag/data.

That is it for now at least 😁 if I figure out how to get the frame-buffer I might try my hand at this myself, but that is pretty far off right now, mostly as it feels like it could be a big overhead 😅 I know the LIV stabilized view does this so I might run that and see how much resources that pulls.

[BOLL7708]

I was going to work on this today, to see if it's viable in my overlay application, but then I realized it will only work properly for overlays that are attached to the headset view, unless it is done in the final compositing step, or if I take the frame buffer and distort it myself to project the blurry image onto the overlay regardless of which position it has in the headset view.

To me that feels a bit overkill and sub-optimal, this as it has to be done separately for each individual overlay, and overlays would be invisible for each other. So my conclusion is basically #​​4 in the previous comment, that having it done as a global effect would be the way to go about it.

The practical steps I imagine, which comes from zero knowledge of how the compositor actually puts all of this together, would be:

1. Acquire rendered frame
2. Draw any parts of overlays that are beneath another overlay, this so they will show up in the blur.
3. Store a blurred version of the image so far, possibly at a lower resolution, whatever is viable as optimization.
4. Draw overlays on top of that, with the blurry texture substituting a color in the overlays, like the typical #00FF00.

Possibly make it an opt in to start with, and if no currently displayed overlay uses the feature the extra blurry texture could be skipped.

[BOLL7708]

Coming back to this, I've been playing games in the Game Theater that is built into SteamVR, to see what it could be like using the Steam Frame headset. I realize the ground there does seem to be an overlay element that grabs the game capture and likely applies a shader to it.

That feels at least tangentially related to having this overlay effect, just that the texture referenced would be the image made for the headset and not coming from a desktop game. The down-scaling/blurring would be a fairly similar implementation at least. I hope this is at least considered at some point, again, just to make overlays look more natural when they show up, like they belong.

Would be pretty neat for dashboard overlays too!