Headset view mirror

[maluoi]

Describe the feature

A window on the desktop that will display/mirror what the user in XR headset sees. Maybe limited to VR for simplicity? Here's some features that would be good to have:

• Fast mirroring, just show one eye from the already-rendered view.
• Smoothed mirroring, render the scene from a smoothed camera viewpoint
• External control, the app can control the camera's movement.

This should work for Win32, UWP, and Linux.

An example use-case

From szefo on Discord

Yeah, im talking about vr media players in general, some call them cinema apps, they all have an mirror to desktop window option build in, but only if u enable it, its a privacy thing also

A 3rd person specator auto-camera from my own experimentation: https://twitter.com/koujaku/status/885609227338752000

[andrewpros]

Just my 2 cents. I would concentrate mostly on the basic mirror functionality as this is the most needed use case, when the user of an app just wants to see or show others what's inside the vr env in the desktop window view, some see it also as privacy feature, so it would be helpful to control it, possibly at runtime.

The fast mirroring eye method is more than enough for basic support.

The other things are like cherry on the top, ofc it would be great to have a concept of cameras or a spectator camera than can even be controlled to look at what u want independent of the headset view, but most of apps that do have the mirroring limits itself for that simple one eye mirror.

I see ability to have or even control an independent spectator camera as an advanced use case not many will need.

This connects maybe with the ability to show some desktop window only UI controls for debug purposes or app functionality, but probably most the former one. Either as an overlay on the vr mirror view or just a somewhat independent of the vr view control panel. But this is probably a separate big ui functionality.

Also not sure it is connected to how the current desktop window is created, but there is just no window, its just the control bar, and it is kinda annoying as all vr apps i have ever used just displays an empty black window if it doesn't have a mirror anyway. So besides the mirror functionality i would propose to already create a normal empty window not just the control bar.

[maluoi]

Thanks for the notes here! I think there's a couple details in there that should be insightful for when I get to implementation :)

And just FYI, fast mirroring is likely the hardest item on the list given StereoKit's current architecture. The other two items are easier stepping stones to get there, in addition to being valuable features of their own :) When I do mess around with this feature, I'll likely be implementing all of them simultaneously.

[Jumbli]

I assume this is not a simple feature, so in the short term, would it be possible to allow us to add some properties to the window that is currently displayed? That way we could configure messages to tell the user how to use Oculus or SteamVR mirroring tools and also inform them to keep the window in focus to capture keyboard input. Maybe display the app's logo too.

[maluoi]

Yeah, this isn't terribly simple, but I'll bump it up in my priorities! I don't think I'll fit any of this into the current version, but I'll try and focus on this soon after.

In the meantime, you can grab the handle to the window via something like this:

IntPtr hwnd = System.Diagnostics.Process.GetCurrentProcess().MainWindowHandle;

I believe this gets you the Win32 HWND that can then be used to modify and do plenty of stuff with the window. I saw someone recently doing this to output 2D text on the window via... GDI I think?

[Jumbli]

Excellent, thanks for the pointer :)