Xbox Game Bar doesn't show up in the stream

[mxrsoon]

Is there an existing issue for this?

• [X] I have searched the existing issues

Is your issue described in the documentation?

• [X] I have read the documentation

Is your issue present in the nightly release?

• [X] This issue is present in the nightly release

Describe the Bug

The Xbox Game Bar isn't captured and sent to the client.

Steps to reproduce:

• Start streaming from a Windows host to any client
• Open the Xbox Game Bar (Win+G by default)

The Game Bar appears in the host, but not in the client.

Expected Behavior

Same as GameStream: the Game Bar shows up in both, the host and the client.

Additional Context

No response

Host Operating System

Windows

Operating System Version

11 (Build 22621.1105)

Architecture

64 bit

Sunshine commit or version

0.17.0

Package

Windows - installer

GPU Type

Nvidia

GPU Model

GeForce GTX 1650

GPU Driver/Mesa Version

528.24

Capture Method (Linux Only)

No response

Relevant log output

No response

[hugouchoasborges]

Currently, this is the only issue I have with sunshine. Using the OneGameLauncher via gamebar was very useful with Nvidia Gamestream

[TheElixZammuto]

According to Microsoft, the Xbox Game Bar is not drawn like a normal application but it uses some hacks to draw a layer on top the one we get from the operating system. So using the Desktop Duplication API currently does not allow to get to draw the Xbox Game Bar

[mxrsoon]

I just took a look at the WindowsGraphicsCapture API, and it can capture the Game Bar. I recorded a video of Microsoft's sample project that shows it capturing the Game Bar:

https://user-images.githubusercontent.com/49368302/215273595-0ec59a31-0198-4f32-9cbe-973cf8de7891.mp4

I don't know how much effort is needed to start using that API in Sunshine, as I'm away from Windows app development for a long time.

[jdjingdian]

Tesla p4-4q with vDWS license, also concur this problem.

[twilightgod]

I think GFE + Moonlight can show Xbox Game Bar. The only feature I want to Xbox Game Bar is the screenshot/recording. Is there any altenatives for Xbox controller-only use case? No keyboard

[Bobbar]

There's a C++ example here which uses the Windows.Graphics.Capture APIs.

https://github.com/robmikh/Win32CaptureSample https://github.com/robmikh/Win32CaptureSample/blob/master/Win32CaptureSample/SimpleCapture.cpp

The interface appears quite similar to DXGI DuplicateOutput. I can't say how performant it might be, but it appears to use DXGI underneath.

[LizardByte-bot]

This issue is stale because it has been open for 90 days with no activity. Comment or remove the stale label, otherwise this will be closed in 10 days.

[mxrsoon]

It would be nice to migrate to the mentioned API, as the Game Bar is a very useful tool for gaming, and also because we don't know if there are any other UI elements that are not captured by the Desktop Duplication API.

[crabnebula007]

It took me a while to understand that the Game Bar doesn't appear in the streamed video, even if it does activate and captures input from the controller.

I agree this would be a very useful feature.

[ghostmalls]

Having the same issue. Would love for this to be resolved if at all possible.

[soulvoid86]

Here to say same issue. I’d love to have this feature to screen record remotely. Using a Shield TV Pro to Stream from my office. Love the app!

[Zheta]

Also having this issue!

[mrallroy]

Would like to see game bar working in Sunshine as well. There's an Auto-HDR adjustment slider that apparently, can only be opened in Game Bar. Would really like access to that slider.

[shustovcreates]

Would love to see it too.

[Lionellanes]

Just echoing this. Cant accept friend game invites when streaming games. Love to see this addressed

[GMan-S]

I actually brought this up way back when sunshine first showed up on the scene and nvidia announced they were killing off the feature. I'm glad someone figured out that WindowsGraphicsCapture API does capture the game bar which is not captured using the Desktop Duplication API.

Nvidia gamestream and the same pigy back using moonlight via nvidia game stream does show the game bar when streaming and xbox gamepad functionality and windows shortcuts work just fine.

Moonlight streaming by way fo sunshine does not show the game bar and it probably stems from using the different API.

I would add my vote to figuring out how to convert so gamebar is captured in the stream to enable full replacement of the nvidia gamestream experience/functionality. Because the game bar is integrated into the Windows OS and always there it eliminated the need to install and use alternative apps to monitor performance, captrure screenshots, etc. It's the only missing QOL element to sunshine IMO because otherwise sunshine is superior since it works with non-nvidia GPUs.

[mrallroy]

Just checking in. Has there been any word as to where this issue might stand priority wise?

[Pirateguybrush]

I ran into issues tonight where I couldn't accept a friend's invite due to the game bar not being captured. Would love to see this fixed if possible please.

[IceYetiWins]

Also would like to see this. I use gamebar for screenshots and recording clips and can't really while streaming.

[chainria]

Also would like to see this. I use gamebar for screenshots and recording clips and can't really while streaming.

+1. Just found out that the Game Bar really IS the best solution for recording / capture and it's a pity we're kinda flying blind here. Using Sunshine / Moonlight for a dedicated gaming pc which should / could run headless but now it seems I still need a monitor to see the Game Bar. That really taints the experience a bit. Converting to the better API absolutely would be great! Maybe one could choose which one to use in the UI to counter regressions?

[tez011]

In 2024, C++/WinRT (which implements Windows.Graphics.Capture) is not supported out-of-the-box by MSYS2 MinGW, which is used to build sunshine. In an old issue from loki's repo the owner says that building in Visual Studio is not supported. That leaves clang, which does have an msys2 package for WinRT headers and libraries.

Remarkably few changes are needed to allow sunshine to compile with clang. However, the link steps fails due to unresolved symbols in libx265.a. This comes from the build-deps submodule and is a prebuilt library; it was presumably built with MinGW which would mean that compatibility with the clang-built Sunshine objects isn't guaranteed.

Owners,

• How palatable is a compiler change to msys2 clang64 on the Windows platform?
• Could we add a dependency on a system-installed ffmpeg development library (which exists in msys2, debian 12, and homebrew) instead of relying on a prebuilt artifact? (If not, nbd)

[ReenigneArcher]

We use a prebuilt ffmpeg for a few reasons.

1. Distro provided versions of ffmpeg vary greatly from one distro to another, and one version of a distro to another
2. We patch ffmpeg to improve performance for low latency capture
3. Many distros are disabling specific hardware encoders (and decoders) for legal reasons

With that said, there is an open PR to allow optionally using the system ffmpeg (something I need to find time to review)... https://github.com/LizardByte/Sunshine/pull/1970

Our windows ffmpeg build is compiled with mingw64. https://github.com/LizardByte/build-deps/blob/bf3821f07bb814fcac96d22984e35a015ead2a80/.github/workflows/build-ffmpeg.yml#L273

I don't think it's necessary to build with clang to get winrt. It's part of the Windows SDK. https://learn.microsoft.com/en-us/windows/uwp/cpp-and-winrt-apis/

[tez011]

Thanks for the quick response!!

It's not strictly necessary to built with clang to get winrt, however, https://packages.msys2.org/base/mingw-w64-cppwinrt** shows that the winrt headers are only supported on clang and ucrt platforms. The msys environment page suggests not mixing msvcrt and ucrt binaries, so I'll see what I can make work.

** compare with, e.g. https://packages.msys2.org/base/mingw-w64-ffmpeg, which lists packages for mingw-w64-x86_64-ffmpeg whereas for cppwinrt there's only a clang and ucrt64 package.

[ReenigneArcher]

Yea, I understand what you are saying... but it doesn't have to be installed through msys2/pacman.

[ReenigneArcher]

@tez011 perhaps you can request that they provide the mingw-w64-x86_64 version? https://github.com/msys2/MINGW-packages/issues/16088

As a temporary measure, see the instructions here about building any package they have.

[ReenigneArcher]

@tez011 I put up a PR. https://github.com/msys2/MINGW-packages/pull/20003

[tez011]

Per the suggestion of the msys maintainer in the pull request linked above, I set up a MSYS2 UCRT64 environment. In UCRT64, Sunshine builds without any code or repository changes. The compiler is equivalent between MINGW64 and UCRT64 so I hope this environment change is passable. @ReenigneArcher if you like, you can close the msys2 pull request as it won't be needed here.

I made a slight modification to the build to link against WinRT. The modified build outputs this to the log:

[2024:02:11:18:31:05]: Info: Trying encoder [nvenc]
[2024:02:11:18:31:05]: Info: Windows graphics capture api supported
[2024:02:11:18:31:06]: Info: ddprobe.exe [1] [] returned: 0x00000000
[2024:02:11:18:31:06]: Info: Set GPU preference: 1

I hope to open a pull request for this soon!

[ReenigneArcher]

That may be okay, but I need to do more testing... specifically to see if our CI still builds properly (without any new errors/warnings)... and to see if CLion handles it okay.

Feel free to open a PR though!

[ReenigneArcher]

Re-opening since the PR was reverted.

[eaferrari24]

Well this has been driving me absolutely crazy and I nearly reinstalled windows in an attempt to fix it. So thankful I found this just in time. I am certain GFE use to be able to interact with the game bar. Hopeful this can be resolved one day.