Overlay doesn't work in Qt Vulkan applications

[tatokis]

Your system information

• Steam client version (build number or date): Mar 3 2020
• Distribution (e.g. Ubuntu): Ubuntu 19.10
• Opted into Steam client beta?: Yes
• Have you checked for system updates?: Yes

Please describe your issue in as much detail as possible:

The overlay does not seem to initialise at all when applications that use Vulkan on Qt are launched through Steam.

The "Access the steam community while playing" popup never shows up, and pressing Shift + Alt does not bring anything up.

However, when running the same applications through big picture with a steam controller connected, pressing the steam button on the controller causes a "fade to black" effect, as if a black rectangle with 0 opacity is being drawn on top of the game, with the opacity quickly going from 0 to max. Pressing the steam button again, fades the application back in.

The steam controller does not switch to the application configuration that is set before launching, which results in the controller being unusable.

This was initially observed in Dolphin git, as I wanted to use my steam controller with it. It used to work before when they had a wxWidgets UI, but they moved to Qt and I've been unable to use my controller with it since.

However, more easily, this issue can be reproduced by launching any of the Qt vulkan samples provided with Qt itself.

On Ubuntu, the package qtbase5-examples contains the "hello vulkan" samples, as well as prebuilt binaries to easily and quickly reproduce the issue. Adding /usr/lib/x86_64-linux-gnu/qt5/examples/vulkan/hellovulkantexture/hellovulkantexture to steam and launching it results in the same bug.

The overlay works as expected if OpenGL is used instead of Vulkan.

Steps for reproducing this issue:

1. Build (if needed) the hellovulkantexture Qt sample (or any of the other Vulkan ones)
2. Add it to steam and run it
3. Try to use the overlay

Using amdgpu + latest mesa git with radv from the oibaf PPA.

OpenGL vendor string: X.Org
OpenGL renderer string: AMD Radeon HD 7700 Series (BONAIRE, DRM 3.33.0, 5.3.8, LLVM 10.0.0)
OpenGL core profile version string: 4.6 (Core Profile) Mesa 20.1.0-devel (git-0103f02 2020-03-07 eoan-oibaf-ppa)

Qt version 5.12.4 is in the eoan repositories.

There are no obvious errors produced when the overlay does not work. This does not appear to be a regression, as I have been experiencing this issue for years with much older versions of these packages.

[kisak-valve]

Hello @tatokis, are you still experiencing this issue on an up to date system?

[tatokis]

@kisak-valve I'm not sure if this would be considered up to date in this case, but at the moment I am using Ubuntu 20.04. I can still reproduce it with the latest Steam beta at the time of writing (meaning I still can't use the steam controller), with:

OpenGL renderer string: AMD Radeon RX 580 Series (polaris10, LLVM 14.0.5, DRM 3.46, 5.18.11)
OpenGL core profile version string: 4.6 (Core Profile) Mesa 22.2.0-devel (git-e8fc5cc 2022-06-22 focal-oibaf-ppa)

and Qt version 5.12.8.

Seems that the oibaf ppa stopped making new builds for focal, so I can't update past e8fc5cc 2022-06-22 without building from source.

[Frogging101]

I was able to reproduce this issue as well.

OpenGL renderer string: AMD Radeon RX 6900 XT (navi21, LLVM 14.0.6, DRM 3.46, 5.18.11-arch1-1)
OpenGL core profile version string: 4.6 (Core Profile) Mesa 22.2.0-devel (git-deb36dc6c2)
Qt version 5.15.5

[Lahvuun]

It does actually work with QT_QPA_PLATFORM=xcb. Here's me running rpcs3 (a PlayStation 3 emulator using Qt): You might not be able to see the popup if using multiple displays, but it's there, off-screen. You'd be able to see it if you were to set the window to span across all the displays.

The problem is that the overlay doesn't react to events, so you can't use hotkeys and gamepads are always stuck in Desktop Configuration. This happens because it hooks Xlib functions to detect events (focus, unfocus, keyboard press, keyboard release, mouse move, etc.), but Qt uses xcb. Both are libraries for interfacing with the X server. wxWidgets programs will work fine because they use Xlib.

It's possible to kind of get it working through some black magic with LD_PRELOAD: steam.webm But properly fixing this is far from trivial (if not impossible), even for Valve with access to the overlay source code.

[Frogging101]

Replying to https://github.com/ValveSoftware/steam-for-linux/issues/6956#issuecomment-1193198437

What's the LD_PRELOAD magic?

[tatokis]

@Lahvuun

You might not be able to see the popup if using multiple displays, but it's there, off-screen. You'd be able to see it if you were to set the window to span across all the displays.

Unfortunately that is impossible with my monitor layout. The bottom right corner is inaccessible.

The problem is that the overlay doesn't react to events, so you can't use hotkeys and gamepads are always stuck in Desktop Configuration. This happens because it hooks Xlib functions to detect events (focus, unfocus, keyboard press, keyboard release, mouse move, etc.), but Qt uses xcb. Both are libraries for interfacing with the X server. wxWidgets programs will work fine because they use Xlib.

It appears you are correct. Under OpenGL, however, the overlay can correctly position itself. I just tested Dolphin and the popup did show up. That said, the steam controller still didn't switch to its profile, which I remember it doing in the past when I first opened the issue. I guess this is all happening because XOpenDisplay/XCreateWindow are never called by the application. It shouldn't need X to read input from the controller though.

But properly fixing this is far from trivial (if not impossible), even for Valve with access to the overlay source code.

I'm not sure why it'd be impossible to rewrite the main logic to use XCB. In addition, Xlib itself is and has been implemented using XCB for many years now, so I assume just hooking into XCB only should be sufficient for most applications. For glx the overlay can snoop the Display* from glXCreateContext and then call XGetXCBConnection to use for the rest of its non glx logic.

In addition, the overlay seems to already override xcb_create_window and xcb_create_window_checked, which I guess explains how it (at least partially) initialises under xcb.

$ nm ./ubuntu12_64/gameoverlayrenderer.so | grep xcb
00000000000350a0 T xcb_create_window
0000000000035300 T xcb_create_window_checked

[Lahvuun]

@Lahvuun

You might not be able to see the popup if using multiple displays, but it's there, off-screen. You'd be able to see it if you were to set the window to span across all the displays.

Unfortunately that is impossible with my monitor layout. The bottom right corner is inaccessible.

You can change the corner to verify: https://gaming.stackexchange.com/a/20133 The reason this happens is that overlay is created with the total resolution of all your displays. So, if using two 1080p monitors side by side, the overlay is rendered at 3840x1080 and the popup is anchored at the bottom right corner, outside of the window, which is usually only 1920x1080. Normally this is not an issue because created windows immediately receive a resize event and the overlay adjusts accordingly, but with xcb it can't.

The problem is that the overlay doesn't react to events, so you can't use hotkeys and gamepads are always stuck in Desktop Configuration. This happens because it hooks Xlib functions to detect events (focus, unfocus, keyboard press, keyboard release, mouse move, etc.), but Qt uses xcb. Both are libraries for interfacing with the X server. wxWidgets programs will work fine because they use Xlib.

It appears you are correct. Under OpenGL, however, the overlay can correctly position itself. I just tested Dolphin and the popup did show up. That said, the steam controller still didn't switch to its profile, which I remember it doing in the past when I first opened the issue. I guess this is all happening because XOpenDisplay/XCreateWindow are never called by the application. It shouldn't need X to read input from the controller though.

It doesn't need X to read input, but it needs X to figure out when to switch your controller from Desktop Configuration to Big Picture Configuration and back upon receiving window focus/unfocus events. With xcb your controller is just forever stuck in Desktop Configuration.

I guess Valve could, maybe, set the controller to Big Picture Configuration on new window creation? Not sure how many people's workflows this would break.

But properly fixing this is far from trivial (if not impossible), even for Valve with access to the overlay source code.

I'm not sure why it'd be impossible to rewrite the main logic to use XCB. In addition, Xlib itself is and has been implemented using XCB for many years now, so I assume just hooking into XCB only should be sufficient for most applications. For glx the overlay can snoop the Display* from glXCreateContext and then call XGetXCBConnection to use for the rest of its non glx logic.

In addition, the overlay seems to already override xcb_create_window and xcb_create_window_checked, which I guess explains how it (at least partially) initialises under xcb.

$ nm ./ubuntu12_64/gameoverlayrenderer.so | grep xcb
00000000000350a0 T xcb_create_window
0000000000035300 T xcb_create_window_checked

There were problems with hooking xcb, but I don't remember them — it's been years since I investigated this issue. One thing I do remember is that Xlib didn't always use xcb. Programs using older versions of Xlib would stop working. These are like 10+ years old now, but that's still a breaking change.

---

Honestly, doing the overlay this way is a joke. You'd also have to do similar things for getting the overlay to work under Wayland, and I'm not sure how well hooking libwayland works.

My assumption is that fixing this is low priority because Valve is planning to just have all games run in gamescope and simply embed the overlay in it. This would certainly fix a lot of problems — gamescope always, under either X11 or Wayland, knows when it's focused, when there are input events, etc.