Wayland/Vulkan: WaylandBackend.cpp:1568 Assertion `u8bitFormat != DRM_FORMAT_INVALID`

[tskaar]

Is there an existing issue for this?

• [X] I have searched the existing issues

Are you using any gamescope patches or a forked version of gamescope?

• [X] The issue occurs on upstream gamescope without any modifications

Current Behavior

Launching gamescope on a Wayland compositor that's running with a Vulkan renderer crashes Gamescope and the following assert triggers:

gamescope: ../src/Backends/WaylandBackend.cpp:1568: virtual void gamescope::CWaylandBackend::GetPreferredOutputFormat(uint32_t*, uint32_t*) const: Assertion `u8BitFormat != DRM_FORMAT_INVALID' failed.

Doing a git bisect shows that 01101093445af20ea25ad386fd872d5f92266c2c is the bad commit.

Workarounds:

1. Using OpenGL as backend for the Wayland compositor
2. Using the SDL backend, requires #1548

Compositor(s):

• Jay with Vulkan backend
• River with Vulkan backend
• River with OpenGL backend (works)

Steps To Reproduce

1. Launch River/Sway with WLR_RENDERER=vulkan
2. Open terminal
3. Launch gamescope

Hardware information

- Distro: Arch Linux
- CPU: AMD 7735HS
- GPU0: Radeon 680M
- GPU1: Radeon RX 7600S
- Driver Version: mesa-git (24.3.0_devel.197058.e69aba2cded-1)

Software information

- Desktop environment: Jay + River
- Session type: Wayland
- Gamescope version: git upstream (3.15.9.r21.g7dd1bcd9-1)
- Gamescope launch command(s): n/a

Which gamescope backends have the issue you are reporting?

• [X] Wayland (default for nested gamescope)
• [ ] DRM (default for embedded gamescope, i.e. gamescope-session)
• [ ] SDL
• [ ] OpenVR

Logging, screenshots, or anything else

Bisect log:

git bisect start
# status: waiting for both good and bad commits
# bad: [7dd1bcd9102a17e039970ccd9a324a9fe8365d6d] rendervulkan: Fix a CVulkanCmdBuffer leak that could result in screenshot request failures
git bisect bad 7dd1bcd9102a17e039970ccd9a324a9fe8365d6d
# status: waiting for good commit(s), bad commit known
# good: [b2505fe4dac917aadd1ab05473d863830a7b5859] Fix a CVulkanCmdBuffer leak that could result in screenshot request failures
git bisect good b2505fe4dac917aadd1ab05473d863830a7b5859
# good: [8045786067f7a7fd9fd3fff98c29588f314218a9] layer: Add hacks for Talos Principle and Serious Sam's bad swapchain usage
git bisect good 8045786067f7a7fd9fd3fff98c29588f314218a9
# bad: [ca7bf9ff4a1987f358a27c173d46fad7a4645658] readme: Fix Meson deprecated command warning
git bisect bad ca7bf9ff4a1987f358a27c173d46fad7a4645658
# bad: [d87f6d9b9d98ea8064e4063c01d483895fef87f0] Add clipboard support from SDL backend
git bisect bad d87f6d9b9d98ea8064e4063c01d483895fef87f0
# good: [7a273743368aaa6368cb0d31d4786d78b4b00dc6] rendervulkan: Fix scanout-able check
git bisect good 7a273743368aaa6368cb0d31d4786d78b4b00dc6
# bad: [a2b45e15009f80ca1ad346840f2574bbe060fa94] Fix how reshade effects are cleared
git bisect bad a2b45e15009f80ca1ad346840f2574bbe060fa94
# bad: [01101093445af20ea25ad386fd872d5f92266c2c] rendervulkan: Attempt at fixing modifierless devices having issues
git bisect bad 01101093445af20ea25ad386fd872d5f92266c2c
# first bad commit: [01101093445af20ea25ad386fd872d5f92266c2c] rendervulkan: Attempt at fixing modifierless devices having issues

[codebam]

Same issue on sway 1.10 and 7900XTX

[labre]

Also reproduced this on Gamescope 3.16.1, Sway 1.10 with vulkan wlr renderer on Gentoo. Would a revert of 01101093445af20ea25ad386fd872d5f92266c2c be accepted?

[mstoeckl]

The workaround I am currently using for this:

diff --git a/src/backend.h b/src/backend.h
index 82db19c..b557ec6 100644
--- a/src/backend.h
+++ b/src/backend.h
@@ -298,7 +298,7 @@ namespace gamescope
         virtual std::span<const uint64_t> GetSupportedModifiers( uint32_t uDrmFormat ) const = 0;
         inline bool SupportsFormat( uint32_t uDrmFormat ) const
         {
-            return Algorithm::Contains( this->GetSupportedModifiers( uDrmFormat ), DRM_FORMAT_MOD_INVALID );
+            return !this->GetSupportedModifiers( uDrmFormat ).empty();
         }
 
         virtual IBackendConnector *GetCurrentConnector() = 0;

I don't understand the code well enough to recommend this workaround for general use. I suspect the "right" thing to do here would be to modify IBackend::SupportsFormat to intersect the set of modifiers supported by the compositor with the set supported by the renderer Gamescope is using; and return true if the intersection is nonempty, but haven't yet looked around the code enough to figure out how to do this or whether there is a better approach.)

[migrgh]

I have the same issue here with Archlinux and 760M Graphic Card running sway 1.10.1 vulkan-radeon (mesa).

Replying to https://github.com/ValveSoftware/gamescope/issues/1604#issuecomment-2603198783

I apply the patch in PKGBUILD and the Game Disco Elysium starts now with. Thank You !

gamescope --mangoapp --adaptive-sync -r 60 -w 1920 -h 1080 --sdr-gamut-wideness 1 -- %command%

but if i try to use up scaling like

gamescope --mangoapp --adaptive-sync -W 3840 -H 2160 -r 50 -w 1920 -h 1080 -W 3840 -H 2160 -F nis --sdr-gamut-wideness 1 -- %command% the window manager (sway) crash.

I don't now if this is related to this fix.

[72f393f4a1b9dfecdd32641633e9e896]

Valve, please fix this so we don't have to build it.

[ninetailedtori]

Is this one still reliant on https://github.com/ValveSoftware/gamescope/pull/1826? Is it possible to convert this to a patch then just toss it into some of the package manager versions that are community-managed for now? Coz I'm still getting this error, so I assume so.

[tskaar]

Is this one still reliant on https://github.com/ValveSoftware/gamescope/pull/1826?

Yes

Patching it is relatively easy:

1. curl -L https://github.com/ValveSoftware/gamescope/pull/1826.patch | git am or
2. git cherry-pick 5266372f377c58d7c5511a962032095af3f56c1e

[ninetailedtori]

Perfect, CR, works flawlessly with the patch ty! Wonder if we can get an engie in on the PR just to merge it soon :)

[fuzunspm]

same issue on Vulkan WLR_RENDERER

[kode54]

Here's an alternative test patch:

diff --git a/src/backend.h b/src/backend.h
index 862257f..9f3f2db 100644
--- a/src/backend.h
+++ b/src/backend.h
@@ -311,7 +311,9 @@ namespace gamescope
         virtual std::span<const uint64_t> GetSupportedModifiers( uint32_t uDrmFormat ) const = 0;
         inline bool SupportsFormat( uint32_t uDrmFormat ) const
         {
-            return Algorithm::Contains( this->GetSupportedModifiers( uDrmFormat ), DRM_FORMAT_MOD_INVALID );
+            auto formats = this->GetSupportedModifiers( uDrmFormat );
+            return Algorithm::Contains( formats, DRM_FORMAT_MOD_INVALID ) ||
+                Algorithm::Contains( formats, DRM_FORMAT_MOD_LINEAR );
         }
 
         virtual IBackendConnector *GetCurrentConnector() = 0;

Yes, at least under wlroots 0.20 with the Vulkan renderer, it exposes DRM_FORMAT_MOD_LINEAR for most/all formats, but not DRM_FORMAT_MOD_INVALID.

[kode54]

It runs deeper than this. It attempts to use implicit modifiers, as it will fail trying to use LINEAR due to the texture size limit of 2560x2560 on RADV.