Severe degradation using obs-vkcapture together with gamescope on Steam games

[Plarpoon]

Describe the bug Hello, my name is Alex, I have been having this bug for almost a week now, I have opened issues at Valve's gamescope, RedHat BugZilla and even talked with the OBS support on their Discord. This is the last place I have left then I don't know how to solve this issue anymore, so please if you have any idea on how I could better identify the problem help me out.

Here is the issue! Since the day after I updated to Fedora 40 KDE (from Fedora 39 KDE) whenever I record a video that is taken from gamescope with the Game Capture function on OBS, using obs-vkcapture the video in the preview and the recorded (or streamed) format will look like the video embedded below.

Here is the list of the odd things:

• The recording worked well until well over 24h after the upgrade to Fedora 40, so it's not something that became off after the upgrade.
• When recording a game using gamescope and obs-vkcapture outside of Steam the video works perfectly fine (example WoW running from Bottles)
• When recording a game on Steam without using gamescope the error doesn't show itself (for example capturing it with pipewire's window capture utility

The solutions I have tried as of now are as follows:

• I have tried downgrading the gamescope package to the version that existed until immediately before the Fedora 40 update, to no anvail.
• I have tried running a Steam game without gamescope enabled and it's being recorded fine.
• I have tried recording a game with gamescope enabled outside Steam and it's working fine
• I have tried completely removing OBS (which I am using the flatpak version btw) and reinstalling it removing every single setting in the .var directory and restarting clean, still nothing.
• I have also tried with or withouth HDR enabled on my system and going back to the standard profile all together, but that wasn't the problem either.

Screenshots or video https://youtu.be/7svRzATLp_0

System (please complete the following information):

• OS: Fedora Linux 40 (KDE Plasma) x86_64
• GPU: AMD Radeon RX 7900 XTX
• Mesa version: Mesa 24.0.6
• obs-vkcapture version:

OBS log:

23:23:04.188: Platform: Wayland
23:23:04.189: CPU Name: AMD Ryzen 9 7900X3D 12-Core Processor
23:23:04.189: CPU Speed: 4785.582MHz
23:23:04.189: Physical Cores: 12, Logical Cores: 24
23:23:04.189: Physical Memory: 64002MB Total, 47288MB Free
23:23:04.189: Kernel Version: Linux 6.8.7-300.fc40.x86_64
23:23:04.189: Flatpak Branch: stable
23:23:04.189: Flatpak Arch: x86_64
23:23:04.189: Flatpak Runtime: runtime/org.kde.Platform/x86_64/6.6
23:23:04.189: App Extensions:
23:23:04.189:  - com.obsproject.Studio.Plugin.OBSVkCapture
23:23:04.189:  - org.freedesktop.LinuxAudio.Plugins.TAP
23:23:04.189:  - org.freedesktop.LinuxAudio.Plugins.swh
23:23:04.189:  - com.obsproject.Studio.Locale
23:23:04.189: Runtime Extensions:
23:23:04.189:  - org.freedesktop.Platform.GL.default
23:23:04.189:  - org.freedesktop.Platform.openh264
23:23:04.189:  - org.freedesktop.Platform.VulkanLayer.OBSVkCapture
23:23:04.189:  - org.freedesktop.Platform.VulkanLayer.gamescope
23:23:04.189:  - org.gtk.Gtk3theme.Breeze
23:23:04.189:  - org.kde.KStyle.Adwaita
23:23:04.189:  - org.kde.Platform.Locale
23:23:04.189:  - org.freedesktop.Platform.GL.default
23:23:04.189: Flatpak Framework Version: 1.15.8
23:23:04.189: Desktop Environment: KDE (KDE)
23:23:04.189: Session Type: wayland
23:23:04.191: Qt Version: 6.6.3 (runtime), 6.6.2 (compiled)
23:23:04.191: Portable mode: false
23:23:04.212: OBS 30.1.2 (linux)
23:23:04.212: ---------------------------------
23:23:04.212: ---------------------------------
23:23:04.212: audio settings reset:
23:23:04.212: 	samples per sec: 48000
23:23:04.212: 	speakers:        2
23:23:04.212: 	max buffering:   960 milliseconds
23:23:04.212: 	buffering type:  dynamically increasing
23:23:04.215: ---------------------------------
23:23:04.215: Initializing OpenGL...
23:23:04.215: Using EGL/Wayland
23:23:04.223: Initialized EGL 1.5
23:23:04.232: Loading up OpenGL on adapter AMD AMD Radeon RX 7900 XTX (radeonsi, navi31, LLVM 17.0.6, DRM 3.57, 6.8.7-300.fc40.x86_64)
23:23:04.232: OpenGL loaded successfully, version 4.6 (Core Profile) Mesa 24.0.6 (git-c659c7e660), shading language 4.60
23:23:04.252: ---------------------------------
23:23:04.252: video settings reset:
23:23:04.252: 	base resolution:   1920x1080
23:23:04.252: 	output resolution: 1920x1080
23:23:04.252: 	downscale filter:  Bicubic
23:23:04.252: 	fps:               60/1
23:23:04.252: 	format:            NV12
23:23:04.252: 	YUV mode:          Rec. 709/Partial
23:23:04.252: NV12 texture support enabled
23:23:04.252: P010 texture support not available
23:23:04.254: Audio monitoring device:
23:23:04.254: 	name: Default
23:23:04.254: 	id: default
23:23:04.254: ---------------------------------
23:23:04.268: No AJA devices found, skipping loading AJA UI plugin
23:23:04.268: Failed to initialize module 'aja-output-ui.so'
23:23:04.279: No AJA devices found, skipping loading AJA plugin
23:23:04.279: Failed to initialize module 'aja.so'
23:23:04.284: Failed to load 'en-US' text for module: 'decklink-captions.so'
23:23:04.291: Failed to load 'en-US' text for module: 'decklink-output-ui.so'
23:23:04.297: A DeckLink iterator could not be created.  The DeckLink drivers may not be installed
23:23:04.298: Failed to initialize module 'decklink.so'
23:23:04.454: [pipewire] Available captures:
23:23:04.454: [pipewire]     - Desktop capture
23:23:04.454: [pipewire]     - Window capture
23:23:04.476: v4l2loopback not installed, virtual camera disabled
23:23:04.486: [obs-browser]: Version 2.23.4
23:23:04.486: [obs-browser]: CEF Version 103.0.5060.134 (runtime), 103.0.0-5060-shared-textures.2594+g17f8588+chromium-103.0.5060.134 (compiled)
23:23:04.501: VAAPI: API version 1.19
23:23:04.502: FFmpeg VAAPI H264 encoding supported
23:23:04.507: FFmpeg VAAPI AV1 encoding supported
23:23:04.512: FFmpeg VAAPI HEVC encoding supported
23:23:04.596: [obs-websocket] [obs_module_load] you can haz websockets (Version: 5.4.2 | RPC Version: 1)
23:23:04.596: [obs-websocket] [obs_module_load] Qt version (compile-time): 6.6.2 | Qt version (run-time): 6.6.3
23:23:04.596: [obs-websocket] [obs_module_load] Linked ASIO Version: 102900
23:23:04.600: [obs-websocket] [obs_module_load] Module loaded.
23:23:04.632: [linux-vkcapture] plugin loaded successfully (version 1.5.0)
23:23:04.632: ---------------------------------
23:23:04.632:   Loaded Modules:
23:23:04.632:     linux-vkcapture.so
23:23:04.632:     text-freetype2.so
23:23:04.632:     rtmp-services.so
23:23:04.632:     obs-x264.so
23:23:04.632:     obs-websocket.so
23:23:04.632:     obs-webrtc.so
23:23:04.632:     obs-vst.so
23:23:04.632:     obs-transitions.so
23:23:04.632:     obs-qsv11.so
23:23:04.632:     obs-outputs.so
23:23:04.632:     obs-libfdk.so
23:23:04.632:     obs-filters.so
23:23:04.632:     obs-ffmpeg.so
23:23:04.632:     obs-browser.so
23:23:04.632:     linux-v4l2.so
23:23:04.632:     linux-pulseaudio.so
23:23:04.632:     linux-pipewire.so
23:23:04.632:     linux-jack.so
23:23:04.632:     linux-capture.so
23:23:04.632:     image-source.so
23:23:04.632:     frontend-tools.so
23:23:04.632:     decklink-output-ui.so
23:23:04.632:     decklink-captions.so
23:23:04.632: ---------------------------------
23:23:04.632: ==== Startup complete ===============================================
23:23:04.662: All scene data cleared
23:23:04.662: ------------------------------------------------
23:23:04.666: pulse-input: Server name: 'PulseAudio (on PipeWire 1.0.5) 15.0.0'
23:23:04.666: pulse-input: Audio format: s32le, 48000 Hz, 8 channels
23:23:04.666: pulse-input: Started recording from 'alsa_output.pci-0000_08_04.0.analog-surround-71.monitor' (default)
23:23:04.666: [Loaded global audio device]: 'Desktop Audio'
23:23:04.666: pulse-input: Server name: 'PulseAudio (on PipeWire 1.0.5) 15.0.0'
23:23:04.666: pulse-input: Audio format: s16le, 48000 Hz, 2 channels
23:23:04.666: pulse-input: Started recording from 'alsa_input.usb-Blue_Microphones_Yeti_Stereo_Microphone_REV8-00.pro-input-0' (default)
23:23:04.666: [Loaded global audio device]: 'Mic/Aux'
23:23:04.666: ext_screencopy_manager_v1 not available
23:23:04.668: Switched to scene 'Scene'
23:23:04.668: ------------------------------------------------
23:23:04.668: Loaded scenes:
23:23:04.668: - scene 'Scene':
23:23:04.668:     - source: 'Game Capture' (vkcapture-source)
23:23:04.668: ------------------------------------------------
23:23:05.215: adding 21 milliseconds of audio buffering, total audio buffering is now 21 milliseconds (source: Desktop Audio)
23:23:05.215: 
23:23:05.237: adding 21 milliseconds of audio buffering, total audio buffering is now 42 milliseconds (source: Mic/Aux)
23:23:05.237: 
23:23:39.373: [linux-vkcapture] Client 1 connected (pid=0)
23:23:41.804: [linux-vkcapture] Creating texture from dmabuf 1272x680 modifier:144115188757872388
23:23:41.804: [linux-vkcapture]  [0] fd:45 stride:5120 offset:0
23:23:41.804: [linux-vkcapture]  [1] fd:46 stride:2048 offset:3932160
23:23:41.825: [linux-vkcapture] Client 2 connected (pid=0)
23:23:56.232: [linux-vkcapture] Client 2 disconnected
23:23:56.330: [linux-vkcapture] Client 1 disconnected
23:24:21.162: ==== Shutting down ==================================================
23:24:21.164: pulse-input: Stopped recording from 'alsa_output.pci-0000_08_04.0.analog-surround-71.monitor'
23:24:21.164: pulse-input: Got 3058 packets with 3669600 frames
23:24:21.164: pulse-input: Stopped recording from 'alsa_input.usb-Blue_Microphones_Yeti_Stereo_Microphone_REV8-00.pro-input-0'
23:24:21.164: pulse-input: Got 3059 packets with 3670800 frames
23:24:21.164: [linux-vkcapture] destroy
23:24:21.185: All scene data cleared
23:24:21.185: ------------------------------------------------
23:24:21.226: [linux-vkcapture] plugin unloaded
23:24:21.227: [obs-websocket] [obs_module_unload] Shutting down...
23:24:21.227: Tried to call obs_frontend_remove_event_callback with no callbacks!
23:24:21.227: [obs-websocket] [obs_module_unload] Finished shutting down.
23:24:21.232: [Scripting] Total detached callbacks: 0
23:24:21.232: Freeing OBS context data
23:24:21.244: == Profiler Results =============================
23:24:21.244: run_program_init: 565.333 ms
23:24:21.244:  ┣OBSApp::AppInit: 3.156 ms
23:24:21.244:  ┃ ┗OBSApp::InitLocale: 0.516 ms
23:24:21.244:  ┗OBSApp::OBSInit: 490.412 ms
23:24:21.244:    ┣obs_startup: 2.325 ms
23:24:21.244:    ┗OBSBasic::OBSInit: 466.688 ms
23:24:21.244:      ┣OBSBasic::InitBasicConfig: 0.091 ms
23:24:21.244:      ┣OBSBasic::ResetAudio: 0.187 ms
23:24:21.244:      ┣OBSBasic::ResetVideo: 41.718 ms
23:24:21.244:      ┃ ┗obs_init_graphics: 39.719 ms
23:24:21.244:      ┃   ┗shader compilation: 19.793 ms
23:24:21.244:      ┣OBSBasic::InitOBSCallbacks: 0.003 ms
23:24:21.244:      ┣OBSBasic::InitHotkeys: 0.018 ms
23:24:21.244:      ┣obs_load_all_modules2: 377.492 ms
23:24:21.244:      ┃ ┣obs_init_module(aja-output-ui.so): 0.064 ms
23:24:21.244:      ┃ ┣obs_init_module(aja.so): 0.072 ms
23:24:21.244:      ┃ ┣obs_init_module(decklink-captions.so): 0 ms
23:24:21.244:      ┃ ┣obs_init_module(decklink-output-ui.so): 0 ms
23:24:21.244:      ┃ ┣obs_init_module(decklink.so): 0.106 ms
23:24:21.244:      ┃ ┣obs_init_module(frontend-tools.so): 53.143 ms
23:24:21.244:      ┃ ┣obs_init_module(image-source.so): 0.01 ms
23:24:21.244:      ┃ ┣obs_init_module(linux-capture.so): 0 ms
23:24:21.244:      ┃ ┣obs_init_module(linux-jack.so): 0.002 ms
23:24:21.244:      ┃ ┣obs_init_module(linux-pipewire.so): 9.43 ms
23:24:21.244:      ┃ ┣obs_init_module(linux-pulseaudio.so): 0.002 ms
23:24:21.244:      ┃ ┣obs_init_module(linux-v4l2.so): 7.214 ms
23:24:21.244:      ┃ ┣obs_init_module(obs-browser.so): 0.048 ms
23:24:21.244:      ┃ ┣obs_init_module(obs-ffmpeg.so): 18.362 ms
23:24:21.244:      ┃ ┃ ┗nvenc_check: 1.363 ms
23:24:21.244:      ┃ ┣obs_init_module(obs-filters.so): 0.021 ms
23:24:21.244:      ┃ ┣obs_init_module(obs-libfdk.so): 0.002 ms
23:24:21.244:      ┃ ┣obs_init_module(obs-outputs.so): 0.008 ms
23:24:21.244:      ┃ ┣obs_init_module(obs-qsv11.so): 12.176 ms
23:24:21.244:      ┃ ┣obs_init_module(obs-transitions.so): 0.006 ms
23:24:21.244:      ┃ ┣obs_init_module(obs-vst.so): 0.003 ms
23:24:21.244:      ┃ ┣obs_init_module(obs-webrtc.so): 0.008 ms
23:24:21.244:      ┃ ┣obs_init_module(obs-websocket.so): 3.552 ms
23:24:21.244:      ┃ ┣obs_init_module(obs-x264.so): 0.003 ms
23:24:21.244:      ┃ ┣obs_init_module(rtmp-services.so): 0.735 ms
23:24:21.244:      ┃ ┣obs_init_module(text-freetype2.so): 0.019 ms
23:24:21.244:      ┃ ┗obs_init_module(linux-vkcapture.so): 0.115 ms
23:24:21.244:      ┣OBSBasic::InitService: 0.662 ms
23:24:21.244:      ┣OBSBasic::ResetOutputs: 0.14 ms
23:24:21.244:      ┣OBSBasic::CreateHotkeys: 0.017 ms
23:24:21.244:      ┣OBSBasic::InitPrimitives: 0.073 ms
23:24:21.244:      ┗OBSBasic::Load: 35.147 ms
23:24:21.244: obs_hotkey_thread(25 ms): min=0 ms, median=0 ms, max=0.001 ms, 99th percentile=0 ms, 100% below 25 ms
23:24:21.244: audio_thread(Audio): min=0.005 ms, median=0.017 ms, max=0.57 ms, 99th percentile=0.037 ms
23:24:21.244: obs_graphics_thread(16.6667 ms): min=0.065 ms, median=0.18 ms, max=111.566 ms, 99th percentile=0.369 ms, 99.8262% below 16.667 ms
23:24:21.244:  ┣tick_sources: min=0 ms, median=0.003 ms, max=0.197 ms, 99th percentile=0.01 ms
23:24:21.244:  ┣output_frame: min=0.041 ms, median=0.067 ms, max=3.365 ms, 99th percentile=0.19 ms
23:24:21.244:  ┃ ┗gs_context(video->graphics): min=0.041 ms, median=0.067 ms, max=3.364 ms, 99th percentile=0.19 ms
23:24:21.244:  ┃   ┣render_video: min=0.006 ms, median=0.013 ms, max=1.625 ms, 99th percentile=0.11 ms
23:24:21.244:  ┃   ┃ ┗render_main_texture: min=0.004 ms, median=0.011 ms, max=1.618 ms, 99th percentile=0.107 ms
23:24:21.244:  ┃   ┗gs_flush: min=0.001 ms, median=0.04 ms, max=0.367 ms, 99th percentile=0.079 ms
23:24:21.244:  ┗render_displays: min=0.001 ms, median=0.091 ms, max=111.488 ms, 99th percentile=0.175 ms
23:24:21.244: =================================================
23:24:21.244: == Profiler Time Between Calls ==================
23:24:21.244: obs_hotkey_thread(25 ms): min=25.009 ms, median=25.056 ms, max=26.326 ms, 99.9675% within ±2% of 25 ms (0% lower, 0.0325415% higher)
23:24:21.244: obs_graphics_thread(16.6667 ms): min=0.263 ms, median=16.667 ms, max=111.572 ms, 99.4134% within ±2% of 16.667 ms (0.304149% lower, 0.282425% higher)
23:24:21.244: =================================================
23:24:21.245: Number of memory leaks: 0

Game log: steam-1113000.log