[Bug]: Cannot capture a game using the OBS Vulkan Capture option

[kozross]

Describe the bug

When I try to run a game inside a bottle with the 'OBS Vulkan Capture' option enabled, OBS does not capture the game.

To Reproduce

1. Install any game that can use DXVK (that is, runs on DirectX 10 or 11) into a bottle.
2. Go to 'Preferences' and toggle the 'OBS Vulkan Capture' option to on.
3. Start the game.
4. Try to use 'Game Capture' in OBS to capture the footage.
5. Observe that it fails.

Package

Flatpak from Flathub

Distribution

Arch Linux

Health Check

Version: 2022.6.28-brescia
Display:
    X.org: true
    X.org (port): :99.0
    Wayland: true
Graphics:
    vendors:
        amd:
            vendor: amd
            envs:
                DRI_PRIME: '1'
            icd: /usr/lib/x86_64-linux-gnu/GL/vulkan/icd.d/radeon_icd.x86_64.json:/usr/lib/i386-linux-gnu/GL/vulkan/icd.d/radeon_icd.i686.json
    prime:
        integrated: null
        discrete: null
Kernel:
    Type: Linux
    Version: 5.18.7-arch1-1
Distro:
    Name: GNOME
    Version: '"42 (Flatpak runtime)"'
Disk:
    Total: 16776712192
    Free: 16776564736
RAM:
    MemTotal: 31.2GiB
    MemAvailable: 26.3GiB
Tools:
    cabextract: true
    p7zip: true
    patool: true
    glibc_min: '2.33'
Bottles_envs: null

Display Protocol

Wayland

Display Backend used by Bottles

Wayland

Additional context

This may be an issue with the capture plugin itself; the reason I am reporting this is because this used to work before, on a very similar game, in a prior Bottles version, but seems to not work any more.

[kozross]

Further note: this is definitely a regression somewhere. I just tried running the old game where capture used to work, and it no longer works, in a similar way to the above.

[mirkobrombin]

We changed nothing on this

[kozross]

@mirkobrombin - how does that option currently work? I have a suspicion, based on documentation for the plugin that the old envvar option no longer works.

[mirkobrombin]

I thought it was using that syntax, I’ll fix with the next release. Can you test through launch options?

[kozross]

@mirkobrombin I checked obs-vkcapture %command% - this works everywhere, and the capture also works. However, when I try to do something similar with Bottles, I get double the Wine desktop (I change launch options to obs-vkcapture %command%), but the actual game does not run.

[mirkobrombin]

Do you have mangohud, gamescope or Gamemode enabled?

[kozross]

I definitely don't have mangohud or gamescope. I tried turning Gamemode off, but this still causes the same problem with obs-vkcapture %command%; I get double Wine desktops, but the game won't start.

[axtloss]

I tried to recreate the issue with no success. Adding game capture to obs as a source and simply enabling vk capture in obs works without any issue and I can capture game output without any issues. Maybe this is an issue with your setup? I'm using obs 27.2.4 from flathub and the latest bottles version also from flathub.

[kozross]

@axtloss and @mirkobrombin I also heard a similar report from someone else. I'll try a clean bottle and see if I can replicate this in that setting. If not, we can close.

[kozross]

Sorry for the delay. I've just set up a brand new, clean bottle, installed ToCS3 into it, then tried to capture it into OBS. I basically get the exact same problem (can't capture anything) if I use the built-in support, while if I use obs-vkcapture %command%, I get double Wine desktop again.

I genuinely am quite confused at this point, since nobody seems to be able to replicate this issue but me. I'm using the latest OBS version from Flathub, as well as the latest Bottles version from the same source.

@axtloss - were your tests on X or Wayland? I'm starting to wonder if this might be the problem here.

[axtloss]

My tests all were on wayland, maybe it's a problem with the game you're testing?

[kozross]

@axtloss I tried recording a game I previously had no issue with: same deal.

[axtloss]

This is really weird, could it maybe be an issue with obs-vkcapture itself and not with bottles?

[kozross]

@axtloss That's probably not it either. Steam games capture just fine.

[Jookia]

I did a lot of debugging today and found a way to reproduce this: Enable Wine's virtual desktop feature. This also breaks launching with obs-vkcapture %command%

Edit: Koz confirmed that disabling it can capture, but then enabling it again lets it still capture. Restarting bottles then breaks capture.

[kozross]

To extend what @Jookia said above: this happens regardless of how you virtualize. I tried doing this using both virtual desktop settings via winecfg, as well as the builtin option in Bottles itself: same effect.

[axtloss]

Then this is not a bottles issue, the wine virtual desktop doesn't use Vulkan and also isolates any window in it, which is why obs can't capture it.

[Jookia]

But if you switch back and forth it can capture just fine.

On Sat, Aug 06, 2022 at 01:44:06AM -0700, axtloss wrote:

Then this is not a bottles issue, the wine virtual desktop doesn't use Vulkan and also isolates any window in it, which is why obs can't capture it.

-- Reply to this email directly or view it on GitHub: https://github.com/bottlesdevs/Bottles/issues/1704#issuecomment-1207176239 You are receiving this because you were mentioned.

Message ID: @.***>

[axtloss]

You mean the capture works when disabling the virtual desktop and re-enabling it?

[Jookia]

Yes.

On Sat, Aug 06, 2022 at 02:34:34AM -0700, axtloss wrote:

You mean the capture works when disabling the virtual desktop and re-enabling it?

-- Reply to this email directly or view it on GitHub: https://github.com/bottlesdevs/Bottles/issues/1704#issuecomment-1207183083 You are receiving this because you were mentioned.

Message ID: @.***>

[mirkobrombin]

Double-check the program launch options to make sure the virtual desktop is not overwritten

[Jookia]

How do you check this? I'm fairly sure it's not overwritten. Though I'm not sure what that means.

On Sat, Aug 06, 2022 at 03:44:23AM -0700, Mirko Brombin wrote:

Double-check the program launch options to make sure the virtual desktop is not overwritten

-- Reply to this email directly or view it on GitHub: https://github.com/bottlesdevs/Bottles/issues/1704#issuecomment-1207192574 You are receiving this because you were mentioned.

Message ID: @.***>

[mirkobrombin]

Your programs > the chosen program > launch options > overrides

[Jookia]

I'm fairly sure unless it gets overriden by the virtual desktop option. Just switching the virtual desktop off then on makes it able to capture again.

[kozross]

@axtloss Also, before I hit this issue, I could capture games in a virtual desktop just fine.