Dota 2 forever stuck in bootscreen after loosing vulkan device when VK_EXT_graphics_pipeline_library is used

[mkjOoB]

Your system information

• System information from steam (Steam -> Help -> System Information) in a gist:https://gist.github.com/mkjOoB/4c95c65f5f81a27682853a4c892e4d75#file-systeminfo-log
• Have you checked for system updates?: Yes
• Are you using the latest stable video driver available for your system? No, vulkan-beta
• Have you verified the game files?: Yes

Please describe your issue in as much detail as possible:

Describe what you expected should happen and what did happen. Please link any large pastes as a Github Gist.

Since the "Battle Report" dota 2 update on june 9 2022, vulkan's VK_EXT_graphics_pipeline_library extension support was added. Now game fails to load with GTX 1080 and -vulkan, when having vulkan-beta drivers supporting this extension installed. VK_EXT_graphics_pipeline_library is not yet in stable driver branch. Tested on Linux with nvidia-drivers 515.49.15 and windows 10 LTSC 1809 with nvidia-drivers 517.22

On Linux Steam was launched with: PRESSURE_VESSEL_WRAP_GUI=1 GAME_DEBUGGER=gdb STEAM_LINUX_RUNTIME_LOG=1 STEAM_RUNTIME=1 steam -console -silent -single_core

The game was ran with steam-linux-runtime debbuging gui enabled, tested both with Steam Runtime 2 (soldier) 0.20220803 and Steam Runtime 3 (sniper) 0.20220119.0 which gave no visible difference. That's as per: https://github.com/ValveSoftware/steam-runtime/blob/master/doc/reporting-steamlinuxruntime-bugs.md#advanced-debugging

Logs were collected by selecting "Run interacting shell" set to "Instead of running the command" and then typing "$@" | tee /tmp/dota2.log

When running the game with -vulkan without VK_EXT_graphics_pipeline_library, it works.

My take is: As far as I've notised, the issue persist on Windows and Linux, proton or native, with STEAM_RUNTIME=1 or by running ./dota2 directly bypassing SteamLaunch (which is how I ussually run the game with -gl being on wayland). Either VK_EXT_graphics_pipeline_library is not ready in the game or in the drivers, or is not supported on my gpu generation -- Nvidia's Pascal. I wasn't able to find anything regarding VK_EXT_graphics_pipeline_library on NVidia's public dev forum. It would be interesting to know whether behaviour differse on Turing or Ampere GPUs or with MESA whom are still figuring VK_EXT_graphics_pipeline_library out. GTX 1080 is my only card someone should retest.

Steps for reproducing this issue:

1. Run dota 2 on Linux with -vulkan when running nvidia drivers (only vulkan-beta branch for now) with VK_EXT_graphics_pipeline_library extension support.
2. The game will open a window with a working cursor over dota 2 logo splashscreen and will load forever.
3. Observe vulkan device lost errors in the terminal, see log.

[danginsburg]

Thanks for the detailed bug report. You are getting a VK_ERROR_DEVICE_LOST which indicates a GPU hang. A question for clarification:

As far as I've notised, the issue persist on Windows and Linux, proton or native, with STEAM_RUNTIME=1 or by running ./dota2

You are saying that you see the same issue on Windows. Which NVIDIA driver is that with? You are using 515.49.15 on Linux, the latest Windows Vulkan beta driver is 517.22.

[danginsburg]

Oh, nevermind, I see you said:

Tested on Linux with nvidia-drivers 515.49.15 and windows 10 LTSC 1809 with nvidia-drivers 517.22

Hmm...OK, we'll have to see if we can repro.

[danginsburg]

Not repro'ing for me here on Win10 517.22 GTX 2080. Can you try adding -vulkan_disable_graphics_pipeline_library to your launch options just to confirm it is indeed VK_EXT_graphics_pipeline_library support that is causing the issue?

[mkjOoB]

I retested with -vulkan_disable_graphics_pipeline_library and without. Adding -vulkan_disable_graphics_pipeline_library makes the game work. Retest with -vulkan -vulkan_disable_graphics_pipeline_library: https://gist.github.com/mkjOoB/1f549bb1fc9b98244e1150573efc0527#file-dota2-with-vk_ext_graphics_pipeline_library-disabled-linux-log Retest with just -vulkan, again same error: https://gist.github.com/mkjOoB/1f549bb1fc9b98244e1150573efc0527#file-dota2-with-vk_ext_graphics_pipeline_library-enabled-linux-log

I also fired up my windows 10 LTSC 1809 and it seems that -vulkan_disable_graphics_pipeline_library makes vulkan work.

~~I would like to get similar logs for windows but idk how to log dota 2 on windows, there's also no minidump generated in Steam/dumps/ since the game does not crash and must be killed.~~

I wonder if this repro's on other gtx 1080 card or rather all pascal 10 series cards.

[mkjOoB]

Ok, so I managed to get dota2's logs for windows. Here's one for '"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe" -g -G -o %command% -vulkan' Log: https://gist.github.com/mkjOoB/1f549bb1fc9b98244e1150573efc0527#file-dota2-with-vk_ext_graphics_pipeline_library-windows10-log and '"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe" -g -G -o %command% -vulkan -vulkan_disable_graphics_pipeline_library' Log: https://gist.github.com/mkjOoB/1f549bb1fc9b98244e1150573efc0527#file-dota2-without-vk_ext_graphics_pipeline_library-windows10-log

[mkjOoB]

Please let me know if there's more I can do to help.

[pdaniell-nv]

@mkjOoB would you mind trying the latest beta driver from here: https://developer.nvidia.com/vulkan-driver. Many thanks.

[mkjOoB]

Here's retest with updated nvidia drivers 517.57 on windows 10, same story, though now with validation layers enabled. Used the standard preset in vkconfig from vulkan 1.3.224 sdk with synchronization stuff enabled, too. Log: https://gist.github.com/mkjOoB/1f549bb1fc9b98244e1150573efc0527#file-dota2-retest-with-synchronization-validation-vk_ext_graphics_pipeline_library-windows10-log

And a Linux retest with latest nvidia drivers 515.49.19 and standard vulkan validation layer settings without additions, same dead hang. Log: https://gist.github.com/mkjOoB/1f549bb1fc9b98244e1150573efc0527#file-dota2-retest-with-standard-validation-vk_ext_graphics_pipeline_library-linux-log

[pdaniell-nv]

@mkjOoB many thanks for the follow up and the logs. I think we may have found the issue and will follow up again when the solution is available. In the meantime you'll need to use the non-beta NVIDIA drivers to play Dota2. Sorry for the inconvenience.

[torgarak]

Can confirm with today's GeForce driver release 522.25. Program's not responding during start-up. VK_EXT_graphics_pipeline_library is already in the stable channel.

[dimon222]

I'm 1080 ti user and I'm seeing the same issue with newer driver stable 526.47. This flag solves it for me -vulkan_disable_graphics_pipeline_library

[starwavex]

Adding -vulkan_disable_graphics_pipeline_library in game launch options works for me. (I got gtx1080 gpu with 520.56.06 nvidia linux driver)

[Squall-Leonhart]

This extension is now available in the release drivers as of 526.47, and the issue is present there.

[dimon222]

To exacerbate problem, the issue is present even on Windows and Dota is getting rid of OpenGL based on latest announcement. I wonder where will all stuck users go. And no, not this random GitHub thread.

[jaccarmac]

I experienced the same issue on a Quadro 2100M, 470.x driver series. The launch flag works. Because it's an old card, I have drivers >= 471 masked.

[tmzhuang]

On linux 6.0.6, nvidia driver version 520.56.06-8. Found this issue because my game never loaded after recent updates, but passing in -novid -console -high -prewarm -map dota to launch properties worked for me.

[KGB333]

To exacerbate problem, the issue is present even on Windows and Dota is getting rid of OpenGL based on latest announcement. I wonder where will all stuck users go. And no, not this random GitHub thread.

yeah. I'm on Windows and since the driver 522.25 update I can no longer load the game without Adding -vulkan_disable_graphics_pipeline_library in game launch options. BTW the issue is only present on older hardware like gtx series cards (mine is TITAN Xp). And I have no issue to launch the game on a RTX platform (2070s).

[Squall-Leonhart]

~~Fullscreen Exclusive also breaks the game with similar crashes on these new drivers, if you are on 526, the new Layered on DXGI Swapchain option gets you back into game, the Native setting is causing crashes with FSE.~~

False alarm, just a corrupt vulkan pipeline cache.

[bricss]

Same stuff here:

• Platform: Dell XPS 15 9560
• OS: Win11 22H2
• Graphics: GeForce GTX 1050
• Driver: GeForce Game Ready Driver 526.61 hf

Game doesn't start w/o -vulkan_disable_graphics_pipeline_library flag and hang-up on load screen

[Strykar]

Noticed the issue since the last driver update on Arch Linux (nvidia 520.56.06) on a GTX 950 and 1080. Adding -vulkan_disable_graphics_pipeline_library fixes the issue and Dota goes past the splash screen and runs fine.

Sysinfo - https://gist.github.com/Strykar/7e5a05c1c252b38adc6d574ba3dfa9fb

[danginsburg]

I believe we have a fix for this coming in the next Dota update. The root cause of the issue is described here: https://github.com/KhronosGroup/Vulkan-ValidationLayers/issues/4768

[danginsburg]

This update has shipped. Please let us know if it has fixed the issue for you.

[Squall-Leonhart]

the framerate is trash with this update.

[jaccarmac]

I can confirm that the issue is fixed with an update. I'm not experiencing the framerate drops others are, either.

[KGB333]

I can confirm that the issue is fixed with an update. I'm not experiencing the framerate drops others are, either.

same here, everthing is good

[starwavex]

After the update, the game loads up to visible game background, but then hangs. Sometimes it shows the main menu and instantly hangs. Yet I'm able to move the cursor, although the interface doesn't respond. Otherwise I just see the background image. Tried completely cleaning vulkan cache - no success. Still only if I add -vulkan_disable_graphics_pipeline_library the issue vanishes. GTX 1080 + nvidia (linux) 520.56.06

[mkjOoB]

@danginsburg Hmm, still broken for me somehow. That's still on 520.56.06 GTX 1080. Tested on wayland and xorg sessions. Removed all the flags but -vulkan. Goes past the dota 2 logo but then displays gradient background for the main menu and stays like this while spewing out same culprit device lost error which seems to be forever. I even used a distribution's precompiled kernel for retesting this, shouldn't be my system's configuration problem. The replay is from sway's wayland session. I purged game files and reinstalled before major testing. Also steam vulkan shader caching settings doesn't seem to change a thing, I usually go without it being on nvidia, I've turned it on and had it do the full rebuild after reinstall just for testing sake. I'll test bit later, if it's still doesn't work under windows 10.

https://user-images.githubusercontent.com/82988916/200928391-a34e0571-f61a-42e2-8bce-915747717bc9.mp4

dota2-still-broken-linux.0.log

[mkjOoB]

@KGB333 @jaccarmac @starwavex Are you sure that's with -vulkan and not with -vulkan -vulkan_disable_graphics_pipeline_library? This should only repro on nvidia pre turing cards.

[KGB333]

@KGB333 @jaccarmac @starwavex Are you sure that's with -vulkan and not with -vulkan -vulkan_disable_graphics_pipeline_library? This should only repro on nvidia pre turing cards.

100% sure, run the game nice and smooth with my Titan Xp.

[starwavex]

@KGB333 @jaccarmac @starwavex Are you sure that's with -vulkan and not with -vulkan -vulkan_disable_graphics_pipeline_library? This should only repro on nvidia pre turing cards.

@mkjOoB I'm sure. And I experience exactly the same what you've recorded on video.