ValveSoftware
std::bad_cast exception from std::regex when using vkconfig to enable vulkan layers
Your system information
- Steam Runtime Version: SteamLinuxRuntime_sniper
- Distribution (e.g. Ubuntu 18.04): Ubuntu 24.04.1 LTS
- Link to your full system information (Help -> Steam Runtime Diagnostics) in a Gist: https://gist.github.com/jeremyg-lunarg/83ff353cd6e47b2cb450ec83266f0cd1
- Have you checked for system updates?: Yes
- What compatibility tool are you using?: Steam Linux Runtime
- What versions are listed in
steamapps/common/SteamLinuxRuntime/VERSIONS.txt? not installed - What versions are listed in
steamapps/common/SteamLinuxRuntime_soldier/VERSIONS.txt? not installed - What versions are listed in
steamapps/common/SteamLinuxRuntime_sniper/VERSIONS.txt?
#Name Version Runtime Runtime_Version Comment
depot 0.20240820.99315 # Overall version number
pressure-vessel 0.20240820.0 scout # pressure-vessel-bin.tar.gz
scripts 0.20240820.0 # from steam-runtime-tools
sniper 0.20240820.99315 sniper 0.20240820.99315 # sniper_platform_0.20240820.99315/
Please describe your issue in as much detail as possible:
I was trying to use a development version of a vulkan debug tool called Crash Diagnostic Layer (https://github.com/LunarG/CrashDiagnosticLayer). When using local builds of this layer AND enabling with vkconfig I get std::bad_cast. This also happens when using the Vulkan-ValidationLayer.
Local builds of both of the above layers work correctly when enabled using the environment variables VK_INSTANCE_LAYERS and VK_LAYER_PATH.
Log from using vkconfig:
>export STEAM_LINUX_RUNTIME_LOG=1
>./run-in-sniper ../Counter-Strike\ Global\ Offensive/game/cs2.sh
pressure-vessel-adverb[85534]: W: Unknown expansion of the dl string token $PLATFORM: /run/host/usr/libexec/steam-runtime-tools-0/x86_64-linux-gnu-detect-platform not found
pressure-vessel-adverb[85534]: W: Unable to set up VDPAU driver search path
pressure-vessel-adverb[85534]: W: Cannot run pressure-vessel-locale-gen: Failed to execute child process “/run/host/usr/lib/x86_64-linux-gnu/pressure-vessel-locale-gen” (No such file or directory)
Loaded /home/jeremyg/.local/share/Steam/steamapps/common/Counter-Strike Global Offensive/game/bin/linuxsteamrt64/libengine2.so, got 0x6368ed668a40
Using breakpad crash handler
[S_API] SteamAPI_Init(): Loaded '/home/jeremyg/.local/share/Steam/linux64/steamclient.so' OK.
Setting breakpad minidump AppID = 730
Forcing breakpad minidump interfaces to load
Looking up breakpad interfaces from steamclient
Calling BreakpadMiniDumpSystemInit
09/25 10:31:00 minidumps folder is set to /tmp/dumps
09/25 10:31:00 Init: Installing breakpad exception handler for appid(730)/version(9173482)/tid(85542)
Looking up breakpad interfaces from steamclient
Calling BreakpadMiniDumpSystemInit
SteamInternal_SetMinidumpSteamID: Caching Steam ID: 76561198029004697 [API loaded yes]
SteamInternal_SetMinidumpSteamID: Setting Steam ID: 76561198029004697
Setting breakpad minidump AppID = 2347771
Loaded libSDL3.so.0, got 0x6368ed77b720
Loaded /home/jeremyg/.local/share/Steam/steamapps/common/Counter-Strike Global Offensive/game/bin/linuxsteamrt64/libtier0.so, got 0x6368ed64a0f0
Loaded /home/jeremyg/.local/share/Steam/steamapps/common/Counter-Strike Global Offensive/game/bin/linuxsteamrt64/libfilesystem_stdio.so, got 0x6368ed7ea1d0
Loaded /home/jeremyg/.local/share/Steam/steamapps/common/Counter-Strike Global Offensive/game/bin/linuxsteamrt64/libinputsystem.so, got 0x6368ed7fca00
Loaded /home/jeremyg/.local/share/Steam/steamapps/common/Counter-Strike Global Offensive/game/bin/linuxsteamrt64/liblocalize.so, got 0x6368ed810350
Loaded /home/jeremyg/.local/share/Steam/steamapps/common/Counter-Strike Global Offensive/game/bin/linuxsteamrt64/librendersystemvulkan.so, got 0x6368ed8227c0
Loaded /home/jeremyg/.local/share/Steam/steamapps/common/Counter-Strike Global Offensive/game/bin/linuxsteamrt64/libresourcesystem.so, got 0x6368ed835410
Loaded /home/jeremyg/.local/share/Steam/steamapps/common/Counter-Strike Global Offensive/game/bin/linuxsteamrt64/libschemasystem.so, got 0x6368ed847f40
cat: write error: Broken pipe
terminate called after throwing an instance of 'std::bad_cast'
what(): std::bad_cast
crash_20240925103101_2.dmp[85559]: Uploading dump (out-of-process)
/tmp/dumps/crash_20240925103101_2.dmp
../Counter-Strike Global Offensive/game/cs2.sh: line 117: 85542 Aborted ${STEAM_RUNTIME_PREFIX} ${GAME_DEBUGGER} "${GAMEROOT}"/${GAMEEXE} "$@"
crash_20240925103101_2.dmp[85559]: Finished uploading minidump (out-of-process): success = yes
crash_20240925103101_2.dmp[85559]: response: Discarded=1
crash_20240925103101_2.dmp[85559]: file ''/tmp/dumps/crash_20240925103101_2.dmp'', upload yes: ''Discarded=1''
Steps for reproducing this issue:
- Install latest vulkan SDK packages:
wget -qO - https://packages.lunarg.com/lunarg-signing-key-pub.asc | sudo apt-key add -
sudo wget -qO /etc/apt/sources.list.d/lunarg-vulkan-1.3.290-noble.list https://packages.lunarg.com/vulkan/1.3.290/lunarg-vulkan-1.3.290-noble.list
sudo apt update
sudo apt install vulkan-sdk
- Get CrashDiagnotic layer and build locally (https://github.com/LunarG/CrashDiagnosticLayer/blob/main/BUILD.md)
- Start vkconfig
- Enable the Crash Diagnostic preset
- Hit 'Edit...' then 'Add User-defined path..' add the
build/srcsubdirectory of the local CrashDiagnosticLayer build. - ./run-in-sniper ../Counter-Strike\ Global\ Offensive/game/cs2.sh which will fail with std::bad_cast crash_20240925103101_2.dmp.gz
Hello Jeremy, thank you for your report.
This is a bit of processing on the minidump:
libengine2.so 0x11 std::terminate()
libengine2.so 0x45 __cxa_throw
libengine2.so 0x2f __cxa_bad_cast
libengine2.so 0x4c std::__cxx11::collate<char> const& std::use_facet<std::__cxx11::collate<char> >(std::locale const&)
libVkLayer_crash_diagnostic.so 0x168bf8
[..]
libVkLayer_crash_diagnostic.so 0x1b24a4
libVkLayer_steam_fossilize.so /data/src/external/fossilize/layer/dispatch.cpp:138 Fossilize::CreateInstance
libVkLayer_MESA_device_select.so 0x3786
steamoverlayvulkanlayer.so /data/src/overlay/gameoverlayrenderer/vulkan/steamoverlayvulkanlayer.cpp:3098 vkCreateInstance
libvulkan.so.1.3.290 0x22408
libvulkan.so.1.3.290 0x31b04
librendersystemvulkan.so /data/src/rendersystem/vulkan/renderdevicemgrvulkan.cpp:852 CRenderDeviceMgrVulkan::Connect(void* (*)(char const*, int*))
That's roughly consistent with what I see locally with gdb, initially I was getting this exception inside std::regex but now it looks like it is std::iostream.
(gdb) bt
#0 0x00007ffff5fabe10 in __cxa_throw () from target:/home/jeremyg/.local/share/Steam/steamapps/common/Counter-Strike Global Offensive/game/bin/linuxsteamrt64/libengine2.so
#1 0x00007ffff5b2f33c in std::__throw_bad_cast() () from target:/home/jeremyg/.local/share/Steam/steamapps/common/Counter-Strike Global Offensive/game/bin/linuxsteamrt64/libengine2.so
#2 0x00007ffff6017cb9 in std::basic_ostream<char, std::char_traits<char> >& std::basic_ostream<char, std::char_traits<char> >::_M_insert<long>(long) ()
from target:/home/jeremyg/.local/share/Steam/steamapps/common/Counter-Strike Global Offensive/game/bin/linuxsteamrt64/libengine2.so
#3 0x00007fffdc97bfc2 in crash_diagnostic_layer::DurationToStr(std::chrono::duration<long, std::ratio<1l, 1000000000l> >) ()
from target:/home/jeremyg/src/CrashDiagnosticLayer/build/src/./libVkLayer_crash_diagnostic.so
#4 0x00007fffdc97c335 in crash_diagnostic_layer::Logger::DefaultLogCallback(VkDebugUtilsMessageSeverityFlagBitsEXT, unsigned int, VkDebugUtilsMessengerCallbackDataEXT const*, void*) ()
from target:/home/jeremyg/src/CrashDiagnosticLayer/build/src/./libVkLayer_crash_diagnostic.so
Can you provide the libVkLayer_crash_diagnostic.so you reproduced this with? Assuming it's the same file (build_id starts with f3e49bb0643fc86bacbe9b443c8d027d[..]) `and has debug symbols. Then I can expand the backtrace a little.
@smcv is our specialist for this kind of thing, but I'm not sure it's particularly good or intentional that the internal usage of std::regex in a Vulkan layer pulled from the host by pressure-vessel ends up calling into libengine.so code.
There's probably several problems at play here, one of them being our exposing of these symbols in libengine.so to begin with, but also the Vulkan layer may need to static link libstdc++) more aggressively.
First order of business is probably to chase broader reproduction and see if we see this happening on Ubuntu 22 or a non debian distro (Arch possibly since it's our primary dev environment).
I'm not sure it's particularly good or intentional that the internal usage of std::regex in a Vulkan layer pulled from the host by pressure-vessel ends up calling into libengine.so code
I think that's probably the root cause here. Is libengine.so statically linking parts of libstdc++ and accidentally or intentionally re-exporting them? If yes, it should avoid doing that: we cannot guarantee that its libstdc++ is completely compatible with anyone else's.
std::bad_cast
This sounds like some component is (quite reasonably!) complaining that the runtime type information for std::regex implementation A doesn't say it's an instance of std::regex implementation B.
One possible mitigation for this would be having Vulkan-Loader load its layers with RTLD_LOCAL if it doesn't already, and having the Source engine similarly load its internal libraries libengine.so, etc. with RTLD_LOCAL if it doesn't already.
One possible mitigation for this would be having Vulkan-Loader load its layers with
RTLD_LOCALif it doesn't already, and having the Source engine similarly load its internal librarieslibengine.so, etc. withRTLD_LOCALif it doesn't already.
The loader is using RTLD_LAZY | RTLD_LOCAL:
https://github.com/KhronosGroup/Vulkan-Loader/blob/31cfa022eff1061a1ec8b3f1cd17b5b709fa6fee/loader/vk_loader_platform.h#L388
but also the Vulkan layer may need to static link libstdc++) more aggressively.
having the layer statically link against libstdc++ seems to avoid the problem:
if(CMAKE_SYSTEM_NAME MATCHES "Linux")
+ target_link_libraries(crash_diagnostic PRIVATE -static-libgcc -static-libstdc++)
having the layer statically link against libstdc++ seems to avoid the problem
Yeah, that's a useful workaround if you can arrange for it to happen on the host system. It would be better if we can avoid the Source2 engine triggering this problem from the engine end, but that's difficult to achieve (I have a couple of ideas which might potentially help there, I'll see what happens after testing them).
The fundamental problem here is that there are two copies of std::regex in action here: the one in the host system's libstdc++ as linked by the Vulkan layer, and the one in the libstdc++ that the game engine links statically. Because the game engine exports its copy of libstdc++, ELF library semantics say that it "interposes" in the place of the one in the shared libstdc++ - the same mechanism that is in use when you use something like libtcmalloc or libasan to replace malloc() with a more optimized or more debuggable substitute. I don't think the Source2 engine necessarily wants to interpose its statically linked libstdc++ symbols, but it's accidentally what happens.
The loader is using
RTLD_LAZY | RTLD_LOCAL
That means the Vulkan layer can't cause the equivalent of this crash in Source2 engine code, but the Source2 engine can still cause this crash in Vulkan layer code. Using RTLD_LOCAL would only fully work if it was done symmetrically, "at both ends".
When using local builds of this layer AND enabling with vkconfig I get std::bad_cast. This also happens when using the Vulkan-ValidationLayer.
Local builds of both of the above layers work correctly when enabled using the environment variables VK_INSTANCE_LAYERS and VK_LAYER_PATH.
What does vkconfig actually do to get CS2 to load these layers?
I'm trying to keep my test systems relatively "ordinary" to make my testing as representative as possible of what will happen on end-user systems, so if possible I'd prefer not to install software from outside the official Arch and Ubuntu repositories that end users would not normally have, which includes the external Vulkan SDK packages.
OK, vkconfig is available in Arch (packaged in vulkan-extra-tools), so I can experiment with it without installing software from outside the distro.
It looks as though it arranges for extra layers to be loaded by writing out ~/.local/share/vulkan/implicit_layer.d/VkLayer_override.json as a temporary meta-layer that has the layer I requested as one of its component_layers, and then deleting that file when finished. I'll try to make an in-distro reproducer for this based on that information.
It looks as though it arranges for extra layers to be loaded by writing out
~/.local/share/vulkan/implicit_layer.d/VkLayer_override.jsonas a temporary meta-layer that has the layer I requested as one of itscomponent_layers, and then deleting that file when finished.
Exactly right. Does the runtime do something special for layers set in the VK_INSTANCE_LAYERS environment variable? That works on my system when vkconfig doesn't. And at this point it is the only part of this issue that doesn't make sense to me.
Does the runtime do something special for layers set in the VK_INSTANCE_LAYERS environment variable?
No, the source code doesn't even mention it. At the moment we "capture" every implicit and explicit layer from the host system, together with its recursive dependencies, and let Vulkan-Loader worry about whether the layer is actually runtime-enabled or not.
We use the environment variables that are part of how layers are discovered, like VK_ADD_LAYER_PATH, but we currently ignore all environment variables that are only used for the decision to enable or not enable.
(We're actually thinking of changing that, to avoid disabled layers creating action-at-a-distance side-effects - but that isn't implemented yet.)
You've said that when the validation layer is activated via VK_INSTANCE_LAYERS, it works, but when it's activated by being listed in a meta-layer, it crashes? At the moment, the only reason I can think of for how that could happen is that the loading order in Vulkan-Loader could end up being different.
I tried to reproduce this on an up-to-date Ubuntu 24.04 Nvidia system with the distro's validation layer (version 1.3.275.0-1), by creating this:
$ cat ~/.local/share/vulkan/implicit_layer.d/SteamRuntimeIssue694.json
{
"file_format_version": "1.1.2",
"layer": {
"api_version": "1.3.250",
"blacklisted_layers": [
],
"component_layers": [
"VK_LAYER_KHRONOS_validation"
],
"description": "Reproducer for https://github.com/ValveSoftware/steam-runtime/issues/694",
"disable_environment": {
"DISABLE_STEAM_RUNTIME_ISSUE_694": "1"
},
"implementation_version": "1",
"name": "STEAM_RUNTIME_ISSUE_694",
"override_paths": [
"/usr/share/vulkan/explicit_layer.d"
],
"type": "GLOBAL"
}
}
/proc/$(pgrep cs2 | tail -n1)/maps said that /run/host/usr/lib/x86_64-linux-gnu/libVkLayer_khronos_validation.so (which is the container's "view" of /usr/lib/x86_64-linux-gnu/libVkLayer_khronos_validation.so from the host system) was loaded, and my framerate was really bad, suggesting that the extra validation was indeed happening - but I couldn't reproduce the crash by loading CS2 and entering a training mission.
If you use a similar hand-written JSON file instead of vkconfig, does that reproduce the crash for you? Perhaps we can converge on a minimal reproducer by both working towards it from opposite directions?
Good idea. I was able to reproduce with the following:
~/.local/share/vulkan/implicit_layer.d/bad_cast.json:
{
"file_format_version": "1.1.2",
"layer": {
"api_version": "1.3.290",
"blacklisted_layers": [
],
"component_layers": [
"VK_LAYER_KHRONOS_validation"
],
"description": "LunarG Override Layer",
"disable_environment": {
"DISABLE_VK_LAYER_LUNARG_override": "1"
},
"implementation_version": "1",
"name": "bad_cast",
"override_paths": [
"/home/jeremyg/src/Vulkan-ValidationLayers/build/layers"
],
"type": "GLOBAL"
}
}
~/.local/share/vulkan/settings.d/vk_layer_settings.txt:
khronos_validation.fine_grained_locking = true
khronos_validation.validate_core = true
khronos_validation.check_image_layout = true
khronos_validation.check_command_buffer = true
khronos_validation.check_object_in_use = true
khronos_validation.check_query = true
khronos_validation.check_shaders = true
khronos_validation.check_shaders_caching = true
khronos_validation.debug_disable_spirv_val = false
khronos_validation.unique_handles = true
khronos_validation.object_lifetime = true
khronos_validation.stateless_param = true
khronos_validation.thread_safety = true
khronos_validation.validate_sync = false
khronos_validation.validate_gpu_based = GPU_BASED_NONE
khronos_validation.gpuav_descriptor_checks = true
khronos_validation.gpuav_warn_on_robust_oob = true
khronos_validation.gpuav_buffer_address_oob = true
khronos_validation.gpuav_max_buffer_device_addresses = 10000
khronos_validation.gpuav_validate_ray_query = true
khronos_validation.gpuav_cache_instrumented_shaders = true
khronos_validation.gpuav_select_instrumented_shaders = false
khronos_validation.gpuav_indirect_draws_buffers = true
khronos_validation.gpuav_indirect_dispatches_buffers = true
khronos_validation.gpuav_indirect_trace_rays_buffers = true
khronos_validation.gpuav_buffer_copies = true
khronos_validation.validate_best_practices = false
khronos_validation.debug_action = VK_DBG_LAYER_ACTION_LOG_MSG
khronos_validation.log_filename = stdout
khronos_validation.report_flags = error
khronos_validation.enable_message_limit = true
khronos_validation.duplicate_message_limit = 10
khronos_validation.message_id_filter =
khronos_validation.message_format_display_application_name = false
I think there's 2 key parts:
- In the .json file, override_paths needs to point to the directory containing the locally built .so filie.
- vk_layer_settings.txt is required to get into a code path that uses std::regex, when just using the env vars we get empty strings querying for those settings and skip the code that parses them using regex. So this explains why I couldn't repro with env vars before.
having the layer statically link against libstdc++ seems to avoid the problem
I created a simplified version of this situation to test various ideas against, and I found that in that project, I had to link the mockup Vulkan layer with -Wl,-Bsymbolic as well. (But perhaps your layers already do that.)
vk_layer_settings.txt is required to get into a code path that uses std::regex
Aha, that's useful information. I couldn't reproduce this today, but I'll try again tomorrow armed with that additional knowledge.
On Ubuntu 24.04 + Nvidia + proprietary driver, I can reproduce a crash. Steps to reproduce:
- Ubuntu 24.04, fully updated as of 2024-09-26
- Nvidia GTX 1080 with proprietary driver 535.183.01
-
sudo apt install vulkan-validationlayersinstalls versionvulkan-validationlayers:amd64 1.3.275.0-1 - my meta-layer from https://github.com/ValveSoftware/steam-runtime/issues/694#issuecomment-2377312510
- pointing to the system-wide validation layer from Ubuntu
- copy-pasting your
~/.local/share/vulkan/settings.d/vk_layer_settings.txtfrom https://github.com/ValveSoftware/steam-runtime/issues/694#issuecomment-2377462453 (an empty file was not sufficient, I haven't attempted to minimize the file contents beyond that)
Unfortunately coredumpctl gdb doesn't tell me anything useful beyond the backtrace being in engine code:
#0 0x000070079363554d in ?? ()
from /home/desktop/SteamLibrary/steamapps/common/Counter-Strike Global Offensive/game/bin/linuxsteamrt64/libtier0.so
#1 0x2f2f2f2f2f2f2f2f in ?? ()
#2 0x00000000ffffffff in ?? ()
#3 0x000070079520f088 in ?? ()
#4 0x000070079566c600 in ?? ()
#5 0x00000000e1066a24 in ?? ()
#6 0x00005d2c20320f58 in ?? ()
#7 0x0000700791a067cc in ?? ()
from /home/desktop/SteamLibrary/steamapps/common/Counter-Strike Global Offensive/game/bin/linuxsteamrt64/libengine2.so
#8 0x0000700793633e56 in je_malloc ()
from /home/desktop/SteamLibrary/steamapps/common/Counter-Strike Global Offensive/game/bin/linuxsteamrt64/libtier0.so
#9 0x000070076a0802fe in ?? ()
#10 0x00007007930bc510 in ?? ()
#11 0x00007007930bc518 in ?? ()
#12 0x00007ffcd9d377f0 in ?? ()
#13 0x0000000000000000 in ?? ()
Crash dumps:
Hopefully this is enough for an engine developer to at least confirm whether I reproduced the same crash? And maybe those steps help an engine developer to reproduce this.
On Arch + AMD + Mesa, I couldn't reproduce this by the same method (this time the validation layer was from vulkan-validation-layers 1.3.290-2. However, I do get a different failure mode: CS2 hangs during startup, with a black screen, using 100% of a CPU core. I don't know whether this is evidence of a different (related?) problem, or whether it's just the validation layer making CS2 startup really, really slow.
In the .json file, override_paths needs to point to the directory containing the locally built .so [file]
If I'm reading the specification correctly, override_paths needs to point to the directory containing the locally-built layer's JSON manifest. In your locally-built layer, maybe that is the directory containing the .so file, but only by coincidence.
The fundamental problem here is that there are two copies of
std::regexin action here: the one in the host system's libstdc++ as linked by the Vulkan layer, and the one in the libstdc++ that the game engine links statically.
On reflection, I don't think this is actually a Steam Runtime problem, more like a game-engine problem. If we were running the game without using the SLR container, its statically-linked copy of (part of) our SDK's compiler's libstdc++ would be equally able to conflict with the host system's libstdc++, unless they happened to be the same version (which is obviously not possible in general, because the environment used to compile the game can't possibly have the same libstdc++ version as both Arch and Ubuntu at the same time).
So I think this issue report should perhaps move to https://github.com/ValveSoftware/csgo-osx-linux, or be closed as "not planned" for the container runtime - it's only CS2 developers who will be in a position to be able to address this.
It would be better if we can avoid the Source2 engine triggering this problem from the engine end, but that's difficult to achieve (I have a couple of ideas which might potentially help there).
I made a simplified mockup of this situation, and while I couldn't figure out how to reproduce a similar crash, I was able to test out my ideas by inspecting the symbol tables. Based on that, I've made some suggestions internally (reference for maintainers: steamrt/tasks#547) which might provide ways to resolve this.
Until then, you have some workarounds (linking your Vulkan layer's libstdc++ statically as well, or loading it via VK_INSTANCE_LAYERS instead of a meta-layer).
I'd rather keep it here since it's focused on the general problem of distributing binaries on Linux that steam-runtime aims to fix, and generally not a problem restricted to CS2.
I'd rather keep it here since it's focused on the general problem of distributing binaries on Linux that steam-runtime aims to fix, and generally not a problem restricted to CS2.
Yeah, reasonable. Perhaps when we've figured out how to avoid this problem in CS2, we can add a note to https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/blob/main/docs/slr-for-game-developers.md?ref_type=heads#making-a-game-container-friendly (or some more general document about building generally-usable Linux binaries) with recommendations for good patterns to use and anti-patterns to avoid?