Vulkan-ValidationLayers icon indicating copy to clipboard operation
Vulkan-ValidationLayers copied to clipboard

Crash/Hang when using GPU-assisted validation

Open Ravbug opened this issue 4 months ago • 3 comments

Environment:

  • OS: Windows 10 22H2 19045.6216
  • GPU and driver version: RTX 2070 Super, driver 577.0
  • SDK or header version if building from repo: 1.4.321.1
  • Options enabled (synchronization, best practices, etc.): GPU-Assisted preset with no modifications

Describe the Issue When running my application under standard validation or synchronization validation, it behaves normally with no errors raised. When I run with GPU-assisted validation enabled I get a segmentation fault inside of vkCreateGraphicsPipelines with no errors logged from the layer. Interestingly, when I enable the ray tracing extensions, this crash turns into an infinite hang. The additional extensions were: VK_KHR_ACCELERATION_STRUCTURE_EXTENSION_NAME, VK_KHR_RAY_TRACING_PIPELINE_EXTENSION_NAME, VK_KHR_RAY_QUERY_EXTENSION_NAME, VK_KHR_DEFERRED_HOST_OPERATIONS_EXTENSION_NAME, VK_KHR_DEFERRED_HOST_OPERATIONS_EXTENSION_NAME, VK_KHR_SPIRV_1_4_EXTENSION_NAME, VK_KHR_SHADER_FLOAT_CONTROLS_EXTENSION_NAME, VK_KHR_SHADER_FLOAT_CONTROLS_2_EXTENSION_NAME

SPIRVs for the vertex and fragment shaders in the pipeline: pipeline-spirvs.zip

Expected behavior

The validation layer should either proceed normally or report an error.

Valid Usage ID If applicable, please include the validation messages encountered leading up to the issue

Messages (hang)

[Vulkan Loader] ERROR:          loader_get_json: Failed to open JSON file C:\Program Files\Rockstar Games\Social Club\SocialClubVulkanLayer.json
[Vulkan Loader] ERROR:          loader_get_json: Failed to open JSON file C:\Program Files\Rockstar Games\Social Club\SocialClubVulkanLayer.json
[Vulkan Loader] ERROR:          loader_get_json: Failed to open JSON file C:\Program Files\Rockstar Games\Social Club\SocialClubVulkanLayer.json
[2025-09-02 00:45:32.6349829] WARN - loader_create_device_chain: Using deprecated and ignored 'ppEnabledLayerNames' member of 'VkDeviceCreateInfo' when creating a Vulkan device.
[2025-09-02 00:45:32.7954873] WARN - vkCreateDevice(): Internal Warning: Ray Query validation option was enabled, but the rayQuery feature was not supported [Disabling gpuav_validate_ray_query]

Additional context

code or terminal output

Call stack (hang):

 	nvoglv64.dll!00007ffcd010ec76()	Unknown
 	nvoglv64.dll!00007ffcd00ff1aa()	Unknown
 	nvoglv64.dll!00007ffcd01cef8f()	Unknown
 	nvoglv64.dll!00007ffcd0158b26()	Unknown
 	nvoglv64.dll!00007ffccefc1755()	Unknown
 	nvoglv64.dll!00007ffccfe575cb()	Unknown
 	nvoglv64.dll!00007ffccfe557ca()	Unknown
 	nvoglv64.dll!00007ffccfe5b570()	Unknown
 	nvoglv64.dll!00007ffccfe5abab()	Unknown
 	nvoglv64.dll!00007ffccfe59b1d()	Unknown
 	nvoglv64.dll!00007ffccfe5f687()	Unknown
 	nvoglv64.dll!00007ffccfe41eaa()	Unknown
 	nvoglv64.dll!00007ffccfd219af()	Unknown
 	VkLayer_khronos_validation.dll!vvl::dispatch::Device::CreateGraphicsPipelines(VkDevice_T * device, VkPipelineCache_T * pipelineCache, unsigned int createInfoCount, const VkGraphicsPipelineCreateInfo * pCreateInfos, const VkAllocationCallbacks * pAllocator, VkPipeline_T * * pPipelines) Line 959	C++
 	VkLayer_khronos_validation.dll!vulkan_layer_chassis::CreateGraphicsPipelines(VkDevice_T * device, VkPipelineCache_T * pipelineCache, unsigned int createInfoCount, const VkGraphicsPipelineCreateInfo * pCreateInfos, const VkAllocationCallbacks * pAllocator, VkPipeline_T * * pPipelines) Line 525	C++
 	vulkan-1.dll!00007ffcd1824627()	Unknown

Ravbug avatar Sep 02 '25 01:09 Ravbug

@Ravbug do you know if this is using VK_EXT_graphics_pipeline_library or anything?

What would be amazing is if you could capture a fossilize of this, basically using https://github.com/ValveSoftware/Fossilize/ you can go

set FOSSILIZE=1
set FOSSILIZE_DUMP_PATH=C:/Users/Ravbug/somewhere
set VK_INSTANCE_LAYERS=VK_LAYER_fossilize
set VK_LAYER_PATH=C:/path/to/Fossilize/build

and it will dump a .foz file that will have everything encoded, and would be the fastest way to reproduce (exactly) these larger pipelines

(Note - the goal is to get it without GPU-AV on so I can run it locally with GPU-AV to see what it generates)

spencer-lunarg avatar Sep 02 '25 22:09 spencer-lunarg

@spencer-lunarg Here is the fossilize result: fossilize.313a69080fbb2641.1.zip I enabled fossilize via Vulkan Configurator because I couldn't get your command line instructions to work. Hope that didn't mess with anything.

I do use Pipeline Library and Graphics Pipeline Library.

Ravbug avatar Sep 03 '25 03:09 Ravbug

@Ravbug Thanks so MUCH!! Will make solving this 100x easier with a fossilize file if you are using Graphics Pipeline Library because we actually don't instrument until the final linking (otherwise startup time is AWFUL) and has caused subtle issues that luckily Fossilize can capture

spencer-lunarg avatar Sep 03 '25 04:09 spencer-lunarg