Trying to understand the rules for set incompatibility when immutable samplers are involved

[DarioSamo]

A recent change in our renderer backend to add immutable samplers has led to this seemingly unrelated error appearing during each frame.

VUID-vkCmdDrawIndexed-None-08600 vkCmdDrawIndexed(): VkPipeline 0x20be00000020be[RID:21891448307752] uses set 2 but that set is not bound. (Need to use a command like vkCmdBindDescriptorSets to bind the set).
VUID-vkCmdDrawIndexed-None-08600 vkCmdDrawIndexed(): VkPipeline 0x20be00000020be[RID:21891448307752] uses set 1 but that set is not bound. (Need to use a command like vkCmdBindDescriptorSets to bind the set).

• The pipeline in question uses three descriptor sets. The engine does not detect any need to rebind sets 1 and 2 unless the pipeline layouts are different, but seemingly it doesn't agree with the internal rules that the VVL has about it.
• It doesn't complain about set #0, nor is set #0 being bound again during the render pass. The engine always rebinds all sets above a specific set if it needs to.
• The immutable sampler in question is always a singular handle that's been created only once in the lifetime of the application.
• The immutable sampler is not the only descriptor in the set, but it is the only immutable sampler to appear in it.
• The immutable sampler is the first descriptor in the set.
• The immutable sampler is skipped from being updated with vkUpdateDescriptorSets, as the sampler is passed directly when creating the pipeline layout with VkDescriptorSetLayoutBinding.
  • One possible thing I suspected here was if the VkSampler * being different despite pointing to the same sampler could be the cause of a false positive, but fixing this to a static value in memory did not change the behavior.
• The problem goes away if sets 1 and 2 are forcefully bound before every draw.
• I've not determined which particular event triggers the sets being "unbound".
• All the pipelines in the render pass use pipelines use the same exact immutable sampler.
• I can confirm the error goes away as well if a dynamic sampler is used instead.
• It's worth clarifying I don't see any rendering errors caused by this. Two descriptor sets not being bound at all would definitely cause something pretty visible to happen, but it's not happening in any of the hardware I've tested (NVIDIA, AMD, Qualcomm, etc).

I've dug around the codebase of the validation layers for a while but haven't been able to turn much up. There's seemingly lots of rules that determine whether a pipeline layout is compatible or not, and the application seems to be able to get past these conditions only by introducing an immutable sampler into the mix. Without it, everything works as expected and there's no validation error claiming that the sets were not rebound as necessary.

Can I ask for a clarification on what are the exact rules of why a descriptor set would become "unbound" between pipeline changes or draws at the presence of an immutable sampler? Most of what I found seems to indicate that at least the numerical value of the VkSampler handle is involved in the hash computation.

I've not been able to build a minimal reproduction yet as I'm not entirely sure what series of events triggers the set being "unbound" so far. Suggestions on what to debug further are welcome.

[darksylinc]

(I'm helping Dario) To put in other words, what doesn't make sense is that if Set 0 in A and B are different, it makes sense that Set 1 and 2 must be rebound. But so should Set 0 be rebound.

However Validation Layers only complain about Sets 1 and 2, but not about set 0. Perhaps Set 0 from A and B are allowed to be "similar enough"?

This reeks of a false positive bug, but we're trying to gather more evidence before as we're not 100% sure. Hence the request for clarification.

[spencer-lunarg]

(edit - ignore this comment) So does the draw have a shader that "access" set 0 (or just declares it?) It is allowed to have a pipeline layout that has a set, but if you never "access it" in a shader, you don't need to bind it... that is one reason I see set 0 being skipped here

the check is not that crazy it is

// active == accessed in shader
for (const auto &[set_index, binding_req_map] : pipeline.active_slots) {
    // last bound == tracking what as been set in the command buffer
    const auto ds_slot = last_bound_state.ds_slots[set_index];
    // we have yet in the command buffer to see something like vkCmdBindDescriptorSets on this set
    if (!ds_slot.ds_state) {
        skip |= LogError(vuid.compatible_pipeline_08600, cb_state.GetObjectList(bind_point), vuid.loc(),
                            "%s uses set %" PRIu32
                            " but that set is not bound. (Need to use a command like vkCmdBindDescriptorSets to bind the set)",
                            FormatHandle(pipeline).c_str(), set_index);
    }

I guess can you confirm your vkCmdBindDescriptorSets is not mixing VK_PIPELINE_BIND_POINT_COMPUTE/VK_PIPELINE_BIND_POINT_GRAPHICS as I have seen that in the past

[spencer-lunarg]

Ok, I think what might be happening now is

so we are not detecting that set 0 is "the same" and that is causing us to "unbound" the set 1 or 2 likely... let me look more into the immutable sampler part, I know we have hit this issue in the past, but also thought we have tested it well and fixed it

[spencer-lunarg]

Can I ask for a clarification on what are the exact rules of why a descriptor set would become "unbound" between pipeline changes or draws at the presence of an immutable sampler?

I wish it was "simpler"... let be try to also formulate a proper answer and get back to you

[beicause]

Maybe https://github.com/godotengine/godot/issues/109566 is related, but it's slightly different:

• In certain case the error message is different:

ERROR: Vulkan Debug Report: object - 1839482953270921
vkCmdDrawIndexed(): The VkPipeline 0x6890000000689 (created with VkPipelineLayout 0x5e800000005e8) statically uses descriptor set 2, but all sets 0 to 2 are not compatible with the pipeline layout bound with vkCmdBindDescriptorSets (VkPipelineLayout 0x5ac00000005ac)
Set 0 binding 0 pImmutableSamplers 0xdab6030 doesn't match 0xde199e0
The Vulkan spec states: For each set n that is statically used by a bound shader, a descriptor set must have been bound to n at the same pipeline bind point, with a VkPipelineLayout that is compatible for set n, with the VkPipelineLayout used to create the current VkPipeline or the VkDescriptorSetLayout array used to create the current VkShaderEXT , as described in Pipeline Layout Compatibility (https://docs.vulkan.org/spec/latest/chapters/drawing.html#VUID-vkCmdDrawIndexed-None-08600)

(Note that set 0 binding 0-1 is actually unused, the immutable sample is set 0 binding 2)

• It does complain that set 0 (and 1, 2) is not bound.

[spencer-lunarg]

@beicause

Set 0 binding 0 pImmutableSamplers 0xdab6030 doesn't match 0xde199e0

ugh, so this seems like a VVL bug looking at it, seems like it was checking the pointers are the same and not the underyling VkSampler is the same... which is a known issue and need to get back to (hopefully this week if things go well) to try and fix/understand

[spencer-lunarg]

We just merged in https://github.com/KhronosGroup/Vulkan-ValidationLayers/pull/10582 and that should fix this, @DarioSamo or @darksylinc could you give it a try when possible (it will be in the Sep/October SDK for sure)