KhronosGroup
Check binary architecture before report Layer
**What enhancement are you suggesting for the Vulkan Loader? Please describe in detail. **
Loader should check the binary validity against running architecture, i.e. layer with x86 library cannot be reported to x64 process and vice-sersa: x64 library should not be reported to x86 process.
Is this specific to a single platform? Windows 10 x64
Additional context
After installing some application we've got a Vulkan CTS error like:
Test case 'dEQP-VK.info.instance_layers'..
Fail (Duplicate layer: VK_LAYER_1)
This actually happens due to the application registered the VK_LAYER_1 twice in the registry first is for x86 binary and second is for x64 binary. This actually I think may cause additional troubles when application will try to enable the layer: it is unclear which binary load attempt will be made. I think Loader should remove invalid for process layers, i.e. layer with x86 library cannot be reported to x64 process and vice-sersa: x64 library should not be reported to x86 process.
I know that this is a mistake of app that it put both binaries into registry key Computer\HKEY_CURRENT_USER\Software\Khronos\Vulkan\ImplicitLayers, but I think it will improve Loader robustness making this basic validation.
The "library_arch" field in layer & driver manifests is the main mechanism to prevent this issue. It isn't 100% effective since layers/drivers must add it themselves, as its an optional field. (And its too late to change the required fields list). A 100% effective solution unfortunately is not a priority.
The difficulty lies in not being able to locate layer/driver binaries from system default locations (ie, when "library_path" contains just the name, like "foo.dll" instead of a relative or absolute path). Once the full path is known, it is relatively easy to load the first few bytes of a binary and check the "is 32/64" field that exectuable formats have. I know PE & ELF both have them, and those are the primary platforms which need it. Unfortunately, system default paths rely on LoadLibrary/dlopen to find the binary, so the solution either means re-implementing the default library searching logic, or calling dlopen() to see if it returns the wrong-arch error (this incurring a major perf penalty by loading/unloading ALL layers.) If you have any alternative solutions, I am all ears.
There is the possibility of only checking absolute & relative paths, but a partial solution is annoying since it only works partially. That said, making it harder for wrong architecture layers to load isn't bad, so I'm definitely not opposed to more robust checking being put in place when it can be.
I will say wrong architecture isn't the only type of pain caused by bad layers. Because the loader doesn't want to incur the penalty of loading all layers all the time, missing dynamic libraries go undetected until vkCreateInstance time, allowing for applications to see layers that will never actually load as 'valid'. Put another way, the loader cannot validate with 100% certainty during vkEnumerateInstanceLayerProperties whether a layer will behave correctly and fixing that is playing whack a mole (and balancing performance costs).
Thank you describing the layer verification problem in more details. I understand trade-off between validity and performance and propose a narrow-down proposal a bit. I was concerned with the fact that application get presented with two layers of the same name. As far as I understand application chooses layer to load exclusively by name, thus making second-in-list layer effectively non-loadable (even if it is valid). So may be I should change the title to "Do not expose layer with same name twice". Then at enumeration time we will need to check only two conflicting layers (layers with same name). For this issue I would go mimicking path search in LoadLibrary/dlopen and checking few bytes instead of full library load as I agree it is too expensive.