WIP: Offline Shader Compilation

[tomadamatkinson]

Disclaimer: PoC does not yet work. Creating a PR for working group visibility

. . . . . . . . . . . . . . .

Description

Please include a summary of the change, new sample or fixed issue. Please also include relevant motivation and context. Please read the contribution guidelines

Fixes #

General Checklist:

Please ensure the following points are checked:

• [ ] My code follows the coding style
• [ ] I have reviewed file licenses
• [ ] I have commented any added functions (in line with Doxygen)
• [ ] I have commented any code that could be hard to understand
• [ ] My changes do not add any new compiler warnings
• [ ] My changes do not add any new validation layer errors or warnings
• [ ] I have used existing framework/helper functions where possible
• [ ] My changes do not add any regressions
• [ ] I have tested every sample to ensure everything runs correctly
• [ ] This PR describes the scope and expected impact of the changes I am making

Note: The Samples CI runs a number of checks including:

• [ ] I have updated the header Copyright to reflect the current year (CI build will fail if Copyright is out of date)
• [ ] My changes build on Windows, Linux, macOS and Android. Otherwise I have documented any exceptions

Sample Checklist

If your PR contains a new or modified sample, these further checks must be carried out in addition to the General Checklist:

• [ ] I have tested the sample on at least one compliant Vulkan implementation
• [ ] If the sample is vendor-specific, I have tagged it appropriately
• [ ] I have stated on what implementation the sample has been tested so that others can test on different implementations and platforms
• [ ] Any dependent assets have been merged and published in downstream modules
• [ ] For new samples, I have added a paragraph with a summary to the appropriate chapter in the samples readme
• [ ] For new samples, I have added a tutorial README.md file to guide users through what they need to know to implement code using this feature. For example, see conditional_rendering
• [ ] For new samples, I have added a link to the Antora navigation so that the sample will be listed at the Vulkan documentation site

[jherico]

I'm curious why you took the path of switching all the existing GLSL headers to have the extension glsl. You could either assume that everything not ending in hlsl was instead GLSL, or you could inspect the file contents for a version directive. That would remove from the changed files most of the files that are just being renamed and have their copyright touched.

Also, you might want to consider abseil as a replacement for separate hash/string manipulation/logging libraries. You could also use it to replace fmt as well, but I personally like fmt more.

[tomadamatkinson]

Thanks @jherico for taking a look.

In terms of hashing and abseil. I would prefer to keep the single header approach for now. No need to import an entirely new library for the sake of a sha256 algorithm.

My goal with this hash approach was to identify binary equivalent files to reduce the amount of compiled shaders that would need to be shipped to a device or stored in headers. The json indexes help to keep track of this in a more human readable way - end of the day we are trying to show people what is happening under the hood

In terms of glsl. I added the type of the shader in the file path as in the future we are wanting to create hlsl alternatives to your glsl shaders. Having the type would allow us to store hlsl/glsl in the same folder with less confusion

I am open to improvements here though. Maybe my renaming was preemptive and that specific change could come in a later PR if needed

[jherico]

I'm confused as to why this particular build is generating some errors on the CI runs when the files themselves haven't changed.

[SaschaWillems]

CI errors are due to some files having been renamed but not all files referring to them have been updated yet. But it's a draft anyway.

As the description suggest, Tom made this as base for discussing possible ways of doing this.

[jherico]

~Sorry, I should habe been more specific. I saw an error where the compiler.complained about RAND_MAX cast to float losing precision.~

Never mind, I think I get it.

[jherico]

Playing around with locally building this and diving into the shader preprocessing a bit more this morning. One thing I'm noticing is the heavy use of add_custom_target. Anything made by add_custom_target is considered always out of date.

I previously had a similar solution in my project where every single shader was a dependency of a single massive shaders target, but that had the annoying by-product of causing a rebuild of every single shader if I wanted to tweak one and re-run a given sample.

Instead I would recommend replacing the add_custom_target items with add_custom_command where you're able to specify the both the inputs and the outputs of the command, so the output are only regenerated if they're actually out of date compared to the inputs. This means you don't have a top-level target for each shader file, so what you end up doing is when you get to a target that actually requires a shader (like the say the add_sample lists the required shaders) you

• Add the shader source to the specific target
• Also add the generated files to the target, which forces them to be built if they're out of date, when you build the project.

You might still need a separate step to build something like an atlas of all the shaders, but it would make it much more convenient for someone working on a given sample to edit the shader used for that sample without triggering the burden of rebuilding every single shader.

If you're interested I can create a branch off of your code and tweak it to do this. This wouldn't prevent the framework or components from declaring shader dependencies, and specific shaders could still easily be referenced by more than one target in the project.

[tomadamatkinson]

This PR is no longer needed. The complexity of this PR is to support our implementation of variants. The working group has opted to reduce our variant usage allowing this PR to be closed