Material Pipeline System Phase 1

[santorac]

What does this PR do?

Added the first phase of implementation of the material pipieline system. See https://github.com/o3de/sig-graphics-audio/blob/main/rfcs/MaterialPipelineAbstraction.md for the full design.

At this stage you can author a material type with custom vertex manipulation, custom surface parameterization, and make it work with any render pipeline. It supports the MainPipeline and LowEndPipeline, as well as a proof-of-concept deferred pipeline that I have been working on here: https://github.com/aws-lumberyard-dev/o3de-atom-sampleviewer/tree/Atom/santorac/DeferredPipelinePOC

Details

• Added new .materialpipeline file
• The .materialtype file can now follow an "abstract" format where a shader list is not provided, and instead the material type provides custom material-specific shader code that will be combined with code from the material pipeline.
• Updated MaterialTypeBuilder to process these abstract .materialtype files by stitching them together with each material pipeline. It uses the new intermediate asset feature to generate intermediate .shader files and an intermediate .materialtype file that uses the original "direct" format.
• Updated the asset database version since the MaterialTypeBuilder has new job keys.
• There are shader templates for each pass.
• Multiple material pipelines can share common shader templates, or provide their own unique ones.
• There are a couple example material types to demonstrate the functionality and test the system.
• The original "direct" .materialtype format is still supported.
• The list of available material pipelines is configured through a .setreg file. Be default, only MainPipeline is enabled. In the AutomatedTesting project we override the list to also include LowEndPipeline for regression testing purposes, so any asset failures will block Jenkins.
• Improved comments AssetBuilderSDK.h

Reviewer Guide

The most important thing to understand is how the shaders get stitched together. Consider the case of running a basic material type on the MainPipeline. The final generated AZSL file looks like this:

materialpipelinetest_basic_mainpipeline_forwardpass.azsl

// This code was generated by MaterialTypeBuilder. Do not modify.
#include <D:\o3de\Gems\Atom\Feature\Common\Assets\Materials\Pipelines\MainPipeline\ForwardPass.azsli>
#include <D:\o3de\Gems\Atom\TestData\TestData\Materials\Types\MaterialPipelineTest_Basic.azsli>
 
#if !MaterialFunction_AdjustVertexData_DEFINED                                                    
    void MaterialFunction_AdjustVertexData(in float3 positionLS, inout VertexData vertexData) {}  
    #define MaterialFunction_AdjustVertexData_DEFINED 1                                           
#endif                                                                                            
                                                                                                  
#if !MaterialFunction_AdjustSurface_DEFINED && MATERIALPIPELINE_SHADER_HAS_PIXEL_STAGE            
    void MaterialFunction_AdjustSurface(inout Surface outSurface) {}                              
    #define MaterialFunction_AdjustSurface_DEFINED 1                                              
#endif                                                                                            

MaterialPipelineTest_Basic.materialtype provides some material shader code in MaterialPipelineTest_Basic.azsli. MainPipeline.materialpipeline provides an AZSLI file specific to the forward pass in MainPipeline\ForwardPass.azsli. The MaterialTypeBuilder stitches these two files together and also appends a bit of utility code.

1. With this in mind, go review MaterialPipelineCallbacks.azsli, MainPipeline\ForwardPass.azsli, and MaterialPipelineTest_Basic.azsli.
2. Next you'll likely be wondering how vertex modifications work, for supporting things like vertex animation, so go look at MaterialPIpelineTest_Animated.azsli.
3. MaterialTypeSourceData.h has a good explanation of the "abstract" and "direct" forms that .materialtype can have.
4. MaterialTypeBuilder.h shows how processing is broken into two phases.
5. MaterialTypeBuilder.cpp is where all the magic happens.
6. Go through the other files at your leisure.

How was this PR tested?

• Created and opened materials in the Material Editor.
• Rotated a model to make sure transforms were applied.
• Tested switching between pipelines in AtomSampleViewer's Mesh sample, including my prototype deferred pipeline.
• Run AtomSampleViewer full test suite. No new failures.
• Lots of testing of asset dependencies and reprocessing...
  • Changed the material_pipelines.setreg list, restarted the AP, and saw the changes get applied in the AP job list.
  • Removed and added a shader template in a .materialpipeline and saw the changes get applied in the AP job list.
  • Changed the .materialtype and saw the appropriate assets get reprocessed (intermediate .materialtype and .material that uses it).
  • Changed the material type's AZSLI file and saw the appropriate assets get reprocessed (intermediate .materialtype, .material that uses it, and each of the shaders).
  • Changed a pipeline-specific shader template file and saw the necessary assets get reprocessed (source material types, the corresponding intermediate .shader files). It also reprocessed the intermediate material types because those depend on the generated .shader files which changed, and the relevant .material files; this is expected.
  • Changed a pipeline-specific AZSLI template file and saw the intermediate .shader files get reprocessed. It also reprocessed the intermediate material types because those depend on the generated .shader files which changed, and the relevant .material files; this is expected. The source .materialfile was NOT reprocessed, which is good, it shouldn't need to be reprocessed, only intermediate assets are impacted here.
  • Changed a common shader template file and saw the necessary assets get reprocessed (source material types, the corresponding intermediate .shader files). It also reprocessed the intermediate material types similar to above.
  • Changed a common AZSLI template file and saw the intermediate .shader files get reprocessed. It also reprocessed the intermediate material types similar to above.
  • Changed ForwardPassVertexAndPixel.azsli which is only used in the main pipeline and saw the expected files get reprocessed. Only the main pipeline shaders were reprocessed, not the low end pipeline shaders. It also reprocessed the intermediate material types similar to above.