Allow customization of shadow-pass vertex shader

[robojeb]

What problem does this solve or what need does it fill?

Currently, if a custom material (impl Material) provides a vertex_shader(...) implementation which performs any transformations of the vertices other than directly projecting them into world and then projection space (eg applying a displacement map or performing procedural vertex displacement), the shadows on that mesh will be incorrectly calculated using the un-displaced vertex positions.

I encountered this issue when trying to create a height-map plugin.

What solution would you like?

I think there is a lot of room for discussion about the best way to allow custom vertex shaders to integrate with the existing shadow_pass. One solution which may have less impact on the engine (but foists responsibility onto the user) is to add a:

fn shadow_vertex_shader(...) -> Option<Handle<Shader>>

function on the bevy::pbr::Material trait.

What alternative(s) have you considered?

Alternatively you could always require a shadow vertex shader be provided and change the signature of fn vertex_shader(...) to:

fn vertex_shader(...) -> Option<(Handle<Shader>, Handle<Shader>)>

(or more likely some struct like CustomVertexShaders which has useful names for each shader)

One other thing that crossed my mind as being useful (and working towards a solution) would be to allow separation of a "VertexMaterial" and "FragmentMaterial" component. Doing it this way would allow easy composition so you could have an entity with ProceduralPlanetVertexMaterial or HeightmapVertexMaterial rendered with either PBRMaterial or ToonMaterial (for example) . The complications here being the coordination between what slots are expected to be filled by the vertex shader.

Additional context

I discussed this on the discord rendering channel and @superdump mentioned that they have been meaning to look into allowing easier re-use of the shadow pass and PBR infrastructure in general.

[pkupper]

The same problem also applies to the Wireframe Plugin which uses the default vertex shader as well. Ideally the solution here wouldn't be restricted to the shadow pass.

[trevex]

I am wondering if the following might be an alternative approach to consider:

Rather than extending Material for a specific rendering technique. Materials are associated with a particular rendering pass (or render graph node) and a given custom material would, for example, implement Material<ForwardPass>, Material<ShadowPass> to not only render to the framebuffer, but also to make sure the geometry is used to throw shadows. This would be more involved than the current approach to custom materials, but it should also be possible to expose a more convenient way for simpler use-cases.

This would keep the Material system flexible enough to also support the introduction of other rendering passes, e.g. depth pre-pass, or alternative rendering setups all together.

[JMS55]

The prepass ran into a similar issue for normals. The solution was to add extra (optional) methods to Material for getting prepass-specific vertex and fragment shaders.

[superdump]

This was solved by using the prepass shaders for shadow mapping and removing the hard-coded shadow mapping vertex shader.