WIP - IBL shadow material blending

[MiiBond]

I'm working on how to combine the generated shadow image with the actual material properties. Shadows should only apply to view-independent lighting and so don't work really well for things like specular reflections. Therefore, I want to apply them 100% to diffuse (indirect) lighting and scale their affect on specular (direct) lighting based on surface roughness. The more rough the surface, the less view-dependent the specular component becomes.

I'm struggling with the best way to do this in Babylon. Currently, I'm trying to use prepass buffers to write out this view-independent/dependent lighting and then I combine them with shadows in a post process. This is simple but has some issues:

1. I ended up hijacking fragData[0] for the indirect lighting component because, otherwise, this data is wasted and I end up using an additional buffer. Then, anything else (like UI) that was written to the fragData[0] is lost when I apply the shadows.
2. Some things, like emissive, might be technically view-independent but they shouldn't have shadows applied.

It might be better to apply the shadows in the material shader directly to handle all the material properties as needed. However, this implies rendering all the prepass buffers before the final material render. This means everything will be drawn an extra time. This is how I've implemented this effect before but in, WebGL, I worry about the performance implications of rendering all the geometry another time...

[bjsplat]

Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s). To prevent this PR from going to the changelog marked it with the "skip changelog" label.

[deltakosh]

Thanks a lot @MiiBond ! But before moving on with more changes I would need for the current version to work on WebGPU with tint so I can merge my webgpu shaders and thus upcoming new PR can update glsl and wgsl simultaneously

[bjsplat]

Snapshot stored with reference name: refs/pull/15544/merge

Test environment: https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15544/merge/index.html

To test a playground add it to the URL, for example:

https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15544/merge/index.html#WGZLGJ#4600

Links to test babylon tools with this snapshot:

https://playground.babylonjs.com/?snapshot=refs/pull/15544/merge https://sandbox.babylonjs.com/?snapshot=refs/pull/15544/merge https://gui.babylonjs.com/?snapshot=refs/pull/15544/merge https://nme.babylonjs.com/?snapshot=refs/pull/15544/merge

To test the snapshot in the playground with a playground ID add it after the snapshot query string:

https://playground.babylonjs.com/?snapshot=refs/pull/15544/merge#BCU1XR#0

[bjsplat]

Visualization tests for WebGPU (Experimental)

https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15544/merge/testResults/webgpuplaywright/index.html

[bjsplat]

WebGL2 visualization test reporter:

https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15544/merge/testResults/webgl2playwright/index.html

[Popov72]

1. I ended up hijacking fragData[0] for the indirect lighting component because, otherwise, this data is wasted and I end up using an additional buffer. Then, anything else (like UI) that was written to the fragData[0] is lost when I apply the shadows.

You should still use a specific index and not fragData[0]. This is because we're developing a render graph, which has a GeometryRenderer task that reuses existing prepass rendering shader code, and fragData[0] is not special in this context (it's no longer the final color), but one of the buffers you want to generate.

Another more immediate reason for fragData[0] to remain the final color is if additional post-processing is added, since in this case the effect of the post-processing is applied to this buffer.

It might be better to apply the shadows in the material shader directly to handle all the material properties as needed. However, this implies rendering all the prepass buffers before the final material render. This means everything will be drawn an extra time. This is how I've implemented this effect before but in, WebGL, I worry about the performance implications of rendering all the geometry another time...

You'll be able to do this when the new render graph is officially supported in Babylon.js, but in the meantime I think you should keep the existing code, as an additional geometry pass would be too heavy!

[MiiBond]

The issue that I'm having is that this isn't really a simple post-process effect. It doesn't really have use for the textureSampler input. The shadows should composite as: indirectLighting * shadows + directLighting * shadows_modified_based_on_roughness It doesn't involve the previously rendered, fully lit material. Can we enforce that the iblShadowsRenderPipeline is always the first pipeline? Even then, the UI won't be rendered into the prepass buffers and will be lost... Maybe some new flags on the prepass renderer could control some of these things?

[Popov72]

It's a limitation of the prepass renderer, the geometry textures can only be used in a post-process.

You can use the geometry buffer renderer instead. This will render the scene once again, but with a simplified shader (no final color calculation). This corresponds better to what will be available in the frame graph, where a geometry renderer task will render geometry textures, which you can use in shaders as input textures.

In your case, I can't see how this can be done without two passes. You need to generate the geometry textures first, then use them when rendering the meshes. You'll need to write a material plugin to implement the “indirectLighting * shadows + directLighting * shadows_modified_based_on_roughness” calculation part in a PBR material.

This is pretty much what we do to implement RSM GI. We use the geometry buffer renderer to generate the geometry textures we need, then we use a material plugin which can merge the GI contribution when rendering a mesh with a standard or PBR material.

As far as the pre-pass renderer is concerned, we don't want to add any new features, since it won't be used anymore once we have full frame graph support in the engine.

[github-actions[bot]]

This pull request has been marked as stale because it has been inactive for more than 14 days. Please update to "unstale".

[Popov72]

Should this PR be closed?

[MiiBond]

This is replaced by https://github.com/BabylonJS/Babylon.js/pull/15634