IBL shadows use geometry buffer

[MiiBond]

I converted the IBL shadows to use the geometry buffer so that I can generate the shadows before the main render pass and then apply them in the materials with a material plugin. This allows me to apply the shadows only to diffuse reflections and rough specular and not to things like emission.

Some current issues:

1. We always need to add the plugin to the material before it's used for rendering. However, in WebGPU, I get errors while trying to add the plugin to material in the callback of ImportMesh. The plugin "IBLShadowsPluginMaterial" can't be added to the material "BoomBox_Mat" because this material has already been used for rendering! #8R5SSE#293

[bjsplat]

Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s). To prevent this PR from going to the changelog marked it with the "skip changelog" label.

[bjsplat]

Snapshot stored with reference name: refs/pull/15634/merge

Test environment: https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15634/merge/index.html

To test a playground add it to the URL, for example:

https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15634/merge/index.html#WGZLGJ#4600

Links to test babylon tools with this snapshot:

https://playground.babylonjs.com/?snapshot=refs/pull/15634/merge https://sandbox.babylonjs.com/?snapshot=refs/pull/15634/merge https://gui.babylonjs.com/?snapshot=refs/pull/15634/merge https://nme.babylonjs.com/?snapshot=refs/pull/15634/merge

To test the snapshot in the playground with a playground ID add it after the snapshot query string:

https://playground.babylonjs.com/?snapshot=refs/pull/15634/merge#BCU1XR#0

[bjsplat]

Visualization tests for WebGPU (Experimental)

https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15634/merge/testResults/webgpuplaywright/index.html

[bjsplat]

WebGL2 visualization test reporter:

https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15634/merge/testResults/webgl2playwright/index.html

[Popov72]

My material plugin currently relies on gl_FragCoord.xy to sample the shadow. This doesn't work if the target doesn't match the resolution of the shadow texture. This is already noticeable with refractive transparency. The TransmissionHelper renders the opaque scene at a lower resolution and the shadow texture is applied incorrectly. I don't see anywhere in the PBR material where a value is passed in for the size of the RT. Any suggestions?

I think you can pass the size to your plugin as uniform values?

We always need to add the plugin to the material before it's used for rendering. However, in WebGPU, I get errors while trying to add the plugin to material in the callback of ImportMesh. The plugin "IBLShadowsPluginMaterial" can't be added to the material "BoomBox_Mat" because this material has already been used for rendering! #8R5SSE#293

I'll take a look at it when I review the PR, but it's strange that you only have this problem with WebGPU...

I'd like to support ShadowOnlyMaterial with the material plugin but that's defined in the materials package so I'm not sure how to go about that.

Material plugins are only supported for standard and PBR materials, unfortunately.

[bjsplat]

WebGL2 visualization test reporter:

https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15634/merge/testResults/webgl2playwright/index.html

[MiiBond]

My last issue is a WebGPU one.

BJS - [16:52:26]: WebGPU uncaptured error (1): [object GPUValidationError] - Bind group layout [BindGroupLayout (unlabeled)] of pipeline layout [PipelineLayout (unlabeled)] does not match layout [BindGroupLayout (unlabeled)] of bind group [BindGroup (unlabeled)] set at group index 1.
 - While encoding [RenderPassEncoder "mrt_z_axis_3RenderPass"].DrawIndexed(13872, 1, 0, 0, 0).
 - While finishing [CommandEncoder "render"].

It occurs only when reloading the playground and not when clicking the Run button. Here's an example playground: #8R5SSE#299 Sometimes the error refers to mrt_z_axis_3RenderPass and sometimes mrt_z_axis_4RenderPass. These are the MRT passes that voxelize the scene along the Z-axis before the voxels are combined and the mips are generated. I can get rid of the error if I render only one of these MRT's per frame, staggering the voxelization of the scene. This is where render is called for each MRT: https://github.com/BabylonJS/Babylon.js/blob/325b7318cb59cc47fb2c2b942378d129687430d9/packages/dev/core/src/Rendering/IBLShadows/iblShadowsVoxelRenderer.ts#L625 Any idea what might be going on?

[Popov72]

We always need to add the plugin to the material before it's used for rendering. However, in WebGPU, I get errors while trying to add the plugin to material in the callback of ImportMesh. The plugin "IBLShadowsPluginMaterial" can't be added to the material "BoomBox_Mat" because this material has already been used for rendering! #8R5SSE#293

Regarding this one, the problem is that the boom box has already been partially built (the ubo has been created) before the plugin is added, which is not supported. As a temporary workaround in your test PG, you can do this before adding the plugin:

            mat._uniformBufferLayoutBuilt = false;
            mat._uniformBuffer.dispose();
            mat._uniformBuffer = new BABYLON.UniformBuffer(mat._scene.getEngine(), undefined, undefined, mat.name);
            mat.resetDrawCache();

PG: http://localhost:1338/#8R5SSE#305

We'll see if we can find a better solution.

I'll look into the WebGPU issue as soon as possible.

[Popov72]

Regarding the "bind group" problem, I don't have it on my side, but I have the fix above applied. It seems this fix does not work correctly for the sky material, so it should be skipped. This PG does work for me in WebGPU:

http://localhost:1338/#8R5SSE#309

[Popov72]

We always need to add the plugin to the material before it's used for rendering. However, in WebGPU, I get errors while trying to add the plugin to material in the callback of ImportMesh. The plugin "IBLShadowsPluginMaterial" can't be added to the material "BoomBox_Mat" because this material has already been used for rendering! #8R5SSE#293

Regarding this one, the problem is that the boom box has already been partially built (the ubo has been created) before the plugin is added, which is not supported. As a temporary workaround in your test PG, you can do this before adding the plugin:

            mat._uniformBufferLayoutBuilt = false;
            mat._uniformBuffer.dispose();
            mat._uniformBuffer = new BABYLON.UniformBuffer(mat._scene.getEngine(), undefined, undefined, mat.name);
            mat.resetDrawCache();

PG: http://localhost:1338/#8R5SSE#305

We'll see if we can find a better solution.

I'll look into the WebGPU issue as soon as possible.

With this PR merged:

https://github.com/BabylonJS/Babylon.js/pull/15710

You don't need the code above anymore, so this PG will work:

http://localhost:1338/#8R5SSE#311

[MiiBond]

Regarding the "bind group" problem, I don't have it on my side, but I have the fix above applied. It seems this fix does not work correctly for the sky material, so it should be skipped. This PG does work for me in WebGPU:

Sorry, I disabled the triplanarVoxelization in the Playground that I linked and the error only happens while rendering the one set of MRT's for the Z-axis during triplanar voxelization. Here's an updated playground that should show the "bind group" issue. http://localhost:1338/#8R5SSE#313

[MiiBond]

I'd like to ask your opinion about usage API for this pipeline. Now that we have the material plugin, we need to add the plugin to any material that will receive shadows. I had initially thought to automate setup by detecting when meshes are added and removed from the scene and automatically set them up for casting and receiving shadows. However, I think the user will want more control over what is a shadow-caster and what is a shadow-receiver. I'm thinking of removing the addExludedMesh function and adding the following: addShadowCastingMesh(mesh) - include the mesh in the voxel grid. addShadowReceivingMaterial(material) - add the material plugin.

Additionally, my first thought was to automatically call updateSceneBounds and updateVoxelization when a shadow-casting mesh is added but I think it might be better to let the user call these when they want to. Voxelization is an expensive operation and we don't want to call it only to have the user need to call it again after they resize or position their mesh. What do you think?

[Popov72]

However, I think the user will want more control over what is a shadow-caster and what is a shadow-receiver. I'm thinking of removing the addExludedMesh function and adding the following: addShadowCastingMesh(mesh) - include the mesh in the voxel grid. addShadowReceivingMaterial(material) - add the material plugin.

I agree with this!

Additionally, my first thought was to automatically call updateSceneBounds and updateVoxelization when a shadow-casting mesh is added but I think it might be better to let the user call these when they want to. Voxelization is an expensive operation and we don't want to call it only to have the user need to call it again after they resize or position their mesh.

I also agree. When operations are heavy, I think it's better for the user to opt for this option manually rather than have it happen automatically. All that's needed is for the documentation to be clear about this.

[Popov72]

The bind group error is due to cache collisions. I'm going to create a PR to solve this problem.

Investigating the problem, I noticed that you create a voxelMaterial for each mesh: why not create a single material that you reuse for all meshes? The material you create doesn't depend on the mesh, so using a single material for all meshes should work.

PGs:

• The PG you linked above didn't work because of your latest changes, I fixed it: http://localhost:1338/#8R5SSE#328
• Minimal PG I used to repro the problem: http://localhost:1338/#8R5SSE#354

[EDIT] PR to fix the bind group problem:

https://github.com/BabylonJS/Babylon.js/pull/15722

[MiiBond]

The material you create doesn't depend on the mesh, so using a single material for all meshes should work

Right now, it uses a single material per MRT, not per mesh. The uniforms that define the slab are different for each MRT but perhaps I could bind to an event on the MRT to set the uniforms before each MRT renders (onBeforeRenderObservable?).

Thank you for looking at the bind group problem!

[MiiBond]

I changed the voxel renderer to only create one voxelization material.

[MiiBond]

@Popov72 I think this is ready to check. There are a lot of fixes in here.

[Popov72]

I will review the PR asap!

[bjsplat]

Visualization tests for WebGPU (Experimental)

https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15634/merge/testResults/webgpuplaywright/index.html

[Popov72]

I think you forgot to click on the "Load more..." link:

;)

[MiiBond]

I think you forgot to click on the "Load more..." link

I think you're right :)

[Popov72]

Only 3 or 4 minor comments and we are good to go!

Also, I think we can enable the visualization test for WebGPU ("IBL Voxel Shadowing")?

[Popov72]

We're ready to merge, unless you have more changes to come?

[MiiBond]

We're ready to merge, unless you have more changes to come?

Nope, I'm good to merge.