Access both voxels and mesh surfaces from a custom shader

[jjhembd]

Some voxel use cases would benefit from being able to read a the depth of a surface inside the voxel shader. For example:

• Computing depth maps, showing the distance from the topography to the first voxel with a given metadata value
• Accumulating voxel properties on one side of a surface

Note that czm_globeDepthTexture is already used during voxel rendering (see IntersectDepth.glsl).

Possibly related:

• https://github.com/CesiumGS/cesium/issues/12296
• https://github.com/CesiumGS/cesium/issues/11019

[jjhembd]

The czm_globeDepthTexture already includes both the terrain and 3D Tiles surfaces.

While a user could sample this texture in the CustomShader, this would be clumsy and slow. The customShader supplied to a VoxelPrimitive is called at every step in the raymarching. Adding potentially dozens or hundreds of texture reads to each pixel would be a significant performance hit. It would be far better if the depth value were read once and stored.

We currently sample czm_globeDepthTexture in intersectScene at the beginning of the VoxelFS.glsl workflow, but do not save the value. To make this available to the custom shader, we will need to:

1. Refactor intersectDepth.glsl to return a value, so that we can write it even if the value is not being used in the Intersections.intersections array (i.e., if VoxelPrimitive.depthTest === false).
2. Store the depth value as a separate property on the Intersections struct in intersectScene, and also write it to the .intersections array if needed
3. In VoxelFS, write the depth value to the fragmentInput.voxel struct, to make it available to the Custom Shader.

The fragmentInput.voxel struct should also be updated to include the accumulated ray marching distance. While the user could accumulate the per-step values, this method would accumulate precision errors.

[jjhembd]

As a first step to generate an end-to-end test, I plan to write a custom shader using a brute-force method:

1. Set VoxelPrimitive.depthTest === false to allow raymarching beyond the topography
2. Accumulate total raymarching travel distance
3. Read czm_globeDepthTexture to get the distance to the surface
4. Use a metadata criteria to identify a target layer, and write out the distance between the topography and the target layer

This method is problematic for the reasons outlined in my previous comment, but it will be a good first check of the workflow.

Note that most real-world analytics will want to consider the distance along some axis other than the camera ray: for example, along the z-axis, or the surface normal. This will need to be addressed once we have the basic thickness calculation working.

[ggetz]

Thanks @jjhembd!

• Do you have any thoughts on what using czm_globeDepthTexture would mean for precision and accuracy?
• For axes other than directly along the current camera ray, is it worth considering sampling the terrain or 3D Tiles heights directly, either on the CPU or in a separate pass?

@lilleyse Do you have any thoughts on approach?

[jjhembd]

@ggetz,

• My initial guess is that czm_globeDepthTexture will have adequate precision/accuracy at relevant camera positions. It loses precision in the distance, but most users will be zoomed in to their area of interest if they are needing accuracy. The alternative would be to load a separate texture with an actual depth value (rather than distance from the camera). But this would require a render pass, since the starting point for the surface is usually a mesh.
• For axes other than the camera ray, I'm assuming we need the mesh height at every pixel, since the Custom Shader runs for every fragment. So CPU would not be realistic, but a separate render pass might make sense--i.e., an off-screen render of an orthographic camera facing straight down. Alternatively, if users only want to analyze at the cursor position, we could run a picking-type pass, in which case the depth could be computed on the CPU and passed as a uniform.

[jjhembd]

Here is a preliminary Sandcastle using the depth texture to compute the thickness of the voxel above the surface. The voxel in this case is a simple box. The surface is the default EllipsoidTerrainProvider.

Some observations:

• The way we accumulate color in the main voxel shader is somewhat limiting for analytics. It is an opacity-weighted sum, which is approximately correct for visualizing translucent voxels, but makes it very hard to recover a linearly accumulated value.
• Measuring distance from a voxel to a deeper surface is easy. Measuring from a surface to a deeper voxel will be harder.
• Analyzing anything other than distances will require an adjustment to the way we accumulate.

We may want to consider adding a flag to VoxelPrimitive to switch between linear and opacity-weighted accumulation.

[jjhembd]

Quick update: I refactored IntersectDepth to store the raw depth buffer value separately, and wrote this to the FragmentInput struct to make it accessible in the Custom Shader. Results are identical for now.

Next step is to try using this value to compute more useful analytics in the shader.

[ggetz]

Looks like this was resolved by https://github.com/CesiumGS/cesium/pull/12636. I'm going to go ahead and close, but @jjhembd please feel free to reopen if I missed anything.