Gaussian Splat SPZ Support

[keyboardspecialist]

Description Opening as a draft as some things are still in progress, and this is a long running feature branch.

This adds support for rendering Gaussian splats from 3D Tiles 1.1 with SPZ compression using a proposed draft extension https://github.com/KhronosGroup/glTF/pull/2490

Another PR will remove old Model pipeline code

Affected files

• GltfLoader - Incorporates SPZ extension support
• GltfSpzLoader - Loads and decodes SPZ buffers within a glTF file
• GltfVertexBufferLoader - SPZ loading support
• PrimitiveLoadPlan - SPZ post-processing. Mostly for interleaving color with alpha and ensuring correct model orientation.
• GaussianSplat3DTilesContent - Manages loading glTF with SPZ Gaussian Splats
• GaussianSplatPrimitive - new primitive for managing a tileset's splats, sorting, and rendering.
• GaussianSplatRenderResources - Container for common render settings and shader building
• PrimitiveGaussianSplatVS.glsl - reworked vertex shader to run outside Model pipeline stages
• PrimitiveGaussianSplatFS.glsl - reworked fragment shader to run outside Model pipeline stages

Known Issues

• ~View matrix projection is currently incorrect. Splats are rendered in wrong orientation in relation to the camera.~ Fixed
• Handling tilesets with Empty3DTileContent root tiles.

Issue number and link

Testing plan

Will need to gather some single and multi-tile glTF SPZ assets to incorporate into the general testing suite.

Updated sandcastle, possibly a new asset with more rendering controls.

Author checklist

• [x] I have submitted a Contributor License Agreement
• [x] I have added my name to CONTRIBUTORS.md
• [x] I have updated CHANGES.md with a short summary of my change
• [x] I have added or updated unit tests to ensure consistent code coverage
• [x] I have updated the inline documentation, and included code examples where relevant
• [x] I have performed a self-review of my code

[github-actions[bot]]

Thank you for the pull request, @keyboardspecialist!

:white_check_mark: We can confirm we have a CLA on file for you.

[ggetz]

I'm taking an additional pass on code and testing now.

Just to make sure everyone is on the same page, this is my current understanding of the plan for this PR. @keyboardspecialist @weegeekps, please correct me if I misunderstood anything.

• The goal is to merge this PR into main after the June 2nd release
• There are a few very minor known issues that @keyboardspecialist will be documenting
• There will be followups for refining the Sandcastles, CHANGES.md updates, and other messaging

[lilleyse]

Another PR will remove old Model pipeline code

I think it should happen in this PR so that old code isn't going into main and it's easier to review.

[weegeekps]

Home stretch todo list:

• [x] Flickering Bug
• [x] Root-tile visibility issue
• [x] Picking makes tile disappear. (Selected tile is empty so hides tileset).
• [x] Potential race condition; Only fire new sort if current sort is done. Could lead to simplifications in the update loop.
• [x] Displaying splats in the same position. (Easy fix: turn on depth test)
• [x] Fix required option in API
• [x] Update with main
• [x] Unit tests
• [x] Documentation
• [x] Change notes
• [x] Update Sandcastle, support split screen

[weegeekps]

Sandcastle example is updated. I took a crack at resolving the test errors, but I'm not sure what exactly is wrong. @ggetz, we might need your insight.

[ggetz]

The Sandcastle with the slider looks great by the way! That really shows the benefits of splats!

[wumingz]

Can the Cesium code in the splat-spz-concept branch render 3DGS data normally?

However, using the spalt-shader branch can render 3dgs data normally.

var tileset = await Cesium.Cesium3DTileset.fromUrl(url, { debugTreatTilesetAsGaussianSplats: true, });

[keyboardspecialist]

Can the Cesium code in the splat-spz-concept branch render 3DGS data normally?

However, using the spalt-shader branch can render 3dgs data normally.

var tileset = await Cesium.Cesium3DTileset.fromUrl(url, { debugTreatTilesetAsGaussianSplats: true, });

This branch and the splat-shader branch target different experimental draft extensions for glTF. By normally do you mean without that debug flag? We're currently using the 3D Tiles extensions to determine whether or not a glTF has Gaussian splats with the KHR_spz_gaussian_splats_compression extension. The Gaussian splat sandcastle on this branch demonstrates loading without needing any extra flags.

See https://github.com/CesiumGS/3d-tiles/pull/810

[RenaudKeriven]

About quaternion errors, a friend of mines pointed me to https://github.com/nianticlabs/spz/issues/31 and a solution he wrote https://github.com/nianticlabs/spz/pull/36. Yet, he told me that for some rotations, there is no way to go under 6 deg error. He is asking me if we looked at SOGS https://blog.playcanvas.com/playcanvas-adopts-sogs-for-20x-3dgs-compression/

[ggetz]

@keyboardspecialist Are we using the correct version of the wasm module here, and in the expected way? I noticed the following deprecation warnings in the console which seem to be coming from @cesium/wasm-splats.

[javagl]

This is not in a "Draft" state, but probably also not finished yet. When this is ready for more thorough testing, then I'd probably give it a try.

Until then: One thing that I stumbled over a while ago: There seem to be some assumptions about the structure of the glTF of which I'm not sure whether they will always hold: https://github.com/CesiumGS/cesium/blob/8a362a0aa2e038cde8e0a50fd83d60e20b1eb9c1/packages/engine/Source/Scene/GaussianSplat3DTileContent.js#L381 (The case of two splat primitives in one glTF shouldn't be too far fetched...).

This may be related to another point: I created some dummy/experimental "Spz-To-Tileset" functionality, and noticed that back then, it was also necessary to introduce an additonal child node in the tileset JSON for which I couldn't explain why it was necessary, but it seemed to not pick up some transforms otherwise. (I'd add details based on a dedicated test pass, when the time is right).

Also related: Someone just asked about How to rotate (and scale) a Gaussian Splat in CesiumJS. And I tried out some sandcastle that modifies the tileset modelMatrix (link in my answer), and it looks like these modifications are not properly taken into account.

[ggetz]

@javagl This is PR is in it's final stages, but the support itself is "experimental". Please let us know if you have issues testing.

We do expect additional updates to the functionality in the near future though, so comments like the one you've added above may not be addressed as part of this PR, but are valid feedback which should be addressed.

[keyboardspecialist]

@keyboardspecialist Are we using the correct version of the wasm module here, and in the expected way? I noticed the following deprecation warnings in the console which seem to be coming from @cesium/wasm-splats.

Fixed with the latest commit. We were passing initSync the raw wasm binary buffer where it expects an object with a module property.

[ggetz]

Looking good @keyboardspecialist! Thanks to you, @weegeekps, and @lilleyse for all the work here!

[GISUser1]

GISBox now fully supports the KHR_spz_gaussian_splats_compression slicing feature. Its groundbreaking slicing compression technology removes performance barriers for large-scale applications of Gaussian splats in Web3D, digital twin, and other scenarios.