Implement 3D Gaussian splatting viewer

[ssheorey]

Milestones:

• Study 3DGS implementations and identify best option, risks and blockers for integration
• Study filament renderer and Open3D-filament integration
• 3DGS Rendering implementation (view independent only)-> PR 1
• View dependent rendering. -> PR 2
• Performance optimization -> PR 3
• Demos -> PR 3

Resources:

Pre-trained INRIA models: https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/datasets/pretrained/models.zip Optional Validate display of 3DGS .ply / .splat files from luma.ai and polycam Vulkan compute shaders implementation: https://github.com/shg8/3DGS.cpp/tree/main [LGPL-v2.1] WebGL implementation using vertex/fragment shader: https://github.com/kishimisu/Gaussian-Splatting-WebGL, https://github.com/antimatter15/splat

WebGL implementations only support view independent rendering. (no spherical harmonics).

Components: (vkGS -> O3D GS)

• GLM
• Vulkan Memory Allocator
• VkRadixSort

Technical open questions:

• How will integration with Filament work? (e.g. Vulkan callbacks, shared Vulkan buffers).
• An alternative is to use Vulkan directly, and only give the final rendered image to filament to display.
• For the initial implementation, round trip to system RAM is OK, but after optimization we want all data and operations on the GPU.

[linghugoogle]

hi，have you integrated with filament work? Thanks.