aframe icon indicating copy to clipboard operation
aframe copied to clipboard

Add 3D gaussian splats support

Open dmarcos opened this issue 8 months ago • 15 comments

Last big item for 1.7.0 release. Still work in progress. Based on https://github.com/quadjr/aframe-gaussian-splatting implementation

I adapted style, made a few minor tweaks and commented thoroughly. I still don't fully understand some of the details. Good opportunity to deepen understanding of 3D gaussian splats for those interested. Before merge we should thoroughly understand the code and optimize.

I recommend getting acquainted with the original paper: https://arxiv.org/pdf/2308.04079

In the code comments you'll see references to other relevant literature.

In VR on a Quest 3 when the sample splat is fully loaded (~1.1M splats rendered) I get ~14fps.

Quick overview:

  1. As output of the training process splats are represented as ellipsoids with a position, scale, color and shape that are later projected to 2D ellipses to be renderer on screen
  2. Each splat is rendered as a quad with a color that gets positioned, scaled and shaped following each ellipse characteristics.
  3. Splats are sorted in a worker. Splats are instances and we have to sort them out to render them in the correct order (back to front)
  4. Current implementation doesn't support spherical harmonics. Color of splats varies depending of point of view vs a fixed color.

dmarcos avatar Feb 09 '25 07:02 dmarcos