aframe
aframe copied to clipboard
Add 3D gaussian splats support
Last big item for 1.7.0 release. Still work in progress. Based on https://github.com/quadjr/aframe-gaussian-splatting implementation
I adapted style, made a few minor tweaks and commented thoroughly. I still don't fully understand some of the details. Good opportunity to deepen understanding of 3D gaussian splats for those interested. Before merge we should thoroughly understand the code and optimize.
I recommend getting acquainted with the original paper: https://arxiv.org/pdf/2308.04079
In the code comments you'll see references to other relevant literature.
In VR on a Quest 3 when the sample splat is fully loaded (~1.1M splats rendered) I get ~14fps.
Quick overview:
- As output of the training process splats are represented as ellipsoids with a position, scale, color and shape that are later projected to 2D ellipses to be renderer on screen
- Each splat is rendered as a quad with a color that gets positioned, scaled and shaped following each ellipse characteristics.
- Splats are sorted in a worker. Splats are instances and we have to sort them out to render them in the correct order (back to front)
- Current implementation doesn't support spherical harmonics. Color of splats varies depending of point of view vs a fixed color.