nerfstudio-project
Gaussian splatting + WebGL
I spent some time studying + adapting https://github.com/antimatter15/splat/. It's really cool stuff.
https://github.com/nerfstudio-project/viser/assets/6992947/744329df-463a-4458-a19e-f72d0c052c87
Work in progress: there's a lot of low hanging fruit for optimization, and the current implementation clashes with the <EffectComposer /> we use for outline/hover effects.
Hey, I just tried to run the demo with Chrome and Firefox, but my display is getting incredibly slow. GPU and CPU usage stays low.
Ubuntu 22.04 LTS + GTX3080 Chrome: 117.0.5938.132 (64-bit) Firefox: 117.0.1 (64-bit) NVIDIA-SMI 525.125.06 Driver Version: 525.125.06
Does anyone have an idea what the problem could be?
Can you confirm hardware acceleration is enabled (chrome://gpu)?
Graphics Feature Status
- Canvas: Hardware accelerated
- Canvas out-of-process rasterization: Enabled
- Direct Rendering Display Compositor: Disabled
- Compositing: Hardware accelerated
- Multiple Raster Threads: Enabled
- OpenGL: Enabled
- Rasterization: Hardware accelerated
- Raw Draw: Disabled
- Skia Graphite: Disabled
- Video Decode: Hardware accelerated
- Video Encode: Software only. Hardware acceleration disabled
- Vulkan: Disabled
- WebGL: Hardware accelerated
- WebGL2: Hardware accelerated
- WebGPU: Hardware accelerated
Which splat file are you loading? Splats are currently rendering smoothly in Chrome on both my laptop (M1 Mac) and phone (Samsung S21+).
sort timing notes:
| # Gaussians | 270K | 1M |
|---|---|---|
| Array.sort | 79ms | 380ms |
| JS counting sort | 25ms | 140ms |
| C++ counting sort | 14ms | 53ms |
I'm leaning toward closing the PR; it's working, but:
- Still clashes with outline effects /
EffectComposer. - Doesn't sort properly when there are multiple splat files.
- Can be optimized a lot. (in particular I would want to move more data into textures, as opposed to attributes)
After I implemented this splatting capabilities have also popped up in external libraries (example: https://github.com/pmndrs/drei#Splat); it may make sense to adapt one of these instead.
This is merged and the majority of kinks are worked out, but it'll be a while before it hits the stable API.
If you want to try it you can install viser from source and call server.scene._add_gaussian_splats() (or run the example in the examples/experimental directory): https://github.com/nerfstudio-project/viser/blob/ba17c7c40c955a85c845b685a767fbdd9bac6ba2/src/viser/_scene_api.py#L935-L963
I ran the command python examples/experimental/gaussian_splats.py --splat-paths ~/data/test1-splat/point_cloud/iteration_30000/point_cloud.ply. It successfully loaded the ply file with the message "PLY file with num_gaussians=410549 loaded in 0.6525325775146484 seconds". But when I opened http://0.0.0.0:8080, it shows a blank web page in my browser.
