playcanvas
Rendering of Postshot PLY looks very different in Supersplat
Good day. I have a data set created in Blender of a wine bottle as test (transparency, different materials, reflections). In Postshot (v. 0.5.28) it looks really good after training, but after exporting to PLY and opening it in Supersplat (v.1.15.0) it looks much worse. Also in the mckellog viewer and Gracia. This never happend before, or I at least never realized this. The SH are at level three in Postshot and Supersplat. The text is readible in Postshot but in Supersplat not anymore. In https://arthurbrussee.github.io/brush-demo/ it looks like in Postshot, with readible text etc. Can somebody help or has an idea what this could be?
here the PSHT file: https://cloud.xr-essential.com/s/rYWXiT8DRRRBZPS
Hi @akutterxressential ,
It looks like like your scene might be generated using MipSplatting (an antialiasing technique), which has the same data as normal PLY, but must be rendered differently.
Does your blender plugin have the option to generate normal PLY without mipsplatting?
It would be awesome to support mipsplatting in SuperSplat as well, but I'm not sure when we'll get the chance.
Hi @slimbuck , I am very interested in this and would like to port mipsplating into supersplat. I am quite familiar with the original CUDA rendering code, but not familiar with the supersplat project. Is there a significant difference between the rendering logic of supersplat and the CUDA version? If not, could you tell me which file contains the code for calculating the 2D covariance matrix?
Oh that's very exciting @wongshek!
I come from the opposite direction to you: I am not familiar with the original cuda code. I took some magic 3d->2d covariance vector math back in the day and implemented rendering using traditional WebGL vertex and fragment shaders.
In order to implement the original rendering approach (and mipsplatting), I believe you will need access to compute shaders which requires WebGPU.
The underlying playcanvas engine already supports WebGPU, but SuperSplat will need a few changes before we can migrate over. To see a few compute examples in the engine see https://playcanvas.vercel.app/#/compute/particles.
If you are serious about working on the rendering side, I can chat with the team about prioritising our move to WebGPU!
@slimbuck, wow, I thought everyone started by reading the original CUDA code before moving on to writing rendering code in other languages.
SuperSplat is an impressive piece of work! In addition to mipsplatting, I’d like to incorporate depth rendering (biased or unbiased) logic, as it should address basic annotation and measurement needs.
I’m definitely interested in working on the rendering side, but I’m not very familiar with WebGL or WebGPU, so I might have a lot of questions. Perhaps we can establish a more efficient way to communicate?
@wongshek Come and join us on the Discord! The link is in the Help menu is SuperSplat. The channel is called "🎨-gaussian-splatting".
Hey @wongshek ,
The playcanvas engine stores "chunks" of shader code, which are concatenated at runtime using a c-like preprocessor to construct the final shader.
To give you a flavour of our gaussian splat rendering, take a look at the main function https://github.com/playcanvas/engine/blob/main/src/scene/shader-lib/chunks/gsplat/vert/gsplat.js
The rest of the folder contains source to the other chunks handling various parts of rendering. For example evaluating sh and extracting the compressed.ply data.
The magic which calculates 2d covariance vectors from 3d happens in https://github.com/playcanvas/engine/blob/main/src/scene/shader-lib/chunks/gsplat/vert/gsplatCorner.js.
The GPU is rendering quads, one per gaussian and this function executes 4 times - once for each corner.
This is how we currently do things, but of course a WebGPU compute pipeline will look very different. I'm not sure if this helps at all, but I will get back to you on our WebGPU plans in SuperSplat.
Thanks!
It seems some of the artifacts are related to blending precision (presumably because this model contains a huge number blended layers of gaussians).
Here is the bottle rendered with increasing precision of backbuffer pixel format:
| 8 | f16 | f32 |
|---|---|---|
@slimbuck So what's the course of action here? Should we bias on performance and stick with a 8? Or bias for visual fidelity and select f32. Or compromise on f16? Or make it an option (or even part of the HQ/LQ setting in the SuperSplat viewer)?
Hi @slimbuck
I also have very different rendering results between SIBR viewer vs Supersplat(v 2.1.0).
The PLY file was obtained by using gsplat(v 1.4.0) mcmc method.
Is this also related to blending precision ?
do you have any suggestions to look into?
thanks.
Hi @jihan1218 ,
This looks to me like sorting related artifacts, not an issue with render target precision.
The issue is that we use a set number of bits (base on the total number of gaussians) to bin-sort splats for rendering.
Since MCMC generates extremely large and very far-away splats, the near to far depth range for sorting is extreme and so few bits are left for the actual scene. What you get is rendering artifacts like here.
You can confirm this yourself by deleting the very far-away gaussians in your scene and look again at the building.
This issue (with MCMC scenes specifically) has come up a number of times for different users. We will try to improve our sorting approach because this should just work!
Thanks!
@slimbuck appreciate for the fast replay and pointing out the issue! I'll try out limiting splats from the far field. Thanks!
@slimbuck So what's the course of action here? Should we bias on performance and stick with a 8? Or bias for visual fidelity and select f32. Or compromise on f16? Or make it an option (or even part of the HQ/LQ setting in the SuperSplat viewer)?
Sorry @willeastcott I forgot to reply. For me it's between always rendering float32 or exposing an option to toggle float32.
Actually... another option is we could possibly render color to RGBA8 and alpha to a float32 using MRT. If we can get blending to work correctly and this layout solves the original issue, this might actually be a reasonable compromise for SuperSplat.
We have more reports of this issue on the forums (see https://forum.playcanvas.com/t/postshot-to-supersplat/39376) and I was wondering @akutterxressential - was your bottle scene generated using the camera array blender plugin tool? https://toppinappi.gumroad.com/l/Camarray
Thanks!
We have more reports of this issue on the forums (see https://forum.playcanvas.com/t/postshot-to-supersplat/39376) and I was wondering @akutterxressential - was your bottle scene generated using the camera array blender plugin tool? https://toppinappi.gumroad.com/l/Camarray
Thanks!
Hi @slimbuck No, I used this tool from one of my students:
https://blendermarket.com/products/splatman?search_id=33504539