[Feature] Handling sky method from WildGaussians

[LeaFendd]

Hey guys! WildGaussians introduced a method to handle sky in Sec 3.4. I think it is a good idea so I want to implement this feature in gsplat.

I divide the implementation of this feature into three steps:

1. Image backgound rasterization in rasterize_to_pixels.
2. Grads computation of the image backgourd, it can be disable in general 3dgs pipeline.
3. Gaussians sky sphere mentioned in WildGaussians Sec3.4.

I want rasterize the Gaussians sky sphere independently as the image background of the scene, so that I don't need to consider the adaptation with general 3dgs pipeline in cuda code.

In this implementation plan, the only modification in general 3DGS pipeline is the input of backgroud, from a single color to a full image.

This is my initial plan, do you guys have any suggestions?

[MrNeRF]

You could pass in just the sphere with the other gaussians. The sphere just needs the gaussians to be at opacity 1. However, you can see from the depth video from the wild gaussian homepage, that their method is not working properly. There are still many gaussian left in the sky. I think masking the sky would be here much more effective

[LeaFendd]

You could pass in just the sphere with the other gaussians. The sphere just needs the gaussians to be at opacity 1. However, you can see from the depth video from the wild gaussian homepage, that their method is not working properly. There are still many gaussian left in the sky. I think masking the sky would be here much more effective

In my impl, the sky sphere is rasterized to a full background image alone. When rendering depth, I'll pass a empty backgroud, so that the sky sphere wouldn't lead to error depth rendering.

[liruilong940607]

Hi, the background modeling is useful in many cases. But I guess you could just take the render_colors and render_alphas from the output of the rasterization() function and do a alpha compositing via:

image = render_colors + (1 - render_alphas) * background_image

This way you don't have to modify anything in gsplat CUDA kernels