nerfstudio-project
Rendering issue with wide-angle cameras using gsplat
Hello, when I try to render using gsplat, I notice that after the field of view (fov) is greater than a certain value, the larger the fov, the grayer the image becomes. What could be the reason for this? And how can I achieve the expected effect?
Below are images rendered from the same perspective using different fovs in sequence.
I noticed the same issue. Using high FOV result in a lot of extra floater gaussians to appear
Quite straightforward to reproduce:
https://github.com/user-attachments/assets/9b97f808-9b0c-4ecd-b372-82dbf181f70f
In my use case, I would be interested in high FOV to produce distorted images (radial, tangentiels) afterhand
Quite straightforward to reproduce:
high_fov.mp4 In my use case, I would be interested in high FOV to produce distorted images (radial, tangentiels) afterhand
I suspect this problem might be related to the pinhole camera model. In the original 3DGS, the light cone model was used. When implementing the pinhole camera model in gsplat, approximate calculations are involved, while the light cone/fisheye camera does not require such calculations. When the field of view (FOV) is relatively large, the covariance of the Gaussian sphere projected to a 2d ellipse at the image edges will increase abnormally. A better solution is to use the fisheye camera model or the light cone model.
I'm also running into a similar problem, but during training with very high FOV images. The images I'm using are very similar to the ones in this issue:
- #530
My guess is the rasterizer is the same for the viewer and the trainer, so they would both have this problem.
It's like increasing the FOV is almost like zooming the camera backwards until it renders splats from behind the camera.
Results from using pinhole images:
Results from using high FOV fisheye images:
https://github.com/user-attachments/assets/05339ec3-db26-4a3c-99d7-277bda5ec8ea
Note: This is being rendered in an external tool, so the artifact seen in the video is not necessarily related to the bug. It's likely just an effect it has on the resulting splat.
