Issue with Gradient Anomalies in 3D Gaussian Optimization Using ThreeStudio

[chensjtu]

I've encountered an issue while trying to optimize 3D Gaussian using ThreeStudio. The gradient behavior seems abnormal. In the 3D Gaussian optimization process, the gradient of visible_points is crucial for determining whether to clone and split Gaussian points. However, during my tests in PyTorch Lightning, the gradients of visible_points are unusually large and often result in NaN values. Is this behavior expected? How can I modify my code to achieve cloning and splitting similar to the original version of the code? the grad:

        [    nan,     nan,  0.0000],
        [-0.1534,  0.1053,  0.0000],
        ...,
        [ 0.0000,  0.0000,  0.0000],
        [ 0.7816,  0.5890,  0.0000],
        [    nan,     nan,  0.0000]], device='cuda:0')```

[DSaurus]

Hi @chensjtu ,

I haven't checked yet, but I'd like to know which branch you are using. I have another branch called saurus/gaussian for Gaussian Splatting. Will this branch encounter the same issues?

[chensjtu]

Actually, I use the codebase of GaussianDreamer, which is developed on threestudio. The visible points' grad is abnormal, causing the number of points doubled every clone operation.

[DSaurus]

That's interesting. I have also noticed the number of points will be doubled every time. I will check it.

[DSaurus]

@chensjtu I haven't encountered the same issue in my implementation. You could consider using threestudio-3dgs extension.

[chensjtu]

wow! That's amazing! Many thanks for the effort made on this issue! I'll check the differences of the code. Maybe I can find what's wrong.