nerfstudio
nerfstudio copied to clipboard
nerfstudio-project
With large dataset, Splatfacto never densify gaussians
Describe the bug
I'm experiencing an issue while attempting to train Splatfacto with a large dataset (>3000 frames) using default parameters. The do_densification flag isn't being triggered, resulting in densification not taking place.
relevant code: https://github.com/nerfstudio-project/nerfstudio/blob/45db2bcfabe6e0644a3a45a50ed80a9a685ddc34/nerfstudio/models/splatfacto.py#L386C1-L395C33
While adjusting parameters like refine_every and reset_alpha_every can work around this issue, it would be helpful to give a warning here or adjust threshold automatically, like the following:
self.step % reset_interval > min(reset_interval //2, self.num_train_data + self.config.refine_every)
Interesting, I've never trained with a dataset that large. The reason that check is there is because if a camera isn't observed before the gaussians get culled, their opacity won't have time to increase and they will be falsely culled. It could be good to automatically set the reset interval high enough to account for this.
Setting reset interval high doesn't prevent the issue occur again once the number of frames exceeds 15000.
In my case, not observing every frame in the dataset before culling wasn't a problem, as there was enough redundancy in the dataset.
Anyway, thanks for the excellent work. With the Inria's official implementation, it wasn't possible to train a dataset of this size.
We need better mechanism of resetting and splitting to handle large set of input images.
A little bit more explanation: when training gaussian splatting/splatfacto, we are resetting opacity every 3000 iterations to reduce the number of floaters. After resetting, we assume the training needs to iterate over all training images before resuming pruning/densification. Otherwise, there is a good chance that resetted gaussians can never be recovered to its full opacity before they are pruned / densified.
When a dataset has a lot more images (thousands), iterating over all training images consumes majority of the 3000 iterations and leave very little space for densification. To support large dataset training by default, we need to adaptively decide the number of resetting window (say, three times more than the number of input images) as well as the number of total iterations.