How to generate a good splat of a room

[adricostas]

Hello,

I'm just opening this issue to see if someone can help me to increase the quality of my splat. I'm trying to generate a splat of a room. After several trials using datasets extracted from a video, I decided to try capturing images in a static mode to avoid the motion blur. My dataset has 76 images (3024 × 4032) with enough overlap between them.

I'm using this command to process the data: ns-process-data images --verbose --data $DATASET_PATH --output-dir $OUTPUT_PATH --no-gpu --matching-method exhaustive --matcher_type superpoint+lightglue

It seems that the result of the SfM is correct:

Reconstruction:
	num_reg_images = 76
	num_cameras = 1
	num_points3D = 13394
	num_observations = 48398
	mean_track_length = 3.61341
	mean_observations_per_image = 636.816
	mean_reprojection_error = 1.45261
	num_input_images = 76
[13:07:24] 🎉 🎉 🎉 All DONE 🎉 🎉 🎉                                                images_to_nerfstudio_dataset.py:132
           Starting with 76 images                                                   images_to_nerfstudio_dataset.py:135
           Colmap matched 76 images                                                  images_to_nerfstudio_dataset.py:135
           COLMAP found poses for all images, CONGRATS!                              images_to_nerfstudio_dataset.py:135

Then, I use the following command to generate the splat:

ns-train splatfacto --data $OUTPUT_PATH --pipeline.model.cull_alpha_thresh=0.005 --pipeline.model.continue_cull_post_densification=False --pipeline.model.use-scale-regularization True

The splat seems to be very good in some parts, however there is a lot of noise "in the air":

I don't know why this noise appears if it does not exist in the sparse pointcloud. Can be due to the parts that do not have features like walls, ceil, etc. or the light changes because of the window, for instance? Any advice to improve the result?

Thank you very much in advance!

[jb-ye]

Could you try remove --matcher_type superpoint+lightglue from the colmap commands? and remove --pipeline.model.use-scale-regularization True as well, or simply use splatfacto-big.

Also use the latest nerfstudio commit which has this fix: https://github.com/nerfstudio-project/nerfstudio/pull/3382

[adricostas]

Removing --matcher_type superpoint+lightglue from the colmap commands and --pipeline.model.use-scale-regularization True from train command the result is similar:

I20240829 06:36:11.320797 140647645972160 bundle_adjustment.cc:942]
    Residuals : 328420
   Parameters : 111823
   Iterations : 27
         Time : 10.5064 [s]
 Initial cost : 0.725944 [px]
   Final cost : 0.715718 [px]
  Termination : Convergence

I20240829 06:36:11.320816 140647645972160 timer.cc:91] Elapsed time: 0.181 [minutes]
[06:36:11] 🎉 Done refining intrinsics.                                                              colmap_utils.py:184
[06:36:13] 🎉 🎉 🎉 All DONE 🎉 🎉 🎉                                                images_to_nerfstudio_dataset.py:132
           Starting with 76 images                                                   images_to_nerfstudio_dataset.py:135
           Colmap matched 76 images                                                  images_to_nerfstudio_dataset.py:135
           COLMAP found poses for all images, CONGRATS!                              images_to_n

With splatfacto-big, also a similar result:

I will try with the last version then, but I'm a bit skeptical about that because I have already tried to generate the sparse reconstruction with a external software and execute ns-process with the --skip-colmap flag, and the result is also noisy.

[hariharan1412]

This is a problem with both the dataset and Gaussian splatting, If an area in image presented with very few features ( white/black area/walls ), the model try to fit that area with few larger Gaussians, which will cause the blurryness in the overall splats covers the finer details, No hyperparameter tuning will help you solve this also

If possible, Stick some wall painting in those walls and try again, this will increase the number of features in the wall, This is an experiment, if you could do this, please share your observation and output also

[AntonioMacaronio]

I've always found splatting inside a room to be very difficult - I recommend giving this paper a try: dn-splatter, it does a much better job inside rooms and is built on top of nerfstudio, so it should be very easy to install

[isso313]

I did something similar and obtained good results by turning around in spirals, and you might wanna increase the number of images and try it out again.

[daniis-h]

Hi. I'm facing issue of exporting train output. the splatfacto training output seems very pleasant but when i export it, it looks like dust

[vigneshrk29]

Hi @adricostas , sorry, could I check if the output you showed for splatfacto was through the viser tool or after you exported the .ply object? I get reasonable results during training but the exported object has a factor of the details shown in the viser tool. How do you export the .ply object to show a fair representation of the room?

[adricostas]

Hi @vigneshrk29, sorry, but to be honest I don't remenber if this output was through the viser tool or not :|.

[AntonioMacaronio]

@daniis-h I'm sorry about the terribly late response, but the reason this is because the nerfstudio models such as nerfacto and splatfacto scale the output to be within a unit cube.

There is a PR here to scale the export here but it is a little old. If you want to export to not be dust, you'd have to scale up everything in the ply file